[00:01:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:02:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.00, 8.86, 8.71 [00:03:09] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 422 failures. Last run 2 minutes ago with 422 failures. Failed resources (up to 3 shown): File[/etc/vim/vimrc.local],File[/usr/local/bin/cgroup-mediawiki-clean],File[wildcard.miraheze.org-2020-2],File[wildcard.miraheze.org-2020-2_private] [00:04:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.04, 8.85, 8.73 [00:05:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.89, 3.79, 3.51 [00:05:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.64, 8.51, 7.89 [00:06:19] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+1/-1/±0] 13https://github.com/miraheze/CreateWiki/compare/5fff0121b6dc...4e8dd827d0d6 [00:06:21] [02miraheze/CreateWiki] 07Universal-Omega 034e8dd82 - Update and rename InsertWikiTest.php to WikiManagerTest.php [00:06:22] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Create InsertWikiTest.php - 13https://github.com/miraheze/CreateWiki/pull/301 [00:06:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.36, 8.47, 8.15 [00:06:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.12, 9.51, 9.20 [00:06:49] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/4e8dd827d0d6...e0316376af63 [00:06:50] [02miraheze/CreateWiki] 07Universal-Omega 03e031637 - Update extension.json [00:06:52] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Create InsertWikiTest.php - 13https://github.com/miraheze/CreateWiki/pull/301 [00:07:51] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.99, 7.58, 7.63 [00:08:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.18, 9.65, 9.06 [00:08:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.39, 8.37, 8.15 [00:08:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:09:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.83, 3.43, 3.03 [00:10:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.78, 8.81, 8.18 [00:10:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.59, 9.93, 9.26 [00:10:34] miraheze/CreateWiki - Universal-Omega the build has errored. [00:11:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.26, 3.07, 2.95 [00:11:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.68, 9.05, 8.20 [00:12:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.78, 9.93, 8.82 [00:12:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.37, 9.78, 9.49 [00:12:49] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e0316376af63...155ba1f114e8 [00:12:51] [02miraheze/CreateWiki] 07Universal-Omega 03155ba1f - Add assertNull [00:12:52] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Create InsertWikiTest.php - 13https://github.com/miraheze/CreateWiki/pull/301 [00:13:17] [02CreateWiki] 07Universal-Omega edited pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [00:13:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.68, 8.73, 8.19 [00:14:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.58, 8.20, 8.09 [00:14:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.36, 9.99, 9.36 [00:15:33] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.01, 9.08, 8.69 [00:15:56] miraheze/CreateWiki - Universal-Omega the build has errored. [00:16:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.38, 9.85, 9.39 [00:16:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.05, 9.74, 9.53 [00:17:33] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.13, 9.13, 8.77 [00:17:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.77, 9.77, 9.18 [00:18:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.27, 9.81, 9.42 [00:18:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.49, 3.61, 3.21 [00:18:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.51, 3.16 [00:20:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.36, 3.24, 3.13 [00:20:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.69, 9.30, 9.27 [00:20:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.60, 9.16, 9.36 [00:20:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.72, 9.42, 8.70 [00:21:49] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.34, 9.57, 9.19 [00:22:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.46, 9.53, 9.34 [00:22:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.49, 3.20, 3.14 [00:22:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.15, 9.94, 8.97 [00:23:31] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.50, 9.11, 8.80 [00:23:37] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/155ba1f114e8...8f7354ee9c04 [00:23:38] [02miraheze/CreateWiki] 07Universal-Omega 038f7354e - Set wgConf suffixes [00:23:40] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [00:23:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.47, 8.76, 8.94 [00:24:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.30, 9.73, 9.02 [00:25:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.62, 8.91, 8.78 [00:27:07] miraheze/CreateWiki - Universal-Omega the build has errored. [00:27:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.47, 9.20, 8.88 [00:27:32] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/8f7354ee9c04...0e9d6afb39cc [00:27:33] [02miraheze/CreateWiki] 07Universal-Omega 030e9d6af - Update WikiManagerTest.php [00:27:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.17, 9.66, 9.29 [00:27:35] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [00:27:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.66, 9.19, 8.65 [00:28:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.66, 3.34 [00:28:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.28, 9.85, 9.52 [00:29:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.67, 9.30, 9.20 [00:29:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.73, 9.31, 8.95 [00:29:51] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.43, 8.31, 8.39 [00:30:25] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.11, 3.87, 3.46 [00:30:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.39, 9.67, 9.50 [00:30:45] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.81, 10.14, 9.41 [00:31:06] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [00:31:10] miraheze/CreateWiki - Universal-Omega the build has errored. [00:34:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.99, 3.66, 3.49 [00:34:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.20, 9.66, 9.40 [00:36:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.08, 10.25, 9.65 [00:37:27] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.76, 9.64, 9.18 [00:37:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.40, 9.14, 8.72 [00:38:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.64, 3.85, 3.58 [00:39:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.50, 9.99, 9.51 [00:39:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.60, 9.07, 9.02 [00:39:51] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.11, 8.00, 8.38 [00:40:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.53, 3.49 [00:40:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.61, 9.51, 9.52 [00:41:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.65, 3.43, 3.90 [00:42:09] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.59, 3.21, 3.38 [00:42:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.25, 9.95, 9.66 [00:43:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.56, 9.62, 9.49 [00:44:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.54, 9.76, 9.92 [00:45:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.70, 3.57, 3.81 [00:45:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.91, 9.31, 9.03 [00:46:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.40, 3.43 [00:46:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.06, 9.64, 9.63 [00:46:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.13, 9.60, 9.36 [00:47:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.50, 3.25, 3.67 [00:48:27] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.13, 9.66, 9.62 [00:48:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.76, 9.73, 9.44 [00:49:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.92, 4.20, 3.96 [00:49:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.53, 9.80, 9.34 [00:50:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.34, 9.14, 9.44 [00:51:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 3.88, 3.87 [00:51:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.25, 9.68, 9.56 [00:51:32] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.84, 2.77, 1.70 [00:51:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.62, 3.12, 3.34 [00:52:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.14, 9.83, 9.65 [00:53:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.69, 9.07, 9.37 [00:53:22] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.51, 9.68, 9.34 [00:54:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.44, 9.87, 9.72 [00:54:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.78, 9.54, 9.58 [00:55:31] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.02, 1.88, 1.59 [00:56:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.66, 9.19, 9.48 [00:56:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.40, 9.34, 9.17 [00:57:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.41, 9.65, 9.52 [00:57:31] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.35, 1.70, 1.55 [00:58:18] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.44, 9.84, 9.65 [00:58:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.98, 9.46, 9.23 [01:00:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.24, 9.21, 9.43 [01:01:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.13, 2.60, 3.26 [01:02:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.43, 9.40, 9.45 [01:02:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.37, 9.72, 9.36 [01:04:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.57, 9.32, 9.44 [01:04:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.68, 9.45, 9.32 [01:05:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.76, 9.92, 9.77 [01:05:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.09, 9.66, 9.80 [01:05:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.32, 4.11, 3.62 [01:07:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.55, 3.54, 3.47 [01:08:41] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [01:09:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 3.01, 3.28 [01:10:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.65, 9.91, 9.52 [01:12:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.54, 9.37, 9.38 [01:14:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.04, 9.56, 9.44 [01:16:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.08, 9.37, 9.38 [01:17:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.12, 9.14, 9.32 [01:19:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.04, 8.32, 8.99 [01:20:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.02, 8.11, 8.48 [01:20:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.56, 8.85, 8.35 [01:21:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.25, 8.80, 9.07 [01:23:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.05, 8.17, 8.81 [01:23:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:24:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.34, 10.01, 9.19 [01:25:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.93, 10.04, 9.42 [01:26:30] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.84, 8.32, 8.29 [01:27:26] RECOVERY - wiki.landev.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.landev.vn' will expire on Sat 02 Apr 2022 03:10:10 GMT +0000. [01:28:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.56, 9.43, 9.29 [01:29:10] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.65, 9.33, 9.06 [01:29:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:30:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.62, 9.28, 9.23 [01:30:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.51, 8.45, 8.95 [01:31:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 3.45, 2.96 [01:31:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.53, 9.27, 9.39 [01:32:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.02, 9.66, 9.35 [01:33:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.27, 3.38, 3.00 [01:33:09] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.32, 9.31, 9.18 [01:33:42] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/0e9d6afb39cc...b361042479b5 [01:33:43] [02miraheze/CreateWiki] 07Universal-Omega 03b361042 - Add tearDown [01:33:45] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [01:34:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:35:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.44, 9.37, 8.93 [01:36:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.24, 9.10, 9.07 [01:36:46] miraheze/CreateWiki - Universal-Omega the build has errored. [01:37:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.92, 8.80, 8.78 [01:38:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.00, 9.99, 9.67 [01:38:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.18, 9.36, 9.19 [01:39:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.08, 7.59, 8.36 [01:41:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.85, 8.11, 8.49 [01:41:09] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/b361042479b5...b08c8961f7ef [01:41:10] [02miraheze/CreateWiki] 07Universal-Omega 03b08c896 - Update WikiManagerTest.php [01:41:12] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [01:42:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.48, 9.95, 9.43 [01:43:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:43:35] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.09, 10.04, 9.40 [01:44:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 5.97, 8.40, 8.92 [01:44:32] miraheze/CreateWiki - Universal-Omega the build has errored. [01:46:21] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/b08c8961f7ef...3cab2e4b683e [01:46:23] [02miraheze/CreateWiki] 07Universal-Omega 033cab2e4 - Update WikiManager.php [01:46:24] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [01:47:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.98, 3.98, 3.40 [01:47:05] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/3cab2e4b683e...dc245ae52500 [01:47:07] [02miraheze/CreateWiki] 07Universal-Omega 03dc245ae - Update WikiManagerTest.php [01:47:08] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [01:47:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 5.58, 8.64, 9.09 [01:48:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:50:35] miraheze/CreateWiki - Universal-Omega the build has errored. [01:51:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.77, 4.68, 3.76 [01:52:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.31, 8.67, 8.70 [01:53:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.47, 8.54, 8.34 [01:53:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:54:14] [02CreateWiki] 07Universal-Omega commented on pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301#issuecomment-1049415528 [01:54:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.20, 8.83, 8.76 [01:55:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.93, 3.57, 3.54 [01:55:01] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.22, 8.15, 8.23 [01:55:14] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/dc245ae52500...1ef107a4200b [01:55:16] [02miraheze/CreateWiki] 07Universal-Omega 031ef107a - Update WikiManager.php [01:55:17] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [01:56:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.23, 9.14, 8.93 [01:57:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.06, 3.17, 3.40 [01:58:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.84, 9.45, 9.08 [01:58:32] miraheze/CreateWiki - Universal-Omega the build has errored. [01:59:07] PROBLEM - wiki.landev.vn - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.landev.vn and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:59:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.24, 7.62, 8.45 [01:59:34] PROBLEM - mon111 Puppet on mon111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[python-is-python3] [02:00:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.12, 9.19, 8.85 [02:02:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.02, 9.14, 8.88 [02:03:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:04:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.53, 9.32, 9.04 [02:06:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.07, 8.85, 8.88 [02:12:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:14:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.95, 3.30, 2.93 [02:14:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.43, 1.13 [02:16:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.09, 2.79, 2.78 [02:16:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.45, 7.89, 8.40 [02:16:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.16, 1.27, 1.11 [02:16:53] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.15, 9.82, 8.72 [02:18:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.32, 9.03, 8.56 [02:19:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.14, 8.87, 8.30 [02:20:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.45, 9.25, 8.68 [02:22:51] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.95, 8.99, 8.64 [02:24:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.22, 3.48, 3.07 [02:25:50] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.59, 8.40, 8.37 [02:26:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.15, 3.17, 3.00 [02:26:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.96, 8.18, 8.41 [02:27:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:29:32] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:30:33] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.65, 1.91, 1.52 [02:30:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.93, 8.96, 8.68 [02:32:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.42, 7.76, 8.40 [02:32:47] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.58, 9.68, 8.99 [02:34:31] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.78, 1.46, 1.43 [02:35:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.82, 8.93, 8.49 [02:36:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:36:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.43, 9.61, 9.19 [02:36:57] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [02:36:57] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4065916538238525 seconds [02:37:18] umm [02:37:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.86, 8.08, 8.23 [02:40:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.78, 3.51, 3.17 [02:40:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.08, 8.32, 7.90 [02:42:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.48, 3.20 [02:42:39] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.93, 8.19, 7.91 [02:43:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.32, 8.91, 8.54 [02:44:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.74, 3.15, 3.11 [02:44:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.68, 10.11, 9.05 [02:46:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.71, 8.67, 8.17 [02:46:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.20, 9.60, 8.99 [02:47:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.26, 8.93, 8.59 [02:47:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.73, 3.37 [02:48:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.33, 9.48, 8.51 [02:49:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.05, 3.49, 3.32 [02:50:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.85, 9.17, 8.51 [02:51:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.90, 8.60, 8.16 [02:51:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.79, 9.54, 8.99 [02:54:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 4.44, 8.02, 8.30 [02:54:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.94, 7.30, 8.18 [02:55:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.31, 7.91, 8.04 [02:55:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.70, 2.95, 3.20 [02:57:15] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.13, 7.72, 8.40 [03:04:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.95, 7.63, 8.35 [03:06:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:11:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:16:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.38, 3.10 [03:18:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 3.54, 3.18 [03:20:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.00, 2.22, 1.67 [03:21:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.08, 3.03, 3.06 [03:26:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.87, 1.89, 1.69 [03:26:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:28:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.05, 1.57, 1.60 [03:33:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.47, 3.17 [03:35:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.54, 8.74, 8.28 [03:35:49] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [03:37:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.80, 3.75, 3.35 [03:39:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.67, 3.35 [03:39:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.38, 8.38, 8.27 [03:43:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.24, 3.26, 3.29 [03:52:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.69, 8.79, 8.36 [03:54:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.62, 7.81, 8.05 [03:59:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.59, 3.32, 3.13 [04:03:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.34, 3.04, 3.07 [04:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:13:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.16, 8.43, 7.57 [04:15:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.90, 8.17, 7.57 [04:17:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.54, 3.70, 3.18 [04:17:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.97, 9.35, 7.98 [04:17:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.33, 8.18, 7.62 [04:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:20:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.79, 1.48, 1.18 [04:21:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.10, 3.18, 3.11 [04:21:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.38, 7.81, 7.65 [04:22:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.57, 1.15, 1.10 [04:23:10] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.22, 7.81, 7.81 [04:36:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:41:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:44:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.21, 2.81 [04:46:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.01, 2.78, 2.70 [04:49:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:51:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.37, 8.84, 8.13 [04:51:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.61, 7.43, 6.77 [04:52:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.50, 9.25, 8.38 [04:53:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.34, 7.65, 6.93 [04:54:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.64, 8.45, 8.19 [04:54:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:59:20] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.14, 1.81, 1.46 [05:00:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:01:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.02, 1.50, 1.38 [05:01:23] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.14, 9.42, 8.72 [05:02:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.52, 3.38, 2.88 [05:03:55] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.89, 9.29, 8.55 [05:04:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.53, 2.99 [05:05:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.68, 9.81, 9.08 [05:05:36] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:05:54] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.92, 8.05, 8.18 [05:08:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.02, 8.69, 8.30 [05:11:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.36, 9.51, 9.13 [05:12:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:12:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.99, 8.65, 8.37 [05:12:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.14, 8.90, 8.98 [05:13:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.22, 3.46, 3.39 [05:14:34] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.92, 8.41, 8.36 [05:15:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.67, 2.82, 3.16 [05:16:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.51, 9.35, 8.68 [05:17:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:18:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.60, 8.77, 8.56 [05:20:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.73, 8.19, 8.38 [05:21:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.75, 8.65, 8.50 [05:22:35] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.65, 7.45, 8.36 [05:29:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.84, 3.63, 3.21 [05:30:55] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.24, 8.44, 8.49 [05:31:23] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.77, 3.31, 3.14 [05:33:29] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [05:33:29] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406315326690674 seconds [05:33:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:38:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:41:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:44:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.94, 1.93, 1.22 [05:46:57] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.51, 1.54, 1.16 [05:47:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.07, 8.68, 8.12 [05:49:46] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.00, 2.21, 1.43 [05:51:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.78, 8.59, 7.81 [05:51:45] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.21, 1.88, 1.41 [05:52:59] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.49, 8.31, 8.18 [05:53:38] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.98, 8.19, 7.78 [05:53:45] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.77, 1.47, 1.31 [05:56:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:58:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.81, 9.07, 8.57 [06:01:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.20, 9.35, 8.47 [06:04:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.95, 9.47, 8.80 [06:05:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:06:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.12, 8.69, 8.60 [06:07:33] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.53, 8.02, 8.17 [06:08:29] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.33, 7.95, 8.35 [06:10:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:13:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.39, 8.84, 7.93 [06:15:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.31, 8.78, 8.42 [06:16:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:16:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.98, 8.38, 7.74 [06:17:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.69, 8.16, 7.89 [06:22:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.01, 8.42, 8.02 [06:23:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.86, 9.58, 8.53 [06:23:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.51, 9.27, 8.80 [06:25:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.22, 9.39, 8.59 [06:25:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.18, 2.66 [06:25:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.75, 9.45, 8.93 [06:27:16] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.58, 2.88, 2.61 [06:28:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.59, 9.39, 8.61 [06:29:34] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.90, 9.12, 8.82 [06:30:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.83, 8.69, 8.44 [06:31:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:32:21] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [06:33:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.95, 8.31, 8.42 [06:33:33] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.63, 8.89, 8.79 [06:34:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.96, 8.21, 8.34 [06:37:32] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.20, 7.54, 8.27 [06:38:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.18, 9.05, 8.64 [06:40:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.17, 8.91, 8.63 [06:42:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:44:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.02, 9.09, 8.74 [06:44:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.51, 8.04, 7.46 [06:46:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.46, 9.06, 8.76 [06:47:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.52, 8.37, 8.24 [06:48:18] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.40, 7.72, 7.52 [06:49:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.74, 8.02, 8.12 [06:49:13] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.07, 8.53, 8.09 [06:50:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.96, 8.85, 8.36 [06:52:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.21, 8.42, 8.25 [06:53:03] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.82, 8.28, 8.16 [06:54:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.44, 8.14, 8.45 [06:58:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.78, 8.52, 8.56 [06:59:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.32, 8.51, 8.31 [07:01:56] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.66, 7.76, 8.05 [07:02:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:04:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.48, 9.26, 8.86 [07:04:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.23, 8.44, 8.12 [07:06:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.15, 9.11, 8.88 [07:06:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.66, 7.65, 7.87 [07:14:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.34, 8.89, 8.75 [07:16:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.72, 8.73, 8.71 [07:19:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:20:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.55, 7.99, 8.41 [07:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:46:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:46:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.09, 8.87, 8.33 [07:48:48] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.64, 8.40, 8.22 [07:49:51] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.76, 1.20, 0.91 [07:51:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:51:51] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.61, 1.00, 0.87 [07:55:03] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.86, 9.73, 8.65 [07:57:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.76, 3.37, 2.74 [07:57:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.91, 9.65, 8.74 [07:59:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.69, 2.77, 2.60 [07:59:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.78, 8.37, 7.51 [08:01:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.47, 7.63, 7.35 [08:03:01] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.01, 8.20, 8.41 [08:09:30] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [08:32:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.54, 9.38, 8.23 [08:33:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.75, 8.23, 7.85 [08:35:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.00, 8.37, 7.87 [08:36:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.01, 9.23, 8.45 [08:37:16] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.93, 8.30, 7.91 [08:37:16] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.47, 7.66, 7.73 [08:38:01] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [08:38:01] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:38:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.49, 8.39, 8.25 [08:51:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:56:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:00:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:02:27] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/li [09:02:27] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query songcontests.eu. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:07:31] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [09:10:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:23:38] [02mw-config] 07RhinosF1 closed pull request 03#4481: cw: run 1.38 backwards compatible SQLs on creation - 13https://github.com/miraheze/mw-config/pull/4481 [09:23:40] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 - 13https://github.com/miraheze/mw-config [09:23:41] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/73ba40b4ae0b...95f4cde0aa13 [09:23:43] [02miraheze/mw-config] 07RhinosF1 0395f4cde - cw: run 1.38 backwards compatible SQLs on creation (#4481) [09:23:44] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 [09:24:40] miraheze/mw-config - RhinosF1 the build passed. [09:25:11] !log [@test101] starting deploy of {'config': True} to skip [09:25:12] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [09:25:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:25:20] !log [@mwtask111] starting deploy of {'config': True} to all [09:25:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:25:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:25:35] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 14s [09:25:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:26:41] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/config/138pre-patch.sql (START) [09:26:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:28:51] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [09:28:52] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404027462005615 seconds [09:29:37] [02puppet] 07Reception123 closed pull request 03#2428: Use default_sync = all everywhere - 13https://github.com/miraheze/puppet/pull/2428 [09:29:38] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/6a524266e1f3...d6aa33b8d1ca [09:29:40] [02miraheze/puppet] 07RhinosF1 03d6aa33b - Use default_sync = all everywhere (#2428) [09:30:41] [02puppet] 07Reception123 closed pull request 03#2427: test: setup redis for beta - 13https://github.com/miraheze/puppet/pull/2427 [09:30:42] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d6aa33b8d1ca...d28ac1379853 [09:30:44] [02miraheze/puppet] 07RhinosF1 03d28ac13 - test: setup redis for beta (#2427) [09:31:24] RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: SSL OK - fc.songcontests.eu reverse DNS resolves to cp21.miraheze.org - CNAME OK [09:31:50] [02puppet] 07RhinosF1 opened pull request 03#2429: test: enable jobqueue - 13https://github.com/miraheze/puppet/pull/2429 [09:36:14] [02puppet] 07RhinosF1 synchronize pull request 03#2429: test: enable jobqueue - 13https://github.com/miraheze/puppet/pull/2429 [09:36:59] [02puppet] 07RhinosF1 synchronize pull request 03#2429: test: enable jobqueue - 13https://github.com/miraheze/puppet/pull/2429 [09:37:51] [02puppet] 07RhinosF1 synchronize pull request 03#2429: test: enable jobqueue - 13https://github.com/miraheze/puppet/pull/2429 [09:39:31] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [09:39:31] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:45:52] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/fa76c5e58585 [09:45:53] [02miraheze/mw-config] 07RhinosF1 03fa76c5e - Beta: Send Jobqueue to test101 [09:45:55] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-2 - 13https://github.com/miraheze/mw-config [09:47:22] [02mw-config] 07RhinosF1 opened pull request 03#4482: Beta: Send Jobqueue to test101 - 13https://github.com/miraheze/mw-config/pull/4482 [09:48:26] miraheze/mw-config - RhinosF1 the build has errored. [09:49:12] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/fa76c5e58585...9ce0eb0618a5 [09:49:14] [02miraheze/mw-config] 07RhinosF1 039ce0eb0 - Update GlobalCache.php [09:49:15] [02mw-config] 07RhinosF1 synchronize pull request 03#4482: Beta: Send Jobqueue to test101 - 13https://github.com/miraheze/mw-config/pull/4482 [09:50:20] miraheze/mw-config - RhinosF1 the build passed. [09:51:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.71, 7.98, 6.90 [09:51:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.16, 8.14, 6.83 [09:52:00] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 10.07, 7.80, 4.77 [09:52:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:53:59] [02puppet] 07RhinosF1 synchronize pull request 03#2429: test: enable jobqueue - 13https://github.com/miraheze/puppet/pull/2429 [09:54:00] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.66, 7.15, 4.90 [09:55:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.03, 7.51, 6.95 [09:55:04] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.88, 8.24, 7.19 [09:56:00] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.13, 6.19, 4.82 [09:58:16] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [10:01:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 3.76, 7.20, 6.10 [10:03:47] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.73, 5.64, 5.66 [10:07:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:08:01] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [10:11:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:12:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.89, 7.35, 6.94 [10:14:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.31, 7.53, 7.05 [10:17:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.25, 7.52, 7.03 [10:21:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:21:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.95, 8.02, 7.38 [10:23:18] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.32, 1.99, 1.46 [10:27:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.73, 1.88, 1.54 [10:28:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:33:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.27, 1.59, 1.54 [10:33:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:40:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:52:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.22, 8.22, 7.63 [10:56:26] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [10:56:26] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404102325439453 seconds [10:56:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.01, 7.67, 7.60 [11:05:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:23:42] PROBLEM - wiki.thunis.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.thunis.eu All nameservers failed to answer the query. [11:25:07] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [11:25:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:29:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.52, 8.56, 7.27 [11:30:36] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mineland.eu All nameservers failed to answer the query. [11:31:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.47, 8.05, 7.24 [11:32:19] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.45, 1.75, 1.33 [11:34:13] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.49, 1.64, 1.34 [11:35:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:39:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:44:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.67, 3.73, 2.78 [11:46:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.71, 3.27, 2.72 [11:47:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.86, 8.39, 7.59 [11:48:55] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.87, 8.14, 7.61 [11:50:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.04, 8.31, 7.66 [11:52:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.62, 7.39, 7.40 [11:52:52] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.34, 5.95, 4.38 [11:53:00] RECOVERY - wiki.thunis.eu - reverse DNS on sslhost is OK: SSL OK - wiki.thunis.eu reverse DNS resolves to cp20.miraheze.org - CNAME OK [11:53:51] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [11:53:51] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406343460083008 seconds [11:56:48] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.59, 5.55, 4.67 [11:57:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.69, 8.97, 7.90 [11:57:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.89, 8.36, 7.00 [11:57:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.88, 8.20, 7.56 [11:59:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.03, 8.09, 7.06 [11:59:40] RECOVERY - wiki.mineland.eu - reverse DNS on sslhost is OK: SSL OK - wiki.mineland.eu reverse DNS resolves to cp20.miraheze.org - CNAME OK [11:59:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.71, 7.99, 7.57 [12:00:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.14, 8.28, 7.71 [12:02:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.97, 8.34, 7.81 [12:04:56] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/lib [12:04:56] /dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query fc.songcontests.eu. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:08:24] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.55, 1.81, 1.52 [12:09:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.12, 8.60, 8.06 [12:09:23] [02miraheze/GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/GlobalNewFiles/compare/68aca4a1de06...dee52283b1b5 [12:09:24] [02miraheze/GlobalNewFiles] 07translatewiki 03dee5228 - Localisation updates from https://translatewiki.net. [12:09:26] [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/1f63e526f6a3...39c4b13acc73 [12:09:27] [02miraheze/IncidentReporting] 07translatewiki 0339c4b13 - Localisation updates from https://translatewiki.net. [12:09:29] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/632c49cdf83c...7a39921ce4ec [12:09:31] [02miraheze/CreateWiki] 07translatewiki 037a39921 - Localisation updates from https://translatewiki.net. [12:09:32] [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/DataDump/compare/3d781062b06e...88351b6aa6bb [12:09:34] [02miraheze/DataDump] 07translatewiki 0388351b6 - Localisation updates from https://translatewiki.net. [12:09:35] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/landing/compare/a241404a2a99...4bfa544cf4fb [12:09:37] [02miraheze/landing] 07translatewiki 034bfa544 - Localisation updates from https://translatewiki.net. [12:09:38] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/compare/84b9c8aecf88...d8022ca8272f [12:09:40] [02miraheze/ManageWiki] 07translatewiki 03d8022ca - Localisation updates from https://translatewiki.net. [12:09:41] [02miraheze/MatomoAnalytics] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/compare/803459b8b7ca...880ae2046731 [12:09:43] [02miraheze/MatomoAnalytics] 07translatewiki 03880ae20 - Localisation updates from https://translatewiki.net. [12:09:44] [02miraheze/RemovePII] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/7491aba237fb...7aa9a4b88f07 [12:09:46] [02miraheze/RemovePII] 07translatewiki 037aa9a4b - Localisation updates from https://translatewiki.net. [12:09:47] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/WikiDiscover/compare/086af2574e9e...87282eae5443 [12:09:49] [02miraheze/WikiDiscover] 07translatewiki 0387282ea - Localisation updates from https://translatewiki.net. [12:09:50] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±8] 13https://github.com/miraheze/MirahezeMagic/compare/743af4499f8a...4553c0f0c85e [12:09:52] [02miraheze/MirahezeMagic] 07translatewiki 034553c0f - Localisation updates from https://translatewiki.net. [12:11:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.34, 8.69, 7.69 [12:12:12] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.96, 2.02, 1.65 [12:12:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.09, 8.45, 8.32 [12:12:54] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.66, 8.96, 7.95 [12:13:08] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.43, 7.96, 7.97 [12:13:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.99, 7.84, 7.51 [12:14:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.84, 1.93, 1.66 [12:14:07] miraheze/landing - translatewiki the build passed. [12:14:49] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.67, 8.27, 7.83 [12:14:59] miraheze/GlobalNewFiles - translatewiki the build passed. [12:16:00] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.93, 2.15, 1.76 [12:16:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 4.22, 3.45 [12:16:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.52, 8.62, 8.43 [12:17:54] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.55, 1.83, 1.68 [12:18:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.33, 3.65, 3.34 [12:18:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.79, 8.30, 8.33 [12:18:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.04, 9.64, 8.48 [12:20:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.27, 8.65, 8.17 [12:21:42] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.27, 1.67, 1.66 [12:22:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.08, 8.98, 8.59 [12:22:34] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [12:22:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.43, 8.94, 8.49 [12:22:53] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.80, 8.15, 8.05 [12:23:31] miraheze/MatomoAnalytics - translatewiki the build passed. [12:23:38] miraheze/RemovePII - translatewiki the build passed. [12:24:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.75, 3.02, 3.24 [12:24:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.50, 8.47, 8.37 [12:25:15] miraheze/WikiDiscover - translatewiki the build passed. [12:25:18] miraheze/DataDump - translatewiki the build passed. [12:25:40] miraheze/ManageWiki - translatewiki the build passed. [12:25:43] miraheze/IncidentReporting - translatewiki the build passed. [12:25:57] miraheze/CreateWiki - translatewiki the build passed. [12:26:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.77, 8.96, 8.67 [12:27:07] miraheze/MirahezeMagic - translatewiki the build passed. [12:28:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.05, 9.67, 8.98 [12:29:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.91, 3.23, 3.21 [12:30:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.76, 9.05, 8.84 [12:31:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.95, 2.82, 3.07 [12:34:50] RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: SSL OK - fc.songcontests.eu reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:35:59] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.10, 8.07, 8.49 [12:40:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.65, 8.33, 8.00 [12:42:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.08, 8.68, 8.18 [12:43:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.87, 9.07, 8.32 [12:45:17] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.80, 8.47, 8.18 [12:46:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.46, 8.43, 8.51 [12:46:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.77, 8.09, 8.08 [12:48:27] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.23, 8.47, 7.73 [12:49:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.05, 9.00, 8.51 [12:50:35] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.02, 9.15, 7.95 [12:50:36] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.26, 8.01, 8.34 [12:51:09] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.10, 8.36, 8.34 [12:52:16] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.33, 8.44, 7.95 [12:52:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.45, 9.07, 8.05 [12:53:25] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.74, 1.84, 1.48 [12:54:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.34, 8.85, 8.57 [12:55:25] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.32, 1.65, 1.45 [12:56:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.63, 8.24, 7.99 [12:56:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.33, 8.57, 8.48 [12:58:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.62, 8.10, 8.32 [13:09:40] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:13:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.76, 3.16, 2.70 [13:14:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.00, 8.81, 8.04 [13:14:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:15:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.05, 3.23, 2.79 [13:16:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.72, 8.05, 7.86 [13:19:58] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [13:19:59] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406836271286011 seconds [13:21:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.16, 8.55, 7.97 [13:23:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.69, 9.12, 8.39 [13:23:26] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.48, 8.20, 7.91 [13:26:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.85, 9.18, 8.30 [13:28:24] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.89, 1.53, 1.12 [13:28:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.16, 8.43, 8.15 [13:29:11] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.55, 8.32, 8.33 [13:30:24] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.90, 1.33, 1.10 [13:30:45] PROBLEM - wiki.slimelab.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.slimelab.net All nameservers failed to answer the query. [13:31:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.49, 8.95, 8.31 [13:34:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.29, 9.26, 8.60 [13:35:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.45, 8.05, 8.08 [13:39:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.07, 6.72, 4.97 [13:40:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.81, 7.85, 8.30 [13:40:21] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:43] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:44] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [13:40:52] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:54] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:56] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:56] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:59] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:41:12] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:41:29] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:41:34] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:41:50] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.39, 6.15, 4.96 [13:41:59] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:42:07] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:42:46] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22297 bytes in 5.652 second response time [13:42:47] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22297 bytes in 9.165 second response time [13:42:48] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.582 second response time [13:42:49] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.303 second response time [13:42:55] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.727 second response time [13:42:55] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.319 second response time [13:42:57] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.852 second response time [13:43:15] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.490 second response time [13:43:23] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.278 second response time [13:43:35] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.177 second response time [13:43:57] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.265 second response time [13:44:07] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.546 second response time [13:44:21] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.319 second response time [13:46:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.23, 8.41, 7.79 [13:46:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.78, 9.09, 8.67 [13:46:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.24, 9.17, 8.50 [13:46:49] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/config/138pre-patch.sql (END - exit=0) [13:46:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:47:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.02, 9.24, 8.77 [13:48:00] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.96, 7.64, 7.58 [13:48:33] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.25, 9.20, 8.40 [13:49:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.89, 9.26, 8.84 [13:50:24] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.18, 1.85, 1.51 [13:50:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.44, 8.73, 8.53 [13:50:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.14, 8.62, 7.72 [13:52:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.34, 8.61, 8.06 [13:52:24] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.70, 1.45, 1.41 [13:52:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.60, 9.44, 8.81 [13:52:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.36, 8.99, 7.98 [13:53:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.67, 10.08, 9.29 [13:54:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.31, 9.04, 8.27 [13:54:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.74, 8.84, 8.67 [13:55:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.30, 9.64, 9.21 [13:56:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.36, 8.65, 8.22 [13:56:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.66, 9.78, 9.05 [13:57:24] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.16, 1.88, 1.55 [13:57:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.41, 10.01, 9.40 [13:58:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.48, 9.65, 8.53 [13:59:18] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.77, 1.77, 1.55 [13:59:24] RECOVERY - wiki.slimelab.net - reverse DNS on sslhost is OK: SSL OK - wiki.slimelab.net reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:59:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.74, 9.81, 9.39 [14:00:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.05, 8.88, 8.38 [14:00:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.82, 9.70, 9.21 [14:00:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.06, 8.81, 8.37 [14:00:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 3.37, 2.67 [14:01:12] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.02, 1.51, 1.47 [14:02:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.55, 8.86, 8.44 [14:02:18] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.13, 9.64, 9.23 [14:02:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.35, 7.99, 8.12 [14:02:57] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.09, 3.08, 2.64 [14:03:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.09, 9.22, 9.19 [14:03:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.52, 9.65, 9.43 [14:04:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.96, 9.14, 9.10 [14:04:36] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:05:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.02, 9.23, 9.21 [14:06:00] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.84, 8.23, 8.28 [14:06:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.72, 8.74, 8.04 [14:06:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 3.48, 2.94 [14:07:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.97, 10.00, 9.50 [14:07:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.37, 10.12, 9.63 [14:08:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.22, 9.36, 9.18 [14:08:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.76, 3.98, 3.16 [14:10:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.94, 10.45, 9.15 [14:12:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.50, 9.51, 8.95 [14:12:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.02, 9.51, 9.30 [14:12:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.65, 3.53, 3.20 [14:13:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.20, 9.95, 9.67 [14:13:51] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.88, 8.24, 8.10 [14:14:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.60, 10.03, 9.22 [14:14:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.41, 3.11, 3.08 [14:15:32] [02puppet] 07paladox closed pull request 03#2429: test: enable jobqueue - 13https://github.com/miraheze/puppet/pull/2429 [14:15:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/d28ac1379853...5be1bc8a9c40 [14:15:35] [02miraheze/puppet] 07RhinosF1 035be1bc8 - test: enable jobqueue (#2429) [14:15:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.23, 9.72, 9.39 [14:16:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.41, 9.87, 9.27 [14:17:25] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [14:17:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.32, 9.36, 9.32 [14:18:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.91, 9.93, 9.34 [14:20:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.50, 8.82, 8.33 [14:22:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.51, 9.56, 9.37 [14:23:14] [02mw-config] 07RhinosF1 closed pull request 03#4482: Beta: Send Jobqueue to test101 - 13https://github.com/miraheze/mw-config/pull/4482 [14:23:16] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 - 13https://github.com/miraheze/mw-config [14:23:17] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 [14:23:19] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/95f4cde0aa13...93d759472bf0 [14:23:20] [02miraheze/mw-config] 07RhinosF1 0393d7594 - Beta: Send Jobqueue to test101 (#4482) [14:23:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.69, 8.59, 8.30 [14:24:25] miraheze/mw-config - RhinosF1 the build passed. [14:24:31] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet is currently disabled, message: Rolling out jobrunner, last run 2 minutes ago with 0 failures [14:25:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.05, 9.57, 9.55 [14:26:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.72, 10.32, 9.64 [14:26:37] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.98, 8.12, 8.24 [14:27:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.55, 9.31, 9.26 [14:28:37] !log [@test101] starting deploy of {'config': True} to all [14:28:38] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [14:28:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:30:07] !log [rhinos@mwtask111] starting deploy of {'config': True} to mwtask111 [14:30:08] !log [rhinos@mwtask111] finished deploy of {'config': True} to mwtask111 - SUCCESS in 0s [14:30:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:30:27] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.76, 8.49, 8.44 [14:30:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:31:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.84, 9.52, 9.39 [14:32:15] !log [rhinos@mwtask111] starting deploy of {'config': True} to all [14:32:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:32:27] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:32:28] !log [rhinos@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 13s [14:32:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.66, 8.73, 8.51 [14:32:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:32:50] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:34:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.78, 9.21, 8.69 [14:34:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.41, 9.65, 9.63 [14:34:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.80, 4.09, 3.35 [14:36:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.31, 9.09, 8.73 [14:36:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.86, 3.37 [14:36:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.70, 10.15, 9.58 [14:38:38] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.06, 9.65, 9.61 [14:40:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.00, 9.97, 9.75 [14:40:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.50, 3.28, 3.25 [14:42:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.93, 10.45, 9.94 [14:46:09] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.89, 7.93, 8.47 [14:46:09] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [14:46:09] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4058167934417725 seconds [14:48:26] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.44, 8.89, 9.69 [14:50:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.61, 9.47, 8.95 [14:50:30] [02puppet] 07RhinosF1 opened pull request 03#2430: Update site.pp - 13https://github.com/miraheze/puppet/pull/2430 [14:51:06] paladox: can you merge ^ [14:51:21] [02puppet] 07paladox closed pull request 03#2430: Update site.pp - 13https://github.com/miraheze/puppet/pull/2430 [14:51:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5be1bc8a9c40...2f6523697961 [14:51:24] [02miraheze/puppet] 07RhinosF1 032f65236 - Update site.pp (#2430) [14:51:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.11, 8.44, 7.91 [14:52:06] paladox: if you can pull puppet, that would be super [14:52:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.92, 8.92, 8.82 [14:52:15] done [14:52:17] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.58, 9.80, 9.83 [14:53:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.20, 8.24, 7.92 [14:54:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.17, 9.29, 9.73 [14:54:44] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:56:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.02, 9.55, 9.72 [14:56:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.73, 10.44, 10.12 [14:57:55] [02puppet] 07RhinosF1 opened pull request 03#2431: Update chron.pp - 13https://github.com/miraheze/puppet/pull/2431 [14:58:04] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.29, 9.63, 9.72 [14:58:28] [02puppet] 07RhinosF1 synchronize pull request 03#2431: Update chron.pp - 13https://github.com/miraheze/puppet/pull/2431 [14:59:59] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.54, 9.79, 9.79 [15:00:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.56, 9.24, 8.95 [15:00:28] [02puppet] 07RhinosF1 synchronize pull request 03#2431: Update chron.pp - 13https://github.com/miraheze/puppet/pull/2431 [15:01:02] [02puppet] 07RhinosF1 edited pull request 03#2431: Allow being chron and runner - 13https://github.com/miraheze/puppet/pull/2431 [15:01:07] paladox: ^ [15:01:24] [02puppet] 07paladox closed pull request 03#2431: Allow being chron and runner - 13https://github.com/miraheze/puppet/pull/2431 [15:01:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/puppet/compare/2f6523697961...05414e096084 [15:01:27] [02miraheze/puppet] 07RhinosF1 0305414e0 - Allow being chron and runner (#2431) [15:01:34] paladox: please pull [15:01:40] i have [15:02:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.67, 9.67, 9.16 [15:03:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.80, 9.38, 9.90 [15:04:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.03, 9.63, 9.99 [15:04:44] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:11] [02puppet] 07RhinosF1 opened pull request 03#2432: runner: fix missing variable - 13https://github.com/miraheze/puppet/pull/2432 [15:05:19] paladox: final PR ^ [15:05:51] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet is currently disabled, message: RhinosF1, last run 12 minutes ago with 0 failures [15:06:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.00, 9.85, 10.02 [15:06:48] [02puppet] 07paladox closed pull request 03#2432: runner: fix missing variable - 13https://github.com/miraheze/puppet/pull/2432 [15:06:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/05414e096084...585d91098865 [15:06:51] [02miraheze/puppet] 07RhinosF1 03585d910 - runner: fix missing variable (#2432) [15:07:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.96, 9.40, 9.58 [15:08:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.63, 3.96, 3.34 [15:08:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.17, 8.75, 9.79 [15:09:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.44, 9.01, 9.42 [15:09:54] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.48, 2.16, 1.04 [15:09:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.41, 3.66, 3.30 [15:11:10] paladox: ty for help [15:11:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.45, 9.66, 9.78 [15:11:32] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.55, 9.44, 9.51 [15:11:52] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:11:54] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.78, 3.35, 3.23 [15:13:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.19, 9.17, 9.60 [15:13:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.13, 9.41, 9.51 [15:13:52] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.21, 1.73, 1.13 [15:14:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.23, 9.60, 9.83 [15:15:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.57, 10.49, 9.88 [15:15:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 3.89, 3.46 [15:15:49] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [15:15:51] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.54, 1.68, 1.18 [15:16:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.78, 9.42, 9.71 [15:17:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.20, 9.35, 9.94 [15:18:19] PROBLEM - jobchron121 Puppet on jobchron121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:18:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.12, 9.92, 9.88 [15:19:00] paladox: can you see why jobchron puppet failing [15:19:08] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.13, 9.16, 9.35 [15:19:11] i don't have time currently [15:19:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.72, 9.74, 9.82 [15:19:27] i merged the changes on the understanding you were going to run puppet? [15:19:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.78, 3.53 [15:19:46] paladox: mw team don't have access to jobchron [15:19:47] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.10, 2.04, 1.45 [15:20:07] oh ok [15:20:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.73, 9.18, 9.62 [15:20:45] it'll be something in https://github.com/miraheze/puppet/compare/2f6523697961...05414e096084 [15:20:45] [url] Comparing 2f6523697961...05414e096084 · miraheze/puppet · GitHub | github.com [15:21:44] RhinosF1: [15:21:45] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find resource 'Service[jobrunner]' in parameter 'notify' (file: /etc/puppetlabs/puppet/environments/production/modules/mediawiki/manifests/jobqueue/shared.pp, line: 24) on node jobchron121.miraheze.org [15:21:47] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.24, 1.73, 1.41 [15:22:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.93, 9.59, 9.01 [15:22:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.88, 9.55, 9.67 [15:22:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/5ca635e68df5 [15:22:33] [02miraheze/puppet] 07paladox 035ca635e - Grant mw team access to jobchron121 [15:22:35] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [15:22:36] [02puppet] 07paladox opened pull request 03#2433: Grant mw team access to jobchron121 - 13https://github.com/miraheze/puppet/pull/2433 [15:23:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.70, 9.45, 9.45 [15:23:09] i can fix [15:23:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.42, 3.05, 3.29 [15:23:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.86, 9.48, 9.68 [15:23:47] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.12, 1.50, 1.36 [15:23:50] [02puppet] 07RhinosF1 opened pull request 03#2434: Update shared.pp - 13https://github.com/miraheze/puppet/pull/2434 [15:24:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.34, 9.09, 8.90 [15:24:31] [02puppet] 07RhinosF1 synchronize pull request 03#2434: Update shared.pp - 13https://github.com/miraheze/puppet/pull/2434 [15:24:52] [02puppet] 07RhinosF1 synchronize pull request 03#2434: Update shared.pp - 13https://github.com/miraheze/puppet/pull/2434 [15:25:16] [02puppet] 07RhinosF1 edited pull request 03#2434: Use subscribe not notify for File['/srv/jobrunner/jobrunner.json'] - 13https://github.com/miraheze/puppet/pull/2434 [15:25:25] paladox: ^ [15:25:38] [02puppet] 07paladox closed pull request 03#2434: Use subscribe not notify for File['/srv/jobrunner/jobrunner.json'] - 13https://github.com/miraheze/puppet/pull/2434 [15:25:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/585d91098865...5d048c107512 [15:25:42] [02miraheze/puppet] 07RhinosF1 035d048c1 - Use subscribe not notify for File['/srv/jobrunner/jobrunner.json'] (#2434) [15:25:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.61, 8.66, 7.95 [15:26:19] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [15:26:24] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.72, 1.66, 1.43 [15:26:41] [02puppet] 07RhinosF1 opened pull request 03#2435: RecordLintJob: move to intensive - 13https://github.com/miraheze/puppet/pull/2435 [15:27:42] [02puppet] 07RhinosF1 synchronize pull request 03#2435: RecordLintJob: move to intensive - 13https://github.com/miraheze/puppet/pull/2435 [15:27:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.70, 9.67, 8.40 [15:28:04] [02puppet] 07RhinosF1 synchronize pull request 03#2435: RecordLintJob: move to intensive - 13https://github.com/miraheze/puppet/pull/2435 [15:29:48] paladox: thanks for your help [15:30:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.85, 8.06, 8.45 [15:30:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.64, 9.32, 9.31 [15:30:56] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.73, 1.21, 1.30 [15:31:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.47, 8.57, 8.20 [15:31:15] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.56, 10.51, 10.03 [15:32:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.43, 8.92, 9.17 [15:32:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.52, 9.83, 9.91 [15:34:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.47, 9.41, 8.59 [15:36:16] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [15:36:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.14, 9.21, 9.74 [15:36:51] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.17, 1.79, 1.48 [15:36:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.46, 8.75, 8.46 [15:38:26] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.58, 7.69, 8.50 [15:38:49] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.37, 1.65, 1.47 [15:40:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.20, 8.36, 8.37 [15:42:23] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.99, 9.27, 8.94 [15:42:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.46, 9.32, 9.56 [15:44:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.11, 8.11, 8.56 [15:44:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.29, 8.82, 9.34 [15:45:32] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [15:45:32] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407124280929565 seconds [15:46:25] [02puppet] 07JohnFLewis commented on pull request 03#2433: Grant mw team access to jobchron121 - 13https://github.com/miraheze/puppet/pull/2433#issuecomment-1049992855 [15:48:19] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.37, 8.01, 8.45 [15:48:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.76, 9.68, 9.55 [15:48:38] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.23, 1.78, 1.16 [15:50:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.20, 9.05, 9.34 [15:50:36] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.55, 1.67, 1.19 [15:52:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.54, 8.43, 8.57 [15:52:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.99, 9.58, 9.48 [15:52:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.56, 8.44, 8.18 [15:54:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.56, 8.02, 8.05 [15:56:16] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.66, 7.91, 8.34 [15:57:30] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 3.87, 3.26 [15:59:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.37, 3.38, 3.16 [16:00:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.42, 9.96, 9.09 [16:02:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.29, 9.28, 9.01 [16:02:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.00, 9.27, 8.95 [16:03:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.92, 3.54, 3.31 [16:05:39] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [16:06:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.89, 9.67, 9.29 [16:07:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.71, 3.19, 3.22 [16:08:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.14, 9.49, 9.83 [16:10:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.38, 9.77, 9.41 [16:10:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.15, 9.81, 9.91 [16:12:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.12, 9.73, 9.88 [16:13:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.43, 3.33 [16:14:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.84, 10.44, 10.11 [16:14:24] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [16:14:31] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.03, 1.91, 1.37 [16:15:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.86, 2.95, 3.17 [16:16:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.60, 9.80, 9.65 [16:16:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.03, 1.69, 1.36 [16:20:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.74, 9.70, 9.27 [16:20:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.00, 9.57, 9.86 [16:22:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.20, 9.31, 9.17 [16:22:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.03, 9.07, 9.98 [16:24:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.30, 9.70, 10.11 [16:26:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.11, 10.61, 9.68 [16:28:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.23, 9.54, 9.62 [16:29:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.67, 8.46, 7.97 [16:30:08] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.14, 7.19, 8.33 [16:30:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.50, 9.25, 9.91 [16:30:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.13, 9.35, 9.57 [16:31:51] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.52, 8.00, 7.86 [16:32:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.32, 9.89, 10.05 [16:34:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.34, 8.98, 8.81 [16:34:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.52, 9.59, 9.93 [16:34:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.55, 10.17, 9.78 [16:36:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.45, 8.88, 8.78 [16:36:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.36, 10.21, 10.11 [16:38:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.41, 9.36, 8.97 [16:39:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.70, 9.09, 8.44 [16:41:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.30, 8.54, 7.83 [16:42:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.29, 9.46, 9.82 [16:43:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.04, 8.26, 7.83 [16:43:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.82, 9.66, 8.81 [16:44:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.88, 10.50, 10.16 [16:45:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.86, 3.80, 3.14 [16:45:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.02, 8.92, 8.64 [16:47:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 4.23, 3.38 [16:49:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.90, 9.08, 8.28 [16:49:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.04, 10.17, 9.16 [16:50:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.56, 8.95, 9.81 [16:51:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.20, 9.55, 8.55 [16:52:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.19, 9.54, 9.93 [16:53:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.62, 9.26, 8.56 [16:53:52] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.96, 1.47 [16:54:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.50, 9.32, 9.91 [16:55:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.73, 3.57 [16:56:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.91, 9.67, 9.94 [16:57:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.40, 9.89, 9.57 [16:57:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.97, 1.63, 1.45 [16:58:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.46, 9.83, 9.97 [16:58:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.23, 9.83, 9.96 [16:58:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.12, 8.12, 8.29 [16:59:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.62, 3.53 [17:00:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.90, 9.95, 9.90 [17:00:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.77, 8.96, 9.65 [17:02:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.27, 10.14, 9.99 [17:02:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.76, 9.54, 9.77 [17:03:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.49, 9.31, 9.27 [17:04:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.82, 8.88, 9.54 [17:04:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.32, 9.51, 9.74 [17:05:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.52, 9.08, 9.20 [17:08:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.34, 8.89, 9.81 [17:08:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.48, 9.35, 9.87 [17:10:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.06, 9.56, 9.53 [17:10:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.63, 9.17, 9.45 [17:10:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.26, 9.11, 8.54 [17:12:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.16, 9.22, 9.45 [17:12:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.77, 8.67, 8.45 [17:13:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.46, 3.74, 3.94 [17:14:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.19, 9.80, 9.80 [17:14:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.16, 9.90, 9.78 [17:14:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.12, 9.87, 9.66 [17:14:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:14:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.52, 8.38, 8.38 [17:16:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.28, 9.91, 9.82 [17:17:51] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.33, 7.98, 8.50 [17:18:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.05, 9.60, 9.64 [17:20:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.57, 9.85, 9.83 [17:20:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.24, 10.11, 9.81 [17:20:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.71, 9.16, 9.60 [17:21:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.42, 8.93, 8.84 [17:22:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.51, 9.11, 9.40 [17:23:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.02, 2.83, 3.36 [17:23:51] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.46, 7.94, 8.49 [17:24:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.20, 9.38, 9.57 [17:25:59] [discord] Was this resolved? It's still happening on our wiki [17:26:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.02, 9.29, 9.46 [17:26:13] [discord] *was still, as of a couple days ago [17:26:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.61, 9.40, 9.73 [17:27:07] [discord] Plus we get an exception in preview with cargo, details of how to reproduce sent to Cosmic Alpha via pm. [17:28:49] [discord] But if it's better to create a phab ticket, I can do that. It's a private wiki though. [17:30:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.50, 9.76, 9.76 [17:32:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.64, 9.27, 9.34 [17:34:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.76, 8.80, 9.17 [17:34:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.53, 8.90, 9.43 [17:38:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.97, 9.91, 9.61 [17:40:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.39, 10.00, 9.67 [17:42:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.61, 9.89, 9.70 [17:42:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.54, 9.90, 9.69 [17:42:23] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [17:42:23] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407085657119751 seconds [17:44:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.77, 10.15, 9.79 [17:46:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.54, 9.95, 9.77 [17:47:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.14, 3.42, 3.21 [17:48:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.93, 9.61, 9.57 [17:50:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.26, 8.30, 7.88 [17:50:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.66, 9.32, 9.95 [17:51:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.82, 3.41 [17:52:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.59, 8.05, 8.41 [17:52:15] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.07, 10.00, 9.83 [17:52:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.36, 8.84, 8.11 [17:52:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.79, 9.68, 10.02 [17:53:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.53, 3.21, 3.23 [17:54:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.95, 9.10, 8.30 [17:56:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.14, 9.69, 9.04 [17:56:13] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.46, 8.42, 8.14 [17:58:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.70, 9.35, 8.99 [17:58:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.32, 8.98, 9.50 [18:00:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.81, 9.82, 9.20 [18:00:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.95, 9.84, 9.99 [18:00:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.90, 9.55, 9.95 [18:02:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.00, 9.53, 9.20 [18:02:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.38, 9.58, 9.60 [18:02:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.10, 9.40, 9.82 [18:04:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.39, 9.97, 9.37 [18:04:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.35, 9.17, 9.43 [18:06:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.87, 9.50, 9.68 [18:07:52] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.30, 9.13, 8.47 [18:08:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.75, 8.81, 9.41 [18:08:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.69, 9.56, 9.51 [18:09:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.59, 8.52, 8.32 [18:10:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.51, 9.81, 9.55 [18:10:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.37, 9.34, 9.52 [18:12:20] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [18:14:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.47, 9.22, 9.45 [18:14:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.26, 3.13 [18:14:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.91, 9.29, 9.57 [18:14:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.25, 8.88, 9.69 [18:15:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.13, 9.42, 8.78 [18:16:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.80, 9.28, 9.32 [18:16:05] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.09, 3.36, 3.20 [18:18:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.97, 10.01, 9.69 [18:19:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.66, 9.83, 9.10 [18:20:04] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/1ef107a4200b...e2bb80658f5f [18:20:06] [02miraheze/CreateWiki] 07Universal-Omega 03e2bb806 - Update WikiManagerTest.php [18:20:07] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [18:20:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.51, 9.89, 9.70 [18:22:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.04, 9.34, 9.41 [18:22:20] PROBLEM - wiki.landev.vn - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.landev.vn All nameservers failed to answer the query. [18:22:23] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e2bb80658f5f...00a8f4c89ee5 [18:22:25] [02miraheze/CreateWiki] 07Universal-Omega 0300a8f4c - Update WikiManager.php [18:22:26] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [18:23:12] miraheze/CreateWiki - Universal-Omega the build has errored. [18:25:23] miraheze/CreateWiki - Universal-Omega the build has errored. [18:26:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.49, 9.14, 9.32 [18:28:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.24, 9.56, 9.45 [18:28:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.01, 9.69, 9.66 [18:28:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.29, 9.02, 9.27 [18:30:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.34, 10.05, 9.79 [18:32:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.19, 9.82, 9.62 [18:32:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.94, 9.62, 9.67 [18:34:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.70, 9.28, 9.16 [18:35:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.19, 9.52, 8.94 [18:36:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.51, 3.31 [18:36:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.23, 10.30, 9.92 [18:36:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.68, 9.26, 9.16 [18:36:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [18:36:25] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:38:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.09, 2.94, 3.12 [18:38:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.89, 9.61, 9.29 [18:39:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.85, 9.05, 8.91 [18:40:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.46, 10.36, 9.89 [18:40:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.69, 9.83, 9.43 [18:40:27] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.59, 2.19, 1.23 [18:42:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.13, 10.55, 9.73 [18:42:23] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [18:42:23] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.405237674713135 seconds [18:42:27] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.14, 1.80, 1.20 [18:43:30] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/00a8f4c89ee5...d6f10f31ae09 [18:43:31] [02miraheze/CreateWiki] 07Universal-Omega 03d6f10f3 - Update WikiManagerTest.php [18:43:33] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [18:44:27] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.79, 1.46, 1.15 [18:45:24] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f9e1e213f519...75877f6ba77b [18:45:25] [02miraheze/ssl] 07MacFan4000 0375877f6 - update www.puzzles.wiki cert to include root domain [18:46:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.91, 8.95, 9.54 [18:46:47] miraheze/CreateWiki - Universal-Omega the build has errored. [18:47:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.02, 9.61, 9.12 [18:48:55] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.16, 1.79, 1.40 [18:49:06] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/d6f10f31ae09...010e2b84774b [18:49:08] [02miraheze/CreateWiki] 07Universal-Omega 03010e2b8 - Update WikiManagerTest.php [18:49:09] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [18:50:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.09, 9.71, 9.74 [18:50:52] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.36, 1.63, 1.39 [18:51:25] PROBLEM - wiki.landev.vn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.landev.vn could not be found [18:51:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.93, 9.94, 9.43 [18:52:41] miraheze/CreateWiki - Universal-Omega the build has errored. [18:53:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.61, 4.22, 3.50 [18:54:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.19, 9.09, 9.86 [18:54:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.08, 10.50, 10.01 [18:56:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.77, 9.75, 10.00 [18:56:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.64, 9.50, 9.44 [18:56:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.42, 9.92, 9.87 [18:58:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.35, 9.78, 10.00 [18:58:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.95, 9.34, 9.38 [18:59:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.62, 3.44, 3.50 [19:00:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.39, 10.21, 10.14 [19:00:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.59, 9.77, 9.50 [19:01:51] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.14, 7.31, 8.40 [19:04:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.35, 9.66, 9.56 [19:04:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.76, 10.61, 10.03 [19:05:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.36, 1.92, 1.50 [19:05:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.72, 3.56 [19:05:47] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [19:06:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.45, 8.74, 8.30 [19:07:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.26, 1.62, 1.43 [19:08:12] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.91, 8.31, 8.19 [19:10:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.24, 9.27, 9.86 [19:12:20] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [19:12:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.87, 9.39, 9.78 [19:14:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.07, 9.52, 9.82 [19:14:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.59, 9.76, 9.85 [19:15:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.69, 3.67 [19:16:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.37, 9.09, 9.62 [19:16:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.76, 9.31, 9.67 [19:17:10] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/010e2b84774b...186daf45a0d0 [19:17:12] [02miraheze/CreateWiki] 07Universal-Omega 03186daf4 - Update WikiManagerTest.php [19:17:13] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [19:17:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.98, 8.50, 8.25 [19:18:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.66, 10.68, 10.13 [19:19:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.80, 4.36, 3.90 [19:19:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.96, 8.25, 8.18 [19:20:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 3.23, 2.63 [19:20:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.94, 9.28, 9.15 [19:20:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.78, 10.26, 9.94 [19:20:31] miraheze/CreateWiki - Universal-Omega the build has errored. [19:21:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.81, 3.75 [19:22:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.74, 3.31, 2.73 [19:22:17] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/186daf45a0d0...349470751f1b [19:22:18] [02miraheze/CreateWiki] 07Universal-Omega 033494707 - Update WikiManagerTest.php [19:22:20] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [19:23:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.19, 8.99, 8.48 [19:23:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.55, 2.04, 1.63 [19:24:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.38, 2.93, 2.66 [19:25:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.91, 4.34, 3.93 [19:25:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.73, 9.70, 8.83 [19:25:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.91, 3.62, 3.06 [19:25:42] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.16, 1.79, 1.59 [19:26:02] miraheze/CreateWiki - Universal-Omega the build has errored. [19:26:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.54, 9.93, 9.54 [19:26:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.56, 9.79, 9.90 [19:27:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.56, 9.78, 8.94 [19:27:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.84, 3.89, 3.21 [19:27:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.64, 3.56, 3.69 [19:27:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.06, 1.50, 1.50 [19:28:02] [02puppet] 07Reception123 closed pull request 03#2435: RecordLintJob: move to intensive - 13https://github.com/miraheze/puppet/pull/2435 [19:28:04] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/5d048c107512...5c8842145294 [19:28:05] [02miraheze/puppet] 07RhinosF1 035c88421 - RecordLintJob: move to intensive (#2435) [19:29:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.81, 9.87, 9.08 [19:29:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 3.63, 3.19 [19:31:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 3.87, 3.77 [19:32:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.94, 9.98, 9.88 [19:33:01] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/349470751f1b...ad79e79589ba [19:33:02] [02miraheze/CreateWiki] 07Universal-Omega 03ad79e79 - Update WikiManagerTest.php [19:33:04] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [19:33:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.57, 3.61, 3.71 [19:36:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.92, 8.88, 9.90 [19:36:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.78, 8.96, 9.91 [19:36:17] miraheze/CreateWiki - Universal-Omega the build has errored. [19:36:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.54, 7.90, 8.42 [19:37:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.08, 2.94, 3.08 [19:38:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.68, 9.11, 9.67 [19:40:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.18, 9.35, 9.83 [19:40:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.26, 8.70, 8.98 [19:40:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.03, 9.84, 10.02 [19:41:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.33, 2.95, 3.37 [19:42:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.75, 9.36, 9.21 [19:42:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.23, 9.95, 9.89 [19:42:23] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [19:42:23] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406286954879761 seconds [19:44:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.65, 9.43, 9.74 [19:44:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.76, 9.90, 9.43 [19:44:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.83, 10.00, 9.92 [19:46:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.64, 10.31, 10.03 [19:48:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.06, 10.31, 10.05 [19:49:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.96, 3.72, 3.50 [19:50:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.45, 9.67, 9.85 [19:50:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.95, 9.14, 9.30 [19:50:09] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ad79e79589ba...058143ff69d5 [19:50:11] [02miraheze/CreateWiki] 07Universal-Omega 03058143f - Update WikiManagerTest.php [19:50:12] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [19:50:27] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/058143ff69d5...58df97c66c77 [19:50:28] [02miraheze/CreateWiki] 07Universal-Omega 0358df97c - Update WikiManagerTest.php [19:50:30] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [19:51:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 4.09, 3.49 [19:51:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.91, 3.60 [19:53:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.67, 3.40 [19:53:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.47, 3.33, 3.42 [19:53:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.85, 8.86, 8.14 [19:54:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.92, 10.25, 10.05 [19:54:11] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/58df97c66c77...fa1d342a9a16 [19:54:12] [02miraheze/CreateWiki] 07Universal-Omega 03fa1d342 - Update WikiManagerTest.php [19:54:14] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [19:54:14] miraheze/CreateWiki - Universal-Omega the build has errored. [19:55:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.57, 3.15, 3.34 [19:57:51] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.29, 8.06, 7.98 [19:58:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.51, 9.95, 9.42 [19:58:08] miraheze/CreateWiki - Universal-Omega the build has errored. [20:00:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.95, 9.70, 9.40 [20:00:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.72, 3.14, 3.32 [20:02:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.06, 9.55, 9.93 [20:02:44] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/fa1d342a9a16...20aebfe3ca79 [20:02:45] [02miraheze/CreateWiki] 07Universal-Omega 0320aebfe - Update WikiManagerTest.php [20:02:47] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [20:04:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.77, 10.33, 9.68 [20:04:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.34, 9.28, 9.84 [20:04:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.61, 10.42, 10.18 [20:06:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.10, 9.76, 9.57 [20:06:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.95, 9.35, 9.82 [20:06:28] miraheze/CreateWiki - Universal-Omega the build has errored. [20:07:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.75, 8.57, 8.23 [20:09:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.70, 9.27, 8.52 [20:10:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.02, 9.03, 9.76 [20:10:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.54, 10.07, 9.71 [20:10:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.20, 10.41, 10.06 [20:11:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.94, 8.78, 8.41 [20:12:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.51, 9.78, 9.65 [20:12:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.09, 8.93, 9.41 [20:12:22] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [20:14:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.81, 10.16, 9.81 [20:16:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.71, 9.58, 9.85 [20:16:22] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/20aebfe3ca79...09a5d23a7ccc [20:16:24] [02miraheze/CreateWiki] 07Universal-Omega 0309a5d23 - Update WikiManagerTest.php [20:16:25] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [20:19:49] miraheze/CreateWiki - Universal-Omega the build has errored. [20:19:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.32, 9.55, 8.83 [20:20:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.33, 9.82, 9.84 [20:20:21] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/09a5d23a7ccc...b3f8e07a22da [20:20:22] [02miraheze/CreateWiki] 07Universal-Omega 03b3f8e07 - Update WikiManagerTest.php [20:20:24] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [20:22:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.11, 10.73, 10.17 [20:24:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.85, 9.73, 9.52 [20:24:07] miraheze/CreateWiki - Universal-Omega the build has errored. [20:24:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.92, 10.00, 9.97 [20:24:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.53, 9.39, 9.50 [20:24:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.67, 9.63, 9.85 [20:26:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.45, 9.53, 9.50 [20:28:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.60, 9.93, 9.87 [20:28:19] [02puppet] 07RhinosF1 commented on pull request 03#2433: Grant mw team access to jobchron121 - 13https://github.com/miraheze/puppet/pull/2433#issuecomment-1050234824 [20:28:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.20, 9.72, 9.74 [20:30:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.88, 10.11, 9.70 [20:30:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.47, 9.78, 9.83 [20:31:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.02, 9.65, 9.48 [20:32:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.43, 9.93, 9.87 [20:34:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.03, 9.56, 9.75 [20:34:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.44, 3.00 [20:36:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.33, 9.18, 9.57 [20:36:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.91, 10.31, 10.02 [20:36:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.45, 4.19, 3.31 [20:38:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.57, 9.65, 9.69 [20:38:12] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.28, 9.28, 8.38 [20:38:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.37, 3.83, 3.29 [20:40:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.58, 9.08, 9.49 [20:40:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.47, 8.67, 8.26 [20:42:00] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.84, 8.12, 8.12 [20:42:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 4.05, 3.49 [20:43:25] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/b3f8e07a22da...4373cf9d57eb [20:43:27] [02miraheze/CreateWiki] 07Universal-Omega 034373cf9 - Update WikiManager.php [20:43:28] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [20:44:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.46, 3.56, 3.38 [20:45:11] RECOVERY - wiki.landev.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.landev.vn' will expire on Sat 02 Apr 2022 03:10:10 GMT +0000. [20:46:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.33, 9.28, 9.35 [20:46:40] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.97, 1.95, 1.42 [20:46:49] miraheze/CreateWiki - Universal-Omega the build has errored. [20:48:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.23, 3.07, 3.24 [20:48:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.69, 2.18, 1.57 [20:49:08] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/4373cf9d57eb...3b798bfbb918 [20:49:10] [02miraheze/CreateWiki] 07Universal-Omega 033b798bf - Update WikiManager.php [20:49:11] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [20:49:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.59, 9.39, 8.71 [20:50:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.57, 1.96, 1.56 [20:52:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.73, 8.65, 9.13 [20:52:17] miraheze/CreateWiki - Universal-Omega the build has errored. [20:52:18] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/3b798bfbb918...9c2c8f27cd1e [20:52:20] [02miraheze/CreateWiki] 07Universal-Omega 039c2c8f2 - Update globals.php [20:52:21] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [20:52:33] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 7.13, 3.42, 2.10 [20:53:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.00, 9.72, 8.93 [20:55:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.98, 9.30, 8.87 [20:56:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.70, 9.30, 9.23 [20:56:56] miraheze/CreateWiki - Universal-Omega the build passed. [20:57:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 10.49, 9.36 [20:58:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.55, 8.49, 8.93 [20:58:27] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.95, 1.91, 1.84 [20:59:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.52, 9.83, 9.26 [20:59:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.03, 9.48, 9.21 [21:01:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.12, 9.42, 9.23 [21:02:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.10, 9.48, 9.22 [21:02:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.10, 9.17, 9.95 [21:02:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.21, 1.42, 1.64 [21:04:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.79, 8.65, 8.97 [21:04:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.08, 9.96, 10.12 [21:06:58] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/7b8f73159c1e [21:07:00] [02miraheze/ssl] 07MacFan4000 037b8f731 - Create commonwealth.0um.io.crt [21:07:01] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [21:07:10] [02ssl] 07MacFan4000 opened pull request 03#482: Add commonwealth.0um.io cert - 13https://github.com/miraheze/ssl/pull/482 [21:07:31] [02ssl] 07MacFan4000 edited pull request 03#482: Add commonwealth.0um.io cert (T8822) - 13https://github.com/miraheze/ssl/pull/482 [21:08:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.75, 9.47, 9.19 [21:08:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.73, 9.27, 9.87 [21:08:59] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7b8f73159c1e...cd3e59d1dc2c [21:09:00] [02miraheze/ssl] 07MacFan4000 03cd3e59d - Update certs.yaml [21:09:02] [02ssl] 07MacFan4000 synchronize pull request 03#482: Add commonwealth.0um.io cert (T8822) - 13https://github.com/miraheze/ssl/pull/482 [21:09:40] [02ssl] 07MacFan4000 closed pull request 03#482: Add commonwealth.0um.io cert (T8822) - 13https://github.com/miraheze/ssl/pull/482 [21:09:41] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/75877f6ba77b...54f0b9aecac4 [21:09:43] [02miraheze/ssl] 07MacFan4000 0354f0b9a - Add commonwealth.0um.io cert (T8822) (#482) [21:09:44] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [21:09:46] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [21:12:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.79, 9.37, 9.26 [21:12:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.48, 10.02, 10.05 [21:12:23] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [21:12:23] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406708240509033 seconds [21:12:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.76, 1.83, 1.45 [21:14:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:14:42] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.01, 1.57, 1.41 [21:16:42] PROBLEM - wiki.landev.vn - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.landev.vn and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [21:16:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.08, 9.53, 9.04 [21:18:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.13, 8.67, 8.79 [21:19:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.11, 9.18, 8.86 [21:21:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.54, 9.16, 8.91 [21:22:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.01, 9.27, 9.86 [21:22:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.81, 9.50, 9.93 [21:26:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.13, 9.61, 9.21 [21:26:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.56, 8.09, 8.45 [21:28:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 3.53, 3.14 [21:28:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.49, 9.25, 9.13 [21:30:38] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.59, 9.80, 9.32 [21:30:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.31, 9.34, 9.60 [21:31:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.41, 3.23, 3.10 [21:32:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.85, 9.27, 9.53 [21:35:04] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/9c2c8f27cd1e...a4da63b4a294 [21:35:06] [02miraheze/CreateWiki] 07Universal-Omega 03a4da63b - Update WikiManagerTest.php [21:35:07] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [21:36:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.27, 9.60, 9.45 [21:36:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.39, 9.80, 9.68 [21:37:10] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.12, 9.41, 9.35 [21:38:27] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.18, 7.99, 8.41 [21:38:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.35, 9.28, 9.51 [21:38:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.74, 8.54, 9.64 [21:39:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.50, 8.69, 9.10 [21:39:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:39:59] miraheze/CreateWiki - Universal-Omega the build passed. [21:40:23] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/a4da63b4a294...c656675509f1 [21:40:25] [02miraheze/CreateWiki] 07Universal-Omega 03c656675 - Add rename and delete tests [21:40:26] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [21:40:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [21:40:56] PROBLEM - cp30 SSH on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:41:14] Huh [21:41:16] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [21:41:17] PROBLEM - ping6 on cp30 is CRITICAL: PING CRITICAL - Packet loss = 100% [21:41:22] paladox: [21:41:26] !sre [21:41:35] cp30 down? [21:41:42] PROBLEM - cp30 HTTPS on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:42:15] PROBLEM - Host cp30 is DOWN: PING CRITICAL - Packet loss = 100% [21:42:20] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [21:43:28] !log reboot cp30 [21:43:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:43:41] miraheze/CreateWiki - Universal-Omega the build has errored. [21:45:37] seems to be an issue ovh end. [21:45:42] i can kvm into the vm [21:46:59] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.11, 9.06, 9.03 [21:48:15] paladox: can you file a ticket with them [21:48:33] already doing [21:48:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.36, 8.72, 8.90 [21:49:09] RECOVERY - Host cp30 is UP: PING OK - Packet loss = 0%, RTA = 78.44 ms [21:49:11] https://phabricator.miraheze.org/T8831 [21:49:11] [url] ⚓ T8831 cp30 down | phabricator.miraheze.org [21:49:14] RECOVERY - cp30 SSH on cp30 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [21:49:14] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: connect to address localhost and port 8102: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:49:14] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [21:49:14] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: connect to address localhost and port 8100: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:49:14] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: connect to address localhost and port 8150: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:49:14] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: connect to address localhost and port 8203: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:49:14] PROBLEM - cp30 Stunnel HTTP for phab121 on cp30 is CRITICAL: connect to address localhost and port 8202: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:49:15] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: connect to address localhost and port 8105: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:49:18] it's ipv4 address is back [21:49:19] PROBLEM - cp30 Stunnel HTTP for mon111 on cp30 is CRITICAL: connect to address localhost and port 8201: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:49:19] PROBLEM - cp30 PowerDNS Recursor on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: connect to address localhost and port 8104: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:49:19] RECOVERY - cp30 PowerDNS Recursor on cp30 is OK: DNS OK: 0.107 seconds response time. miraheze.org returns 149.56.141.75,2607:5300:201:3100::5ebc [21:49:29] PROBLEM - cp30 Stunnel HTTP for test101 on cp30 is CRITICAL: connect to address localhost and port 8180: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:49:39] RECOVERY - ping6 on cp30 is OK: PING OK - Packet loss = 0%, RTA = 78.45 ms [21:49:48] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.20, 2.23, 1.51 [21:49:59] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: connect to address localhost and port 8101: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:50:04] PROBLEM - cp30 Stunnel HTTP for mail121 on cp30 is CRITICAL: connect to address localhost and port 8200: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:50:08] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.48, 9.57, 9.25 [21:50:09] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:50:09] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: connect to address localhost and port 8103: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:50:17] paladox: ping6 came up too [21:50:21] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.366 second response time [21:50:33] RECOVERY - cp30 Stunnel HTTP for phab121 on cp30 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.331 second response time [21:50:36] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.324 second response time [21:50:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.54, 7.94, 7.57 [21:50:48] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.320 second response time [21:50:51] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.416 second response time [21:51:06] seems resolved [21:51:09] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.613 second response time [21:51:17] RECOVERY - cp30 Stunnel HTTP for mon111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 33717 bytes in 0.343 second response time [21:51:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.19, 9.06, 8.54 [21:51:19] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.321 second response time [21:51:27] RECOVERY - cp30 Stunnel HTTP for test101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.324 second response time [21:51:52] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.326 second response time [21:51:58] RECOVERY - cp30 Stunnel HTTP for mail121 on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.258 second response time [21:52:02] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [21:52:04] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.309 second response time [21:52:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.93, 9.42, 9.25 [21:52:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.93, 9.73, 9.45 [21:52:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:52:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.39, 7.86, 7.57 [21:52:52] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File [21:52:52] ib/python3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query franchise.franchising.org.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:53:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.17, 9.53, 8.78 [21:53:46] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.57, 1.90, 1.55 [21:54:02] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.84, 9.90, 9.44 [21:54:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:54:26] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [21:54:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.19, 9.49, 9.14 [21:54:42] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.61, 9.64, 9.25 [21:54:57] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:55:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.59, 9.37, 8.80 [21:55:39] RECOVERY - cp30 HTTPS on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3210 bytes in 0.407 second response time [21:55:45] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.03, 1.58, 1.48 [21:56:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.36, 9.55, 9.45 [21:56:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.23, 8.58, 7.92 [21:56:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.19, 9.17, 9.14 [21:57:58] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.79, 3.63, 2.85 [21:58:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.27, 2.90 [21:58:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.46, 9.98, 9.60 [21:58:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.83, 8.35, 7.92 [21:59:22] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/c656675509f1...646ef346c17c [21:59:23] [02miraheze/CreateWiki] 07Universal-Omega 03646ef34 - Update WikiManager.php [21:59:25] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [21:59:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.64, 2.96 [22:00:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.29, 3.53, 3.03 [22:01:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.50, 4.42, 3.59 [22:01:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.92, 3.15 [22:02:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.26, 3.59, 3.12 [22:02:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.28, 9.17, 8.37 [22:02:25] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.23, 10.40, 9.64 [22:02:50] miraheze/CreateWiki - Universal-Omega the build has errored. [22:03:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.16, 10.13, 9.38 [22:03:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.70, 3.15 [22:04:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.11, 3.40, 3.11 [22:05:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.23, 9.63, 9.28 [22:05:39] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.25, 3.18, 3.03 [22:05:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.27, 9.33, 9.60 [22:06:16] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.66, 9.88, 9.64 [22:06:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.68, 9.78, 8.77 [22:07:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [22:07:25] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:07:41] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.20, 9.67, 9.69 [22:08:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.04, 9.64, 9.84 [22:08:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 3.83, 3.33 [22:08:24] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [22:08:25] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/646ef346c17c...46e4d8529e6e [22:08:27] [02miraheze/CreateWiki] 07Universal-Omega 0346e4d85 - Update WikiManagerTest.php [22:09:03] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.29, 9.71, 9.37 [22:09:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.61, 4.25, 3.50 [22:09:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.17, 9.51, 9.63 [22:10:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.39, 9.01, 8.72 [22:11:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.40, 8.81, 9.07 [22:11:27] miraheze/CreateWiki - Universal-Omega the build has errored. [22:11:37] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/46e4d8529e6e...d154bb489a29 [22:11:38] [02miraheze/CreateWiki] 07Universal-Omega 03d154bb4 - Update WikiManagerTest.php [22:11:40] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [22:12:03] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/d154bb489a29...46e3a16e2c64 [22:12:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.07, 9.51, 9.74 [22:12:05] [02miraheze/CreateWiki] 07Universal-Omega 0346e3a16 - Update WikiManager.php [22:12:06] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [22:13:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.75, 3.48 [22:13:59] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.80, 9.17, 9.31 [22:14:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.76, 10.71, 10.15 [22:14:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.68, 8.17, 8.43 [22:14:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.92, 10.24, 9.96 [22:14:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.03, 9.05, 9.04 [22:15:15] miraheze/CreateWiki - Universal-Omega the build has errored. [22:15:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.59, 3.38, 3.38 [22:15:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.98, 9.19, 9.29 [22:16:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.27, 3.59, 3.58 [22:16:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.45, 8.90, 9.00 [22:17:13] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/46e3a16e2c64...bd26d5fb635a [22:17:15] [02miraheze/CreateWiki] 07Universal-Omega 03bd26d5f - Update WikiManager.php [22:17:16] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [22:17:26] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.07, 9.97, 9.61 [22:17:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.58, 10.05, 9.59 [22:18:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.05, 9.09, 8.78 [22:18:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.30, 3.90, 3.69 [22:19:10] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.66, 3.93, 3.58 [22:19:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.61, 9.47, 9.48 [22:19:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.34, 9.67, 9.93 [22:19:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.67, 9.64, 9.86 [22:20:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.74, 3.66 [22:21:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.71, 3.54 [22:21:20] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.50, 9.80, 9.59 [22:21:33] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/bd26d5fb635a...191ea36fdcd1 [22:21:34] [02miraheze/CreateWiki] 07Universal-Omega 03191ea36 - Update WikiManager.php [22:21:36] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [22:21:37] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/191ea36fdcd1...42bd905aba77 [22:21:39] [02miraheze/CreateWiki] 07Universal-Omega 0342bd905 - Update WikiManagerTest.php [22:21:40] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [22:21:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.93, 10.61, 10.24 [22:21:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.68, 10.10, 10.01 [22:22:01] miraheze/CreateWiki - Universal-Omega the build passed. [22:22:25] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp20.miraheze.org - CNAME OK [22:23:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.49, 3.78, 3.57 [22:23:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.47, 9.51, 9.52 [22:23:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.35, 9.73, 9.88 [22:23:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.46, 9.94, 9.72 [22:24:34] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/42bd905aba77...ec491d8d34d4 [22:24:35] [02miraheze/CreateWiki] 07Universal-Omega 03ec491d8 - Update WikiManagerTest.php [22:24:37] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [22:24:46] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ec491d8d34d4...fa2c8c5f06c2 [22:24:47] [02miraheze/CreateWiki] 07Universal-Omega 03fa2c8c5 - Update WikiManager.php [22:24:49] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [22:24:50] miraheze/CreateWiki - Universal-Omega the build has errored. [22:25:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.56, 10.62, 10.17 [22:26:23] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/fa2c8c5f06c2...ff128f939dc6 [22:26:24] [02miraheze/CreateWiki] 07Universal-Omega 03ff128f9 - Update WikiManagerTest.php [22:26:26] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [22:27:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.28, 10.54, 9.94 [22:28:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 3.68, 3.58 [22:28:49] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ff128f939dc6...055bfafa7ecd [22:28:50] [02miraheze/CreateWiki] 07Universal-Omega 03055bfaf - Update WikiManagerTest.php [22:28:52] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [22:29:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.11, 3.94, 3.77 [22:29:05] miraheze/CreateWiki - Universal-Omega the build has errored. [22:31:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.73, 10.26, 9.68 [22:31:50] miraheze/CreateWiki - Universal-Omega the build has errored. [22:31:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.54, 9.98, 9.88 [22:32:13] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/055bfafa7ecd...a8417a6987ec [22:32:14] [02miraheze/CreateWiki] 07Universal-Omega 03a8417a6 - Update WikiManagerTest.php [22:32:16] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [22:33:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.85, 9.83, 9.99 [22:34:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.60, 9.21, 8.94 [22:34:59] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.15, 9.94, 9.74 [22:35:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.70, 2.98, 3.38 [22:36:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.58, 8.26, 8.63 [22:37:22] miraheze/CreateWiki - Universal-Omega the build passed. [22:39:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.99, 9.84, 9.87 [22:40:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.58, 8.08, 8.49 [22:40:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.87, 7.79, 8.46 [22:40:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.14, 9.60, 9.59 [22:43:41] RECOVERY - wiki.landev.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.landev.vn' will expire on Sat 02 Apr 2022 03:10:10 GMT +0000. [22:43:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.15, 10.66, 10.14 [22:44:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.73, 9.31, 9.48 [22:45:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.96, 9.50, 9.77 [22:51:09] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.74, 2.04, 1.41 [22:51:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.63, 8.95, 8.75 [22:51:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.23, 9.26, 9.80 [22:53:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.25, 3.62, 3.36 [22:53:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.34, 1.76, 1.38 [22:53:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.21, 10.08, 9.70 [22:54:07] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:54:10] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.022 second response time [22:54:23] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.70, 3.12, 3.20 [22:55:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.33, 1.69, 1.41 [22:55:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.98, 9.08, 9.90 [22:55:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.24, 7.34, 8.16 [22:55:24] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.78, 9.92, 9.71 [22:56:00] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.38, 6.90, 8.19 [22:57:26] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.033 second response time [22:58:03] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.341 second response time [22:58:16] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.153 second response time [22:58:18] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.632 second response time [22:58:33] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22297 bytes in 2.445 second response time [22:58:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.53, 3.98 [22:59:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.29, 10.25, 10.18 [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:03] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.52, 9.69, 9.98 [23:01:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.68, 8.26, 8.20 [23:01:48] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/a8417a6987ec...00b7c647f994 [23:01:50] [02miraheze/CreateWiki] 07Universal-Omega 0300b7c64 - Update WikiManagerTest.php [23:01:51] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [23:02:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.00, 9.21, 8.81 [23:03:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.48, 10.16, 10.10 [23:03:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.89, 8.62, 8.43 [23:03:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.20, 7.67, 7.98 [23:05:14] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [23:05:26] miraheze/CreateWiki - Universal-Omega the build has errored. [23:05:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.87, 9.48, 9.42 [23:07:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.66, 8.05, 8.24 [23:07:25] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/00b7c647f994...5394fb529e18 [23:07:26] [02miraheze/CreateWiki] 07Universal-Omega 035394fb5 - Update WikiManagerTest.php [23:07:28] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [23:07:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.35, 8.60, 9.11 [23:10:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.69, 2.74, 3.33 [23:11:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.00, 10.17, 9.32 [23:12:18] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/5394fb529e18...80cc258fda93 [23:12:20] [02miraheze/CreateWiki] 07Universal-Omega 0380cc258 - Update WikiManagerTest.php [23:12:21] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [23:12:45] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.14, 9.20, 8.86 [23:13:13] miraheze/CreateWiki - Universal-Omega the build passed. [23:13:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.49, 9.37, 9.23 [23:14:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.55, 9.29, 8.94 [23:15:40] PROBLEM - wiki.landev.vn - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.landev.vn and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [23:15:43] miraheze/CreateWiki - Universal-Omega the build has errored. [23:15:58] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 956s [23:16:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:16:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.32, 3.38 [23:16:56] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1013s [23:17:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:17:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.82, 9.90, 9.52 [23:17:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.17, 9.43, 9.36 [23:18:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.22, 3.14, 3.29 [23:18:45] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.07, 9.80, 9.17 [23:19:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.24, 9.52, 9.39 [23:20:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.82, 9.52, 8.76 [23:20:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.22, 3.79 [23:21:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.59, 9.67, 9.95 [23:21:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.35, 9.08, 9.24 [23:22:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.81, 9.70, 9.29 [23:23:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.04, 9.60, 9.87 [23:23:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.06, 9.46, 9.36 [23:24:25] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/80cc258fda93...67f6e7fa3505 [23:24:26] [02miraheze/CreateWiki] 07Universal-Omega 0367f6e7f - Do database drops; add grants for renamewikitest [23:24:28] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [23:25:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.74, 9.80, 9.92 [23:25:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.69, 9.25, 9.30 [23:26:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.54, 2.69, 3.34 [23:27:30] miraheze/CreateWiki - Universal-Omega the build has errored. [23:29:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.02, 9.79, 9.89 [23:29:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.36, 9.96, 9.66 [23:30:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.24, 3.42 [23:31:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.88, 8.94, 9.56 [23:31:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.03, 9.38, 9.50 [23:31:50] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.90, 7.93, 8.44 [23:32:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.64, 3.82, 3.61 [23:32:56] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/67f6e7fa3505...ab50ce7efbc4 [23:32:57] [02miraheze/CreateWiki] 07Universal-Omega 03ab50ce7 - Update WikiManagerTest.php [23:32:59] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [23:33:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.27, 9.37, 9.63 [23:33:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.90, 3.49, 3.18 [23:34:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.88, 3.03, 3.34 [23:35:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.94, 3.63, 3.25 [23:37:08] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ab50ce7efbc4...9f9d28f2511f [23:37:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.40, 9.35, 9.68 [23:37:10] [02miraheze/CreateWiki] 07Universal-Omega 039f9d28f - Update globals.php [23:37:11] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [23:37:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.67, 7.44, 8.32 [23:40:15] miraheze/CreateWiki - Universal-Omega the build has errored. [23:40:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 4.01, 3.67 [23:41:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.42, 9.40, 9.16 [23:41:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.57, 8.87, 8.51 [23:41:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.60, 9.23, 8.84 [23:42:45] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.09, 9.63, 9.23 [23:43:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.87, 9.25, 9.42 [23:43:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.08, 8.79, 8.52 [23:44:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.07, 3.74, 3.64 [23:45:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.31, 9.96, 9.51 [23:45:50] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/9f9d28f2511f...546512cff44a [23:45:51] [02miraheze/CreateWiki] 07Universal-Omega 03546512c - Update globals.php [23:45:53] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [23:45:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.91, 9.39, 9.03 [23:46:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.72, 9.65, 9.37 [23:47:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 9.99, 10.12, 9.64 [23:47:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.16, 10.02, 9.31 [23:48:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.71, 3.93, 3.82 [23:49:06] miraheze/CreateWiki - Universal-Omega the build has errored. [23:49:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.06, 9.93, 9.72 [23:49:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.13, 9.54, 9.49 [23:49:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.00, 10.43, 9.22 [23:49:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.63, 8.13, 7.76 [23:49:58] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.97, 1.74, 1.08 [23:50:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.88, 3.71 [23:50:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.88, 3.80 [23:50:45] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/546512cff44a...7142ab66ee95 [23:50:47] [02miraheze/CreateWiki] 07Universal-Omega 037142ab6 - Update globals.php [23:50:48] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [23:51:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.55, 10.15, 9.82 [23:51:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.69, 9.97, 9.65 [23:51:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.20, 8.83, 8.05 [23:52:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.64, 3.47, 3.59 [23:52:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.64, 3.38, 3.63 [23:55:35] miraheze/CreateWiki - Universal-Omega the build passed. [23:56:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.87, 2.95, 3.33 [23:56:36] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.97, 2.96, 3.38 [23:57:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.08, 9.44, 9.60 [23:59:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.19, 9.24, 8.56 [23:59:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.18, 9.84, 9.84