[00:00:05] PROBLEM - graylog121 Puppet on graylog121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private] [00:00:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.44, 3.61 [00:00:54] PROBLEM - mon111 Puppet on mon111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private] [00:01:03] PROBLEM - gluster111 Puppet on gluster111 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private],File[glusterfs.key] [00:01:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.02, 8.99, 9.19 [00:02:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.45, 8.59, 8.33 [00:02:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.63, 3.08, 3.46 [00:03:02] PROBLEM - mail121 Current Load on mail121 is WARNING: WARNING - load average: 0.15, 0.78, 1.85 [00:03:03] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.17, 9.06, 8.98 [00:03:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.98, 8.83, 9.13 [00:04:23] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 567 failures. Last run 3 minutes ago with 567 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private],File[antiguabarbudacalypso.com_private],File[spiral.wiki_private],File[permanentfuturelab.wiki_private] [00:05:02] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 0.14, 0.55, 1.63 [00:05:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.07, 8.45, 8.78 [00:05:04] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.40, 9.26, 9.24 [00:06:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.87, 3.78, 3.63 [00:06:51] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 365 failures. Last run 2 minutes ago with 365 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private],File[antiguabarbudacalypso.com_private],File[spiral.wiki_private],File[permanentfuturelab.wiki_private] [00:07:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.93, 9.12, 9.19 [00:07:26] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Puppet has 365 failures. Last run 2 minutes ago with 365 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private],File[antiguabarbudacalypso.com_private],File[spiral.wiki_private],File[permanentfuturelab.wiki_private] [00:07:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.19, 9.26, 9.20 [00:08:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.19, 9.67, 8.89 [00:09:03] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.98, 9.57, 9.16 [00:10:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.61, 9.59, 8.96 [00:10:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:11:17] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private],File[/etc/ssl/private/miraheze.wiki.key] [00:11:24] PROBLEM - db111 Puppet on db111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private] [00:11:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.86, 8.40, 7.79 [00:13:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.90, 9.77, 9.40 [00:13:33] PROBLEM - ldap111 Puppet on ldap111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private] [00:13:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.17, 7.83, 7.65 [00:13:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.73, 8.47, 8.97 [00:14:07] PROBLEM - mw102 Puppet on mw102 is CRITICAL: CRITICAL: Puppet has 365 failures. Last run 2 minutes ago with 365 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private],File[antiguabarbudacalypso.com_private],File[spiral.wiki_private],File[permanentfuturelab.wiki_private] [00:14:34] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 365 failures. Last run 3 minutes ago with 365 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private],File[antiguabarbudacalypso.com_private],File[spiral.wiki_private],File[permanentfuturelab.wiki_private] [00:15:03] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.18, 10.27, 9.62 [00:15:50] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private] [00:15:57] PROBLEM - es121 Puppet on es121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private] [00:16:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.27, 9.85, 9.21 [00:16:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.89, 7.65, 8.48 [00:17:03] PROBLEM - gluster101 Puppet on gluster101 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private],File[glusterfs.key] [00:18:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.70, 9.72, 9.24 [00:18:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.38, 8.54, 8.24 [00:19:05] [02ssl] 07MacFan4000 closed pull request 03#480: Add theapplesecretswiki.ml cert - 13https://github.com/miraheze/ssl/pull/480 [00:19:06] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/bed731330bd1...14d643dc999b [00:19:08] [02miraheze/ssl] 07MacFan4000 0314d643d - Add theapplesecretswiki.ml cert (#480) [00:19:09] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [00:19:11] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [00:19:17] PROBLEM - es101 Puppet on es101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private] [00:19:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.53, 7.79, 8.48 [00:19:46] puppet is failing everyewhere with that cert [00:20:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.20, 10.07, 8.85 [00:20:01] (wildcard) [00:20:48] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 365 failures. Last run 2 minutes ago with 365 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private],File[antiguabarbudacalypso.com_private],File[spiral.wiki_private],File[permanentfuturelab.wiki_private] [00:20:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.47, 8.15, 8.56 [00:21:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.92, 9.78, 9.72 [00:21:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.24, 8.93, 8.16 [00:22:00] PROBLEM - db121 Puppet on db121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private] [00:22:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.01, 9.44, 9.21 [00:22:43] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private] [00:22:47] PROBLEM - mw112 Puppet on mw112 is CRITICAL: CRITICAL: Puppet has 365 failures. Last run 2 minutes ago with 365 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private],File[antiguabarbudacalypso.com_private],File[spiral.wiki_private],File[permanentfuturelab.wiki_private] [00:23:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.80, 9.44, 8.46 [00:24:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.88, 9.25, 9.17 [00:24:02] paladox: your permissions change for ssl is causing puppet failures, it cant read the keys now [00:24:06] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 365 failures. Last run 3 minutes ago with 365 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private],File[antiguabarbudacalypso.com_private],File[spiral.wiki_private],File[permanentfuturelab.wiki_private] [00:24:07] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.03, 1.68, 1.25 [00:24:12] oh [00:24:47] !log chown -R root:root /etc/puppetlabs/puppet/ssl-keys [00:25:03] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.61, 10.02, 9.80 [00:25:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:25:12] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org-2020-2_private] [00:25:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.97, 9.96, 8.77 [00:25:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.84, 8.91, 8.73 [00:25:57] issue is still present [00:26:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.10, 10.06, 9.46 [00:26:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.66, 1.31, 1.16 [00:26:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.61, 3.29 [00:26:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.58, 8.19, 8.50 [00:27:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.49, 9.93, 9.80 [00:27:45] i think the "puppet" user needs to be able to read that dir [00:27:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.65, 3.70, 3.30 [00:28:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.11, 3.07, 3.13 [00:28:17] [02CreateWiki] 07Universal-Omega edited pull request 03#297: Add hook to support modifying available database lists - 13https://github.com/miraheze/CreateWiki/pull/297 [00:28:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/0ada0dd8a131 [00:28:37] [02miraheze/mw-config] 07Universal-Omega 030ada0dd - Utilise hook for beta database list support [00:28:39] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [00:28:46] based on the permissions of other directories in that location [00:29:03] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.24, 9.97, 9.83 [00:29:20] [02mw-config] 07Universal-Omega opened pull request 03#4479: Utilise hook for beta database list support - 13https://github.com/miraheze/mw-config/pull/4479 [00:29:28] the certs are set to 755 where as the keys are set to 770 [00:29:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.69, 9.82, 9.03 [00:29:36] paladox: ^ [00:29:41] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.72, 4.16, 3.51 [00:30:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.66, 9.19, 8.90 [00:31:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.19, 9.84, 9.80 [00:32:34] i would chown puppet:puppet-users [00:33:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.36, 3.38, 3.36 [00:35:03] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.55, 9.80, 9.76 [00:35:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.09, 9.27, 8.89 [00:35:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.44, 9.32, 8.89 [00:35:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.50, 9.74, 9.62 [00:36:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.28, 9.98, 9.26 [00:37:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.61, 8.96, 8.85 [00:37:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.29, 8.95, 8.79 [00:38:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.21, 9.25, 9.09 [00:38:51] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/623108bfc2ae...67549a43e605 [00:38:52] [02miraheze/CreateWiki] 07Universal-Omega 0367549a4 - Update CreateWikiJson.php [00:38:54] [02CreateWiki] 07Universal-Omega synchronize pull request 03#297: Add hook to support modifying available database lists - 13https://github.com/miraheze/CreateWiki/pull/297 [00:39:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.22, 9.44, 8.98 [00:39:56] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.72, 9.95, 9.71 [00:40:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.86, 9.99, 9.39 [00:40:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.07, 3.45, 4.00 [00:41:40] paladox: I think we need to do chown -R puppet:puppet-users /etc/puppetlabs/puppet/ssl-keys (I believe puppet tries to access them as the "puppet" user and not root) [00:41:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.56, 9.55, 9.09 [00:41:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0ada0dd8a131...335b25a12f05 [00:41:48] [02miraheze/mw-config] 07Universal-Omega 03335b25a - Update MirahezeFunctions.php [00:41:50] [02mw-config] 07Universal-Omega synchronize pull request 03#4479: Utilise hook for beta database list support - 13https://github.com/miraheze/mw-config/pull/4479 [00:42:11] ok [00:42:18] !log chown -R puppet:puppet-users /etc/puppetlabs/puppet/ssl-keys [00:42:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:43:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.90, 10.80, 9.60 [00:43:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.99, 9.26, 9.52 [00:44:01] miraheze/CreateWiki - Universal-Omega the build passed. [00:45:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.09, 7.99, 8.45 [00:47:54] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.54, 10.12, 9.82 [00:49:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.79, 8.59, 8.61 [00:51:07] !log chown -R root:root /etc/puppetlabs/puppet/ssl-keys [00:51:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.77, 9.37, 8.89 [00:51:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.64, 3.67, 3.67 [00:53:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.42, 9.59, 9.79 [00:54:22] @Void: around/got access? [00:54:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.48, 3.22, 3.51 [00:55:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.67, 9.76, 9.17 [00:55:52] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.89, 10.11, 9.96 [00:56:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.47, 2.71, 3.30 [00:57:21] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.72, 10.39, 9.47 [00:57:26] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:57:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.00, 9.75, 9.85 [00:59:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.69, 9.94, 10.00 [00:59:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.21, 10.00, 9.93 [01:00:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.48, 9.70, 9.99 [01:00:57] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:01:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.72, 9.78, 9.51 [01:01:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.90, 10.07, 10.04 [01:05:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.46, 9.71, 9.44 [01:05:20] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [01:06:45] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:07:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.95, 9.92, 9.57 [01:07:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.28, 9.41, 9.83 [01:08:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.42, 9.29, 9.57 [01:09:16] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [01:09:23] RECOVERY - db111 Puppet on db111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:09:49] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.09, 9.57, 9.82 [01:10:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.32, 9.37, 9.58 [01:11:33] RECOVERY - ldap111 Puppet on ldap111 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [01:11:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.67, 9.14, 9.65 [01:12:17] RECOVERY - mw102 Puppet on mw102 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [01:12:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.02, 10.00, 9.79 [01:13:17] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:13:45] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [01:13:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.20, 9.51, 9.71 [01:13:56] RECOVERY - es121 Puppet on es121 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [01:14:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.36, 9.79, 9.74 [01:14:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.86, 4.35, 3.63 [01:17:04] RECOVERY - gluster101 Puppet on gluster101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:17:15] RECOVERY - es101 Puppet on es101 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [01:18:52] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [01:19:59] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [01:20:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.89, 3.90, 3.69 [01:20:42] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [01:20:43] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [01:21:31] RECOVERY - mw112 Puppet on mw112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:22:38] RECOVERY - es111 Puppet on es111 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [01:22:58] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [01:23:11] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:23:13] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:24:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.64, 4.09, 3.81 [01:25:26] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [01:25:48] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [01:25:49] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:26:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.67, 3.68 [01:27:01] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:28:04] RECOVERY - graylog121 Puppet on graylog121 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [01:30:34] RECOVERY - gluster121 Puppet on gluster121 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [01:30:36] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:30:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.31, 9.21, 9.78 [01:31:07] RECOVERY - gluster111 Puppet on gluster111 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [01:32:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.94, 10.11, 10.03 [01:34:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.51, 9.63, 9.16 [01:35:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 4.06, 3.85 [01:36:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.09, 9.26, 9.08 [01:37:10] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.65, 9.76, 9.44 [01:37:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.84, 3.87, 3.80 [01:40:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.75, 9.56, 9.98 [01:42:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.82, 9.78, 10.00 [01:45:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.60, 3.61 [01:47:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.60, 9.59, 9.82 [01:47:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.84, 3.12, 3.42 [01:48:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.80, 8.91, 9.78 [01:49:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.51, 10.12, 9.99 [01:52:09] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.13, 9.48, 9.82 [01:52:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.79, 8.76, 9.88 [01:53:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 3.73, 3.61 [01:54:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.59, 9.51, 9.79 [01:55:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.79, 3.65 [01:57:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 3.98, 3.74 [01:58:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.81, 9.87, 9.85 [01:58:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.23, 9.53, 9.34 [02:00:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.85, 9.18, 9.24 [02:00:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.54, 9.81, 9.80 [02:00:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.27, 9.19, 9.99 [02:01:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.88, 3.58, 3.64 [02:03:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.96, 9.35, 9.91 [02:04:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.52, 9.81, 9.80 [02:05:54] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [02:05:54] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4027698040008545 seconds [02:07:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.76, 9.10, 9.84 [02:08:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.15, 9.93, 9.84 [02:08:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.48, 9.75, 9.87 [02:09:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.86, 10.57, 10.17 [02:09:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.46, 9.67, 9.97 [02:10:13] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.80, 9.59, 9.23 [02:11:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.79, 3.05, 3.37 [02:12:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.86, 9.24, 9.17 [02:15:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.02, 4.04, 3.66 [02:17:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.69, 3.57 [02:18:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.67, 9.51, 9.21 [02:18:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.69, 9.68, 9.86 [02:19:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 3.72, 3.58 [02:20:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.01, 9.69, 9.99 [02:21:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.05, 3.76, 3.64 [02:22:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.14, 8.77, 8.97 [02:22:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.46, 9.57, 9.94 [02:24:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.52, 9.99, 10.03 [02:25:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.72, 2.90, 3.33 [02:26:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.67, 9.44, 9.85 [02:27:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.36, 8.67, 9.65 [02:27:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.87, 8.65, 9.71 [02:28:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.08, 9.03, 8.92 [02:30:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.79, 9.12, 8.96 [02:30:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.46, 9.72, 9.63 [02:31:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.73, 9.54, 9.82 [02:34:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.36, 3.08 [02:35:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.22, 8.52, 9.05 [02:35:04] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [02:35:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.31, 8.54, 9.39 [02:35:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.73, 3.54, 3.46 [02:36:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.00, 3.32, 3.11 [02:36:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.47, 9.80, 9.73 [02:37:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.56, 8.71, 9.06 [02:37:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.52, 4.34, 3.76 [02:38:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.41, 9.81, 9.72 [02:40:01] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.43, 7.06, 8.25 [02:40:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.63, 3.51, 3.27 [02:40:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.52, 10.02, 9.62 [02:41:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.12, 9.23, 9.18 [02:41:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.95, 3.76 [02:42:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.13, 9.47, 9.66 [02:42:44] [discord] Not really, no [02:43:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.80, 8.51, 8.91 [02:44:07] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.09, 3.15 [02:45:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.25, 7.10, 8.28 [02:45:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.76, 8.26, 8.49 [02:47:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 3.80, 3.74 [02:48:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.07, 9.84, 9.67 [02:48:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.33, 9.35, 9.57 [02:49:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.65, 3.70 [02:51:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.75, 7.97, 8.44 [02:51:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.98, 3.81 [02:55:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.84, 8.75, 8.58 [02:56:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.89, 9.35, 9.66 [02:57:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.91, 8.62, 8.56 [02:58:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.15, 9.86, 9.64 [02:58:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 3.57, 3.22 [02:59:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.09, 8.02, 8.35 [03:00:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.92, 8.89, 8.58 [03:00:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.53, 3.24 [03:02:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.40, 9.71, 9.65 [03:02:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.07, 9.13, 8.49 [03:02:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 4.02, 3.46 [03:04:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 9.59, 8.87 [03:04:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.37, 3.37, 3.29 [03:04:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.70, 9.82, 9.79 [03:05:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.54, 7.73, 8.33 [03:06:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.05, 8.83, 8.67 [03:06:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.81, 9.92, 9.81 [03:07:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.02, 3.13, 3.88 [03:08:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.26, 8.19, 8.45 [03:10:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.69, 8.23, 8.40 [03:10:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.92, 9.24, 9.59 [03:11:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.89, 8.73, 8.46 [03:11:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.95, 9.09, 8.74 [03:11:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.51, 3.42 [03:13:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.32, 9.11, 8.61 [03:13:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.55, 3.12, 3.29 [03:14:04] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.61, 9.98, 9.03 [03:14:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.37, 10.16, 9.84 [03:14:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.65, 8.85, 9.43 [03:15:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.65, 9.01, 8.62 [03:15:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.92, 4.15, 3.97 [03:16:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.42, 9.44, 8.93 [03:17:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.41, 3.94, 3.92 [03:18:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.02, 9.65, 9.74 [03:19:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.79, 10.11, 9.21 [03:20:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.22, 9.60, 9.06 [03:21:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.38, 9.98, 9.17 [03:21:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.58, 9.18, 8.98 [03:21:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 4.24, 4.05 [03:22:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.18, 9.46, 9.08 [03:24:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.81, 10.31, 9.92 [03:25:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.35, 9.63, 9.26 [03:27:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.35, 9.45, 9.14 [03:27:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.42, 3.58, 3.88 [03:28:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.10, 9.26, 9.04 [03:29:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.08, 9.24, 9.09 [03:30:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.93, 3.60, 3.23 [03:30:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.19, 8.62, 8.38 [03:32:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.97, 9.64, 9.24 [03:32:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.56, 3.25, 3.15 [03:32:13] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.06, 8.04, 8.20 [03:33:21] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [03:33:21] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406573057174683 seconds [03:34:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.38, 9.75, 9.90 [03:36:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.25, 9.84, 9.91 [03:36:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.66, 8.49, 8.32 [03:37:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.57, 4.03, 3.77 [03:38:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.84, 9.61, 9.80 [03:38:09] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.23, 8.45, 8.33 [03:40:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:41:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.70, 9.27, 9.04 [03:42:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.24, 9.45, 9.26 [03:43:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.16, 9.06, 8.98 [03:43:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.57, 7.75, 8.31 [03:47:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.77, 3.33, 3.75 [03:47:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:48:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.80, 9.84, 9.24 [03:49:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.96, 9.15, 8.67 [03:51:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.10, 7.84, 8.44 [03:51:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.03, 3.74, 3.80 [03:52:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.06, 9.78, 9.62 [03:52:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.86, 9.36, 9.23 [03:54:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.58, 10.52, 9.66 [03:55:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.74, 9.58, 9.04 [03:57:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.30, 9.82, 9.00 [03:57:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.86, 10.13, 9.30 [03:57:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.32, 3.59 [03:58:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.06, 9.76, 9.35 [03:58:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.73, 8.60, 8.35 [03:59:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.02, 9.52, 9.03 [03:59:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.40, 9.40, 9.14 [03:59:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.90, 3.80, 3.72 [04:00:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.74, 9.02, 9.13 [04:00:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.12, 9.38, 9.50 [04:00:53] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.65, 7.67, 8.02 [04:01:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.65, 3.68 [04:02:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.69, 9.57, 9.28 [04:02:28] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [04:04:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.82, 9.18, 9.16 [04:04:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.38, 9.72, 9.38 [04:06:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.09, 9.26, 9.17 [04:06:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.41, 9.48, 9.46 [04:07:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.56, 8.05, 8.49 [04:07:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:08:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.70, 9.02, 9.21 [04:08:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.31, 8.48, 8.90 [04:08:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.48, 8.70, 9.18 [04:09:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.69, 7.39, 8.27 [04:10:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.40, 9.43, 9.36 [04:11:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.54, 9.05, 8.81 [04:12:01] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.24, 7.70, 8.47 [04:12:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.67, 9.57, 9.43 [04:12:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.81, 8.56, 8.22 [04:15:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.90, 9.10, 8.73 [04:15:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.07, 4.12, 3.73 [04:16:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.64, 8.14, 8.52 [04:16:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:17:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.07, 7.76, 8.39 [04:17:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.45, 3.14 [04:18:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.85, 10.44, 9.79 [04:18:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.43, 8.30, 8.26 [04:19:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.18, 7.71, 8.50 [04:20:01] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.98, 8.21, 8.49 [04:20:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.72, 9.99, 9.70 [04:23:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.58, 8.56, 8.49 [04:23:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.97, 4.00 [04:23:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.07, 3.25, 3.21 [04:23:26] [02CreateWiki] 07Universal-Omega closed pull request 03#296: Bump stylelint-config-wikimedia from 0.11.1 to 0.12.1 - 13https://github.com/miraheze/CreateWiki/pull/296 [04:23:28] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/69969da762e0...8862ddb80f98 [04:23:29] [02miraheze/CreateWiki] 07dependabot[bot] 038862ddb - Bump stylelint-config-wikimedia from 0.11.1 to 0.12.1 (#296) [04:23:31] [02CreateWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.12.1 - 13https://github.com/miraheze/CreateWiki [04:23:32] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.12.1 [04:23:56] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.62, 9.50, 9.00 [04:25:50] miraheze/CreateWiki - Universal-Omega the build has errored. [04:26:19] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03revert-296-dependabot/npm_and_yarn/stylelint-config-wikimedia-0.12.1 [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/commit/41c2db4a2483 [04:26:21] [02miraheze/CreateWiki] 07Universal-Omega 0341c2db4 - Revert "Bump stylelint-config-wikimedia from 0.11.1 to 0.12.1 (#296)" [04:26:22] [02CreateWiki] 07Universal-Omega created branch 03revert-296-dependabot/npm_and_yarn/stylelint-config-wikimedia-0.12.1 - 13https://github.com/miraheze/CreateWiki [04:26:24] [02CreateWiki] 07Universal-Omega opened pull request 03#299: Revert "Bump stylelint-config-wikimedia from 0.11.1 to 0.12.1" - 13https://github.com/miraheze/CreateWiki/pull/299 [04:27:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.94, 4.64, 4.17 [04:27:17] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.91, 3.87, 3.44 [04:27:40]