[00:02:15] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 2 minutes ago with 5 failures. Failed resources (up to 3 shown): File[/home/johnflewis],File[/home/paladox],File[/home/reception],File[/home/void] [00:05:22] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.07, 19.83, 17.22 [00:08:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.52, 3.79, 3.43 [00:09:22] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.05, 21.68, 18.50 [00:10:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.68, 3.45 [00:13:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.10, 6.56, 5.50 [00:13:44] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.22, 5.59, 4.27 [00:15:03] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.39, 5.72, 5.33 [00:19:58] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.30, 6.70, 5.31 [00:20:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.74, 3.54 [00:20:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.53, 7.39, 6.07 [00:22:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:23:48] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.22, 7.93, 6.47 [00:23:56] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.12, 7.99, 6.24 [00:24:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.99, 6.06, 5.09 [00:24:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.58, 7.25, 6.34 [00:25:48] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.06, 7.66, 6.56 [00:26:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.44, 5.71, 5.08 [00:26:52] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.32, 6.58, 6.20 [00:27:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:28:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.75, 3.52, 3.55 [00:29:48] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.07, 8.54, 7.16 [00:31:48] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.10, 7.55, 6.96 [00:31:51] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.58, 6.37, 6.21 [00:32:09] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:48] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.37, 6.71, 6.71 [00:34:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.45, 2.73, 1.90 [00:36:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.07, 3.12, 3.40 [00:36:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.59, 1.98, 1.73 [00:38:17] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.57, 1.57, 1.61 [00:40:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.79, 4.09, 3.71 [00:42:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.99, 3.73 [00:44:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.88, 6.39, 5.72 [00:46:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.10, 5.78, 5.57 [00:50:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.41, 3.02, 3.38 [00:54:01] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.49, 4.47, 5.76 [00:57:22] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.15, 20.02, 23.58 [01:02:01] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.03, 3.35, 4.78 [01:05:22] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.76, 15.80, 20.10 [01:06:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.96, 4.41, 3.73 [01:18:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.73, 3.84 [01:20:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.76, 4.24, 4.00 [01:21:48] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.27, 1.58, 1.10 [01:23:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.35, 1.07 [01:26:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.91, 3.99 [01:36:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 3.61, 3.71 [01:38:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.61, 3.70 [01:39:05] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JDnJZ [01:39:07] [02miraheze/mediawiki] 07Universal-Omega 03a738210 - Labeler: add more file types to existing + add schema-change [01:39:08] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [01:39:10] [02mediawiki] 07Universal-Omega opened pull request 03#4557: Labeler: add more file types to existing + add schema-change - 13https://git.io/JDnJn [01:40:48] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JDnJP [01:40:50] [02miraheze/mediawiki] 07Universal-Omega 0331a136d - Update label.yml [01:40:51] [02mediawiki] 07Universal-Omega synchronize pull request 03#4557: Labeler: add more file types to existing + add schema-change - 13https://git.io/JDnJn [01:46:11] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.88, 6.70, 5.23 [01:48:09] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.65, 6.63, 5.39 [01:52:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.41, 3.48 [01:54:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.38, 3.45 [01:57:48] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.97, 6.31, 5.22 [01:58:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.28, 3.16, 3.37 [01:59:28] [02mediawiki] 07Universal-Omega edited pull request 03#4557: Labeler: add more file types to existing + add `schema-change` - 13https://git.io/JDnJn [01:59:48] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.45, 5.95, 5.22 [02:02:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.59, 3.77, 3.56 [02:04:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.41, 3.44 [02:06:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.23, 3.04, 3.31 [02:14:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.42, 3.34 [02:18:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.40, 3.76, 3.48 [02:22:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.01, 3.78, 3.59 [02:26:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.81, 3.16, 3.37 [02:38:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.76, 3.45 [02:44:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.49, 3.29, 3.35 [02:48:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.87, 3.94, 3.58 [02:49:20] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.10, 5.86, 5.02 [02:50:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.68, 3.53 [02:51:16] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.71, 5.43, 4.96 [02:54:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.25, 2.90, 3.26 [03:06:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.32, 7.20, 5.41 [03:08:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.96, 7.39, 5.73 [03:12:52] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.14, 5.87, 5.49 [03:18:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.50, 6.88, 6.02 [03:20:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.49, 6.42, 5.96 [03:22:43] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JDnnM [03:22:45] [02miraheze/landing] 07Universal-Omega 0330fe809 - Update getTranslations.php [03:22:46] [02landing] 07Universal-Omega synchronize pull request 03#70: Support [link|text] style links - 13https://git.io/JDZ9H [03:23:39] miraheze/landing - Universal-Omega the build passed. [04:44:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.27, 6.94, 5.42 [04:46:52] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.22, 6.50, 5.46 [05:03:44] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.09, 1.68, 1.27 [05:05:44] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.20, 1.49, 1.25 [06:05:07] That is the first time in a long time that I have seen there not be a single icinga alert for an entire hour! [06:07:35] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.08, 5.86, 3.80 [06:08:04] An hour and 2 minutes to be exact... [06:13:35] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.31, 6.15, 4.67 [06:23:31] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.10, 6.23, 5.07 [06:25:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.45, 7.43, 5.64 [06:27:31] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.65, 6.94, 5.68 [06:29:31] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.03, 6.29, 5.59 [06:30:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.84, 6.04, 4.83 [06:32:52] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.39, 5.87, 4.92 [07:03:33] Wow [07:03:58] Icinga has always been a disturbance [07:11:03] PROBLEM - gluster3 Gluster Disk Space on gluster3 is CRITICAL: DISK CRITICAL - free space: /mnt/mediawiki-static 129676 MB (5% inode=80%); [08:01:58] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JDcJY [08:02:00] [02miraheze/mw-config] 07Reception123 03d075eea - add sitenotice [08:02:01] [02mw-config] 07Reception123 synchronize pull request 03#4280: Disable 3D and Math temporarily - 13https://git.io/JD3jZ [08:04:59] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JDcJD [08:05:00] [02miraheze/mw-config] 07Reception123 03d285fd7 - Update Sitenotice.php [08:05:02] [02mw-config] 07Reception123 synchronize pull request 03#4280: Disable 3D and Math temporarily - 13https://git.io/JD3jZ [08:10:43] PROBLEM - mw13 Puppet on mw13 is WARNING: WARNING: Puppet is currently disabled, message: Reception123 - Math/3D, last run 20 minutes ago with 0 failures [08:11:12] PROBLEM - mw11 Puppet on mw11 is WARNING: WARNING: Puppet is currently disabled, message: Reception123 - Math/3D, last run 29 minutes ago with 0 failures [08:11:23] !log disable puppet on mw* [08:11:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:12:03] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JDcU6 [08:12:04] [02miraheze/mw-config] 07Reception123 038bc75a1 - Update Sitenotice.php [08:12:06] [02mw-config] 07Reception123 synchronize pull request 03#4280: Disable 3D and Math temporarily - 13https://git.io/JD3jZ [08:12:17] PROBLEM - mw9 Puppet on mw9 is WARNING: WARNING: Puppet is currently disabled, message: Reception123 - Math/3D, last run 23 minutes ago with 0 failures [08:12:22] PROBLEM - mw12 Puppet on mw12 is WARNING: WARNING: Puppet is currently disabled, message: Reception123 - Math/3D, last run 15 minutes ago with 0 failures [08:12:28] PROBLEM - mw10 Puppet on mw10 is WARNING: WARNING: Puppet is currently disabled, message: Reception123 - Math/3D, last run 20 minutes ago with 0 failures [08:12:59] PROBLEM - mwtask1 Puppet on mwtask1 is WARNING: WARNING: Puppet is currently disabled, message: Reception123 - Math/3D, last run 32 minutes ago with 0 failures [08:13:51] [02miraheze/mw-config] 07Universal-Omega pushed 034 commits to 03Universal-Omega-patch-3 [+0/-0/±7] 13https://git.io/JDcU5 [08:13:53] [02miraheze/mw-config] 07Universal-Omega 032a97198 - Merge branch 'master' into Universal-Omega-patch-3 [08:13:54] [02mw-config] 07Universal-Omega synchronize pull request 03#4280: Disable 3D and Math temporarily - 13https://git.io/JD3jZ [08:14:12] PROBLEM - mw8 Puppet on mw8 is WARNING: WARNING: Puppet is currently disabled, message: Reception123 - Math/3D, last run 32 minutes ago with 0 failures [08:14:15] [02mw-config] 07Universal-Omega synchronize pull request 03#4280: Disable 3D and Math temporarily - 13https://git.io/JD3jZ [08:14:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JDcUx [08:14:18] [02miraheze/mw-config] 07Universal-Omega 03a6aa989 - Update Sitenotice.php [08:14:56] miraheze/mw-config - Universal-Omega the build passed. [08:15:21] miraheze/mw-config - Universal-Omega the build passed. [08:15:33] [02mw-config] 07Reception123 closed pull request 03#4280: Disable 3D and Math temporarily - 13https://git.io/JD3jZ [08:15:35] [02mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [08:15:36] [02miraheze/mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-3 [08:15:38] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JDcTT [08:15:39] [02miraheze/mw-config] 07Universal-Omega 038ed5fec - Disable 3D and Math temporarily (#4280) [08:15:40] !log [reception@mw11] starting deploy of {'pull': 'config', 'config': True} to all [08:15:44] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.41, 1.76, 1.15 [08:15:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:15:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4280-Universal-Omega-patch-3 [+0/-0/±2] 13https://git.io/JDcTI [08:15:50] [02miraheze/mw-config] 07Universal-Omega 0321fb5db - Revert "Disable 3D and Math temporarily (#4280)" [08:15:50] !log [reception@mw11] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [08:15:51] [02mw-config] 07Universal-Omega created branch 03revert-4280-Universal-Omega-patch-3 - 13https://git.io/vbvb3 [08:15:53] [02mw-config] 07Universal-Omega opened pull request 03#4283: Revert "Disable 3D and Math temporarily" - 13https://git.io/JDcTL [08:15:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:16:38] miraheze/mw-config - Reception123 the build passed. [08:16:41] [02puppet] 07Reception123 closed pull request 03#2142: mediawiki: use deploy repos for mathoid and 3d2png - 13https://git.io/JMQhM [08:16:42] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-1/±4] 13https://git.io/JDcTC [08:16:44] [02miraheze/puppet] 07Universal-Omega 036f626e5 - mediawiki: use deploy repos for mathoid and 3d2png (#2142) [08:16:49] miraheze/mw-config - Universal-Omega the build passed. [08:17:44] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.77, 1.37, 1.08 [08:17:44] [02mw-config] 07Reception123 closed pull request 03#4256: Update to new paths for `$wgMathoidCli` and `$wg3dProcessor` - 13https://git.io/JM7uk [08:17:45] [02mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [08:17:47] [02miraheze/mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-1 [08:17:48] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDcTu [08:17:50] [02miraheze/mw-config] 07Universal-Omega 03f3b549e - Update to new paths for `$wgMathoidCli` and `$wg3dProcessor` (#4256) [08:18:12] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:18:25] Reception123: your sitenotice was never enabled [08:18:37] Yes it was. I see it. [08:18:48] miraheze/mw-config - Reception123 the build passed. [08:18:52] Oh it was from the moment it was enabled [08:19:11] https://git.io/JDcJY alone didn't [08:19:31] I didn't see the follow up [08:19:38] Ah yeah... [08:20:18] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [08:20:29] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:20:50] !log [@mw11] starting deploy of {'config': True} to all [08:20:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:20:56] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 6s [08:20:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.99, 6.38, 4.86 [08:21:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:21:12] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:21:39] !log sudo rm -rf /srv/3d2png && sudo rm -rf /srv/mathoid on mw*/mwtask1 and re-enable puppet/run [08:21:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:21:48] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.84, 5.72, 4.56 [08:22:01] [02mw-config] 07Reception123 closed pull request 03#4283: Revert "Disable 3D and Math temporarily" - 13https://git.io/JDcTL [08:22:03] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JDckL [08:22:04] [02miraheze/mw-config] 07Universal-Omega 034d9ae33 - Revert "Disable 3D and Math temporarily (#4280)" (#4283) [08:22:06] [02mw-config] 07Reception123 deleted branch 03revert-4280-Universal-Omega-patch-3 - 13https://git.io/vbvb3 [08:22:07] [02miraheze/mw-config] 07Reception123 deleted branch 03revert-4280-Universal-Omega-patch-3 [08:22:13] !log [reception@mw11] starting deploy of {'pull': 'config', 'config': True} to all [08:22:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:22:19] !log [reception@mw11] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 5s [08:22:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:22:24] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:22:44] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [08:22:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.91, 5.93, 4.89 [08:22:59] RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:23:05] miraheze/mw-config - Reception123 the build passed. [08:23:48] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.40, 5.62, 4.68 [08:23:50] !log [@test3] starting deploy of {'config': True} to skip [08:23:51] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [08:23:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:23:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:53:41] !log deploy-mediawiki --world --l10n --servers=all on mw11/test3 [08:53:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:54:33] Reception123: that should automatically log? [08:55:07] CosmicAlpha: oh whoops, logged the wrong thing [08:55:08] I meant [08:55:18] !log sudo -u www-data rm -rf /srv/mediawiki-staging/w/skins/Femiwiki/node_modules on mw11/test3 [08:55:26] !log [reception@mw11] starting deploy of {'world': True, 'l10n': True} to all [08:55:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:55:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:56:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.47, 6.51, 4.79 [08:56:26] Reception123: just to confirm, before deploy-mediawiki did you run puppet before deploy-mediawiki? If not cancel deploy and do that. Otherwise Femiwiki dependencies will be missing from prod. [08:56:34] yes [08:57:09] Ok. Thanks, just wanted to confirm. [08:57:30] Thanks for doing that deployment Reception123! [08:58:14] no problem :) [08:58:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.91, 5.90, 4.78 [08:58:24] !log [reception@test3] starting deploy of {'world': True, 'l10n': True} to skip [08:58:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:59:24] Reception123: Also by the way if you want (otherwise I can tomorrow), switch DiscordNotifications to my fork, to fix the last remaining high priority task. I can definitely do tomorrow if you'd rather though. [09:00:48] yeah I can probably do that [09:01:45] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.60, 6.79, 5.93 [09:03:44] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 4.96, 6.06, 5.77 [09:06:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.36, 9.12, 6.45 [09:10:08] !log [reception@test3] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS in 704s [09:10:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:10:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.34, 6.85, 6.16 [09:12:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.54, 6.57, 6.14 [09:16:14] !log [reception@mw11] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 1248s [09:16:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:28:50] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[femiwiki_npm] [09:30:26] Reception123: ^ [10:22:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.61, 2.80 [10:22:08] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.18, 6.20, 4.50 [10:24:08] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.94, 5.30, 4.37 [10:26:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.81, 3.27, 2.86 [11:22:08] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDcoD [11:22:10] [02miraheze/mw-config] 07RhinosF1 0341362bc - Update PhpAutoPrepend.php [11:22:22] Reception123: ^ [11:22:27] But I'm mobile and F1 is on [11:22:30] So not around [11:23:10] miraheze/mw-config - RhinosF1 the build passed. [11:23:31] Ack, will look a bit later [11:24:06] !log [@test3] starting deploy of {'config': True} to skip [11:24:07] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [11:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:24:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:41:20] !log [@mw11] starting deploy of {'config': True} to all [11:41:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:41:33] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 13s [11:41:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:44:17] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:45:00] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.51, 10.87, 6.60 [11:45:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:47:45] Reception123: you got access? [11:48:17] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15891 bytes in 2.132 second response time [11:50:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:55:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:56:19] Reception123: it OOM'd [11:56:22] RhinosF1: works now! [11:56:33] I suggest we save the files and stop profiling [11:56:33] RhinosF1: as I said in the other chat, it seems like all mw*s are OOMing actuall [11:56:35] *actually [11:56:41] RhinosF1: sure [11:56:44] They are going to use up far too much disk [11:56:53] Reception123: move the files out of the cache folder [11:56:55] yeah [11:57:00] I'll just move them to my homedir for now [11:57:01] I got a cron to kill them after 30 minutes [11:57:04] Cool [11:57:08] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.10, 6.81, 5.53 [11:59:02] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.69, 6.06, 5.41 [12:00:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:02:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:03:51] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:04:06] Reception123: stop the profiling [12:04:43] RhinosF1: /doing [12:06:30] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JDcD9 [12:06:31] [02miraheze/mw-config] 07Reception123 03498b029 - stop profiling mw8 [12:06:33] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [12:06:34] [02mw-config] 07Reception123 opened pull request 03#4284: stop profiling mw8 - 13https://git.io/JDcDH [12:07:35] miraheze/mw-config - Reception123 the build passed. [12:07:58] [02mw-config] 07Reception123 closed pull request 03#4284: stop profiling mw8 - 13https://git.io/JDcDH [12:07:59] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [12:08:01] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [12:08:02] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDcyJ [12:08:04] [02miraheze/mw-config] 07Reception123 038e1b43d - stop profiling mw8 (#4284) [12:08:08] !log [reception@mw11] starting deploy of {'world': True, 'l10n': True} to all [12:08:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:09:04] miraheze/mw-config - Reception123 the build passed. [12:12:11] !log [@mw11] starting deploy of {'config': True} to all [12:12:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.19, 5.76, 4.52 [12:12:40] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 28s [12:12:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:12:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:13:54] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15891 bytes in 0.014 second response time [12:14:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.01, 5.59, 4.60 [12:14:51] !log [reception@mw11] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 403s [12:14:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:17:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:24:26] !log [@test3] starting deploy of {'config': True} to skip [12:24:28] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s [12:24:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:30:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 3.81, 5.20, 7.88 [12:36:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.24, 4.52, 6.70 [13:22:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.34, 2.87 [13:24:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.11, 3.22, 2.88 [13:51:05] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.86, 6.61, 5.16 [13:51:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.11, 3.99, 3.17 [13:53:05] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.79, 5.86, 5.06 [13:54:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.81, 3.28 [13:56:52] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.41, 3.29, 3.15 [14:20:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.33, 2.98 [14:22:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.51, 2.69, 2.79 [14:40:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.35, 2.91 [14:42:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.57, 3.04 [14:42:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.98, 6.77, 5.12 [14:44:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.16, 2.95 [14:44:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.76, 6.27, 5.13 [14:49:48] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.41, 8.28, 6.42 [14:50:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.35, 7.28, 5.88 [14:51:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.74, 8.54, 6.38 [14:51:48] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.36, 7.94, 6.53 [14:52:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.53, 6.82, 5.88 [14:54:09] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.81, 1.80, 1.37 [14:54:13] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.75, 7.26, 5.91 [14:54:33] PROBLEM - gluster3 Disk Space on gluster3 is CRITICAL: DISK CRITICAL - free space: / 64155 MB (5% inode=81%); [14:54:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.67, 5.95, 5.67 [14:55:05] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.05, 7.31, 6.37 [14:56:13] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.80, 7.52, 6.18 [14:57:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.63, 8.05, 6.74 [14:57:30] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.70, 7.60, 5.84 [14:57:48] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.81, 9.15, 7.35 [14:58:09] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.96, 1.46, 1.34 [14:58:12] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.67, 8.18, 6.59 [14:58:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.05, 7.87, 6.59 [14:59:31] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.06, 6.90, 5.80 [14:59:58] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.31, 22.18, 19.15 [15:00:11] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.90, 7.12, 6.41 [15:00:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.60, 6.72, 6.33 [15:01:05] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.50, 7.54, 6.86 [15:01:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.48, 3.23 [15:01:31] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.94, 6.22, 5.69 [15:01:48] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.03, 7.78, 7.22 [15:03:05] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.77, 6.30, 6.50 [15:03:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 3.18, 3.15 [15:03:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.15, 19.62, 18.81 [15:04:09] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.88, 5.95, 6.12 [15:05:48] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.12, 6.19, 6.67 [15:22:20] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.30, 4.26, 3.61 [15:24:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.98, 3.58 [15:27:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:27:48] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.00, 9.62, 7.26 [15:28:01] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.69, 5.06, 3.62 [15:28:08] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.10, 7.43, 5.95 [15:28:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.10, 8.48, 6.85 [15:29:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 7.98, 8.46, 7.06 [15:29:22] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.31, 20.42, 18.45 [15:30:01] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.36, 4.06, 3.43 [15:30:08] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.42, 7.47, 6.15 [15:30:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.42, 7.58, 6.70 [15:31:05] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.60, 7.35, 6.82 [15:31:22] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.52, 18.49, 18.00 [15:31:48] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.74, 7.61, 7.06 [15:32:08] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.13, 6.19, 5.84 [15:32:21] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.21, 6.49, 6.41 [15:32:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:33:05] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.25, 6.04, 6.42 [15:33:48] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.56, 6.68, 6.79 [15:40:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.23, 3.02, 3.33 [15:46:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 3.49, 3.37 [15:47:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.07, 7.22, 6.35 [15:48:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.73, 3.12, 3.24 [15:49:05] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.88, 6.54, 6.20 [16:24:32] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.96, 7.92, 5.61 [16:25:30] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.17, 6.33, 5.09 [16:26:30] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.32, 6.91, 5.53 [16:27:31] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.47, 5.54, 4.95 [16:28:30] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.49, 6.03, 5.38 [16:30:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.85, 4.02, 3.48 [16:32:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.80, 3.46 [16:36:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.33, 3.27, 3.34 [16:42:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.54, 4.27, 3.74 [16:48:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.83, 3.73 [16:50:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 4.10, 3.85 [16:56:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.45, 3.47, 3.72 [17:04:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.58, 3.59 [17:06:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.08, 3.08, 3.41 [17:08:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.05, 3.36 [17:16:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.31, 4.01, 3.61 [17:17:22] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.12, 18.59, 15.86 [17:18:01] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.32, 5.47, 3.72 [17:18:09] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.22, 5.81, 3.42 [17:19:22] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.24, 21.70, 17.28 [17:20:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.94, 3.66 [17:20:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.77, 7.32, 5.66 [17:22:36] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.14, 4.52, 3.92 [17:23:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.05, 7.57, 6.13 [17:23:22] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.67, 21.26, 18.29 [17:24:01] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.96, 5.71, 4.60 [17:24:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.63, 7.31, 6.12 [17:25:03] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.39, 6.73, 5.99 [17:26:01] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.35, 4.91, 4.44 [17:26:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.67, 3.73, 3.75 [17:28:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.73, 7.23, 6.35 [17:29:28] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.55, 22.95, 19.86 [17:30:01] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.14, 5.64, 4.91 [17:30:52]