[00:01:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 9.35, 9.84, 10.01 [00:01:39] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 1879 MB (19% inode=80%); [00:01:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.93, 9.22, 8.69 [00:02:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.58, 8.54, 8.42 [00:03:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.14, 9.45, 9.84 [00:03:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.95, 9.57, 9.57 [00:03:45] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 56 failures. Last run 2 minutes ago with 56 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:04:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.38, 4.29, 3.43 [00:04:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.19, 9.19, 8.91 [00:04:14] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.68, 8.23, 8.32 [00:04:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.59, 4.80, 3.74 [00:05:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.12, 3.46, 3.23 [00:05:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.52, 10.01, 9.98 [00:05:46] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.81, 8.40, 8.47 [00:07:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.82, 4.25, 3.56 [00:07:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.65, 9.51, 9.81 [00:08:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.95, 8.53, 8.80 [00:08:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.77, 3.42 [00:09:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.45, 3.67, 3.44 [00:11:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 4.15, 3.66 [00:11:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.29, 9.89, 9.60 [00:13:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.59, 3.70, 3.56 [00:13:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.67, 9.33, 9.48 [00:13:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.40, 9.45, 9.46 [00:13:59] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.07, 9.33, 8.64 [00:14:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.24, 3.15, 3.30 [00:14:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.89, 3.81 [00:15:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.97, 3.14, 3.38 [00:15:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.82, 9.38, 9.48 [00:15:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.70, 9.37, 8.73 [00:17:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.39, 10.02, 9.67 [00:17:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.53, 9.81, 9.20 [00:17:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.49, 8.32, 8.43 [00:19:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.75, 9.27, 9.07 [00:20:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.95, 3.82 [00:20:49] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.98, 1.95, 1.19 [00:23:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.95, 5.37, 4.20 [00:23:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.10, 9.57, 9.24 [00:25:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.74, 9.32, 9.17 [00:26:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.09, 3.45, 3.71 [00:27:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.89, 7.80, 8.48 [00:28:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 3.65, 3.74 [00:31:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.26, 8.99, 8.91 [00:31:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.76, 3.42 [00:31:39] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [00:32:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.69, 3.74 [00:33:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.26, 9.26, 8.82 [00:33:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.67, 7.90, 8.51 [00:35:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.58, 9.95, 9.41 [00:35:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.14, 9.24, 8.87 [00:35:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.71, 3.12, 3.28 [00:35:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.67, 7.60, 8.34 [00:37:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.43, 9.01, 9.14 [00:37:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.06, 3.51, 3.95 [00:37:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.19, 9.13, 9.81 [00:38:49] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.56, 1.49, 1.87 [00:40:49] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.16, 1.05, 1.66 [00:42:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.61, 3.14, 3.38 [00:43:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.22, 9.49, 9.02 [00:45:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.14, 9.12, 8.93 [00:46:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.56, 9.14, 8.69 [00:46:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.56, 3.48 [00:48:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.86, 9.75, 8.97 [00:49:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.31, 2.88, 3.39 [00:49:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.04, 9.57, 9.58 [00:50:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.59, 9.60, 9.00 [00:50:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.64, 3.57, 3.53 [00:51:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.83, 9.50, 9.54 [00:52:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 3.63, 3.25 [00:53:19] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.75, 1.73, 1.29 [00:53:32] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.54, 8.82, 8.24 [00:53:49] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.12, 9.54, 9.09 [00:54:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.86, 3.35, 3.19 [00:55:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.10, 1.44, 1.24 [00:55:32] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.02, 8.15, 8.07 [00:55:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.69, 9.26, 9.02 [00:56:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.18, 3.10, 3.39 [00:57:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.31, 9.57, 9.17 [00:57:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.62, 9.65, 9.55 [00:59:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.76, 9.50, 9.52 [00:59:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.67, 8.79, 8.43 [01:00:23] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.69, 4.37, 3.69 [01:01:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.31, 9.21, 9.14 [01:01:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.31, 3.75, 3.53 [01:03:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 4.19, 3.78 [01:03:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.25, 2.34, 1.67 [01:03:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.92, 9.71, 9.32 [01:03:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.48, 9.48, 9.09 [01:03:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.66, 3.52 [01:04:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.73, 3.98, 3.72 [01:05:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.85, 9.62, 9.14 [01:05:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.02, 3.39, 3.54 [01:05:19] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.27, 1.97, 1.62 [01:05:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.34, 9.26, 9.34 [01:05:35] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.03, 3.07, 3.32 [01:07:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.83, 9.50, 9.17 [01:07:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.48, 1.44, 1.46 [01:07:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.52, 9.20, 9.27 [01:07:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.78, 9.46, 9.19 [01:07:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.70, 8.57, 9.07 [01:07:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.68, 8.30, 8.46 [01:08:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.52, 3.10, 3.39 [01:10:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.75, 4.02, 3.72 [01:11:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.64, 10.62, 9.67 [01:11:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.38, 9.25, 9.22 [01:11:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.85, 9.93, 9.43 [01:13:27] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.62, 8.59, 8.51 [01:15:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.57, 9.80, 9.54 [01:15:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.32, 9.96, 9.63 [01:15:26] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.43, 8.06, 8.33 [01:16:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.83, 3.87, 3.83 [01:17:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.59, 9.71, 9.29 [01:21:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.69, 9.78, 9.80 [01:22:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.63, 2.93, 3.37 [01:23:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.12, 9.16, 9.23 [01:25:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.73, 10.17, 9.92 [01:25:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.30, 9.27, 9.46 [01:27:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.15, 9.34, 9.33 [01:27:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.02, 10.49, 9.86 [01:27:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.94, 4.01, 3.60 [01:27:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.69, 3.88, 3.34 [01:28:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.58, 3.52 [01:28:05] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.44, 5.16, 3.07 [01:29:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.66, 8.56, 9.32 [01:29:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.30, 3.51, 3.48 [01:29:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.48, 9.51, 9.60 [01:29:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.97, 3.66, 3.31 [01:30:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.26, 3.08, 3.34 [01:33:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.57, 7.41, 8.39 [01:33:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.85, 3.61 [01:33:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.51, 9.19, 9.31 [01:33:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.88, 4.50, 3.69 [01:34:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.26, 3.46, 3.41 [01:35:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.82, 9.06, 9.26 [01:35:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.52, 7.44, 8.36 [01:35:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.55, 8.89, 9.13 [01:37:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.36, 8.33, 8.90 [01:39:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.13, 9.22, 8.89 [01:43:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.67, 9.52, 9.13 [01:44:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.89, 3.92 [01:45:59] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.43, 1.18, 1.92 [01:47:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.70, 3.95 [01:47:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.60, 9.42, 9.23 [01:47:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.91, 9.63, 9.17 [01:48:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 4.04, 3.96 [01:49:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.91, 9.39, 9.26 [01:49:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.73, 1.51, 1.29 [01:49:59] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.60, 0.90, 1.63 [01:51:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.15, 4.32, 4.12 [01:51:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.54, 9.24, 9.09 [01:51:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.40, 1.76, 1.37 [01:51:57] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.14, 1.36, 1.26 [01:53:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.11, 9.57, 9.33 [01:53:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.16, 9.06, 9.06 [01:53:50] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.39, 1.56, 1.35 [01:57:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.27, 9.18, 9.06 [01:57:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.78, 9.62, 9.61 [01:58:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.78, 3.50, 3.88 [02:00:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.62, 4.49, 4.19 [02:01:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.52, 9.56, 9.53 [02:02:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.71, 8.39, 7.74 [02:05:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.69, 9.21, 8.62 [02:05:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.71, 9.69, 9.53 [02:05:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.11, 9.74, 9.51 [02:06:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.33, 8.08, 7.78 [02:07:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.31, 8.26, 8.36 [02:07:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.72, 3.50, 3.95 [02:07:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.71, 9.82, 9.61 [02:09:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.24, 9.69, 9.52 [02:13:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.05, 9.90, 9.60 [02:13:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.95, 9.73, 9.61 [02:14:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.74, 3.93 [02:15:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.58, 9.33, 9.25 [02:17:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.24, 8.58, 8.38 [02:17:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.46, 9.98, 9.72 [02:19:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.91, 9.30, 8.66 [02:19:43] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.52, 1.29 [02:20:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.29, 2.96, 3.78 [02:21:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.97, 8.88, 8.59 [02:21:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.63, 2.56, 3.28 [02:21:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.52, 9.89, 9.72 [02:21:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.24, 10.47, 9.79 [02:21:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.15, 9.93, 9.64 [02:21:42] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.18, 1.37, 1.26 [02:22:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.16, 2.49, 3.27 [02:23:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.41, 10.01, 9.69 [02:25:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.14, 9.54, 9.61 [02:25:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.49, 9.39, 9.50 [02:27:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.35, 9.78, 9.68 [02:27:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.14, 10.32, 9.84 [02:28:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.54, 3.68, 3.65 [02:29:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.48, 9.92, 9.75 [02:30:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.18, 3.46 [02:32:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.23, 3.28, 3.44 [02:33:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.07, 9.83, 9.14 [02:33:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.35, 9.50, 9.85 [02:34:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.27, 3.42 [02:37:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.22, 9.27, 9.08 [02:37:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:38:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.27, 4.00, 3.67 [02:38:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.71, 3.39 [02:39:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.71, 9.82, 9.96 [02:39:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.44, 9.39, 9.62 [02:39:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.21, 9.78, 9.53 [02:43:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.56, 9.43, 9.72 [02:43:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.78, 9.57, 9.55 [02:43:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:44:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.54, 3.63, 3.66 [02:44:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.21, 3.71, 3.54 [02:45:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.99, 8.96, 9.52 [02:45:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.84, 9.42, 9.70 [02:46:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.34, 2.91, 3.39 [02:46:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.22, 2.86, 3.25 [02:47:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 2.99, 6.35, 8.03 [02:47:45] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:52] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:53] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:53] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:55] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:03] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:09] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:11] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:11] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.236 second response time [02:48:14] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.010 second response time [02:48:14] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:17] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.244 second response time [02:48:18] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:19] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:25] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:26] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:35] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [02:48:41] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:45] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:46] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:47] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:55] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [02:48:55] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:12] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:14] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:15] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:18] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 3.75, 6.00, 8.16 [02:49:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 3.31, 6.04, 8.24 [02:49:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 3.93, 6.00, 8.01 [02:49:28] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.217 second response time [02:49:43] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:53] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:50:18] What the heck happened to Miraheze? [02:51:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 12.27, 4.71, 2.32 [02:52:53] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [02:53:02] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw121 [02:53:08] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.321 second response time [02:53:08] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.515 second response time [02:53:15] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.020 second response time [02:53:26] Is it just me, or is Miraheze having some sort of shit-fest with longer load periods? [02:53:38] alerting : [FIRING:2] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:53:56] I keep getting 502s [02:53:58] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.019 second response time [02:54:15] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.330 second response time [02:54:18] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.990 second response time [02:54:21] Miraheze just died. [02:54:24] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 2.448 second response time [02:54:26] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.791 second response time [02:54:32] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [02:54:33] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.102 second response time [02:54:35] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.959 second response time [02:54:36] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [02:54:36] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.311 second response time [02:54:39] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.330 second response time [02:54:40] F in chat [02:54:42] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 4 backends are down. mw101 mw111 mw112 mw121 [02:54:43] F [02:55:00] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [02:55:15] GUYS IT JUST WENT BACK UP [02:55:18] :D [02:55:34] Never mind :( [02:55:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 3 backends are down. mw101 mw102 mw122 [02:56:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.01, 3.03 [02:56:03] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.065 second response time [02:56:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.45, 3.15 [02:56:15] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.748 second response time [02:56:29] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.322 second response time [02:56:32] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [02:56:50] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [02:57:07] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [02:57:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 3.18, 2.83 [02:57:14] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.706 second response time [02:57:32] Miraheze are you ok??? [02:57:36] I am concerned. [02:57:40] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.083 second response time [02:57:56] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.843 second response time [02:58:02] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [02:58:09] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.343 second response time [02:58:12] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.022 second response time [02:58:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.65, 3.24, 3.12 [02:58:23] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 5.740 second response time [02:58:28] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:58:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:58:41] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.347 second response time [02:58:42] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [02:58:43] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.425 second response time [02:58:47] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:58:52] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.083 second response time [02:58:54] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.371 second response time [02:58:55] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:59:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.76, 2.98, 2.80 [02:59:27] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:00:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.42, 3.20 [03:00:20] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:01:19] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.54, 1.52, 1.84 [03:01:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [03:02:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 4.17, 3.54 [03:03:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.29, 3.98, 3.25 [03:05:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.68, 3.24 [03:05:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.39, 0.97, 1.55 [03:05:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.67, 8.09, 7.21 [03:05:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.56, 8.76, 7.58 [03:07:00] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.160 second response time [03:07:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.47, 8.27, 7.40 [03:07:27] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.448 second response time [03:07:45] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.166 second response time [03:08:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.93, 3.67 [03:08:32] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.186 second response time [03:08:49] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.359 second response time [03:09:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.23, 9.09, 7.99 [03:10:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.76, 4.94, 4.07 [03:11:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.31, 8.69, 7.80 [03:11:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.29, 8.74, 7.70 [03:13:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.04, 9.01, 8.01 [03:15:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.99, 3.37, 3.30 [03:15:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.04, 8.97, 8.12 [03:16:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.22, 9.38, 8.39 [03:16:56] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.04, 8.45, 7.19 [03:18:56] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.43, 8.37, 7.31 [03:19:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.52, 8.40, 8.12 [03:19:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.57, 9.38, 8.64 [03:22:54] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.20, 8.74, 7.74 [03:23:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.86, 9.74, 8.85 [03:23:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.77, 9.05, 8.48 [03:24:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.40, 3.49, 3.98 [03:24:54] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.89, 8.24, 7.69 [03:25:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.08, 8.89, 8.66 [03:25:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.10, 9.98, 9.44 [03:26:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.97, 4.09 [03:27:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.63, 8.35, 8.48 [03:27:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.48, 8.20, 8.30 [03:27:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.79, 10.09, 9.19 [03:27:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.02, 9.96, 9.48 [03:27:59] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.89, 4.05, 3.55 [03:29:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.90, 9.78, 9.20 [03:29:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.44, 9.60, 9.39 [03:31:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.73, 10.07, 9.60 [03:33:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.09, 9.24, 8.70 [03:33:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.91, 9.23, 9.35 [03:35:03] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:35:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.75, 8.14, 8.36 [03:35:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.53, 3.91, 3.81 [03:37:06] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.688 second response time [03:41:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.87, 8.43, 8.36 [03:42:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.55, 8.03, 7.51 [03:43:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.08, 9.29, 8.68 [03:43:21] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.04, 3.93, 3.75 [03:43:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.96, 9.73, 9.18 [03:43:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.20, 9.85, 9.45 [03:45:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.93, 9.60, 8.87 [03:45:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.89, 9.43, 9.14 [03:45:58] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.05, 9.01, 7.98 [03:47:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.13, 9.71, 9.28 [03:47:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.60, 9.12, 8.14 [03:48:02] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.84, 1.77, 1.17 [03:49:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.63, 3.67 [03:49:56] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.16, 1.60, 1.18 [03:51:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.15, 4.13, 3.85 [03:51:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.19, 9.90, 9.50 [03:53:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.45, 8.81, 8.17 [03:53:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.55, 8.45, 8.18 [03:57:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.18, 8.61, 8.22 [03:57:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.98, 10.02, 9.32 [03:57:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.25, 9.53, 9.84 [03:57:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.71, 8.31, 8.14 [03:59:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.26, 8.35, 8.20 [03:59:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.79, 9.13, 9.08 [04:03:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.20, 8.99, 9.08 [04:03:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.09, 8.66, 8.33 [04:05:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.69, 3.95 [04:05:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.69, 9.39, 9.23 [04:05:42] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.20, 7.78, 8.05 [04:07:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.21, 8.81, 8.48 [04:07:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.51, 9.77, 9.35 [04:07:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.17, 9.96, 9.46 [04:09:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.04, 8.40, 8.36 [04:09:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 4.41, 4.17 [04:09:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.85, 9.96, 9.48 [04:11:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.07, 3.83, 3.98 [04:11:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.06, 10.12, 9.60 [04:11:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.73, 9.62, 9.37 [04:13:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.23, 9.33, 8.75 [04:13:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.80, 9.76, 9.54 [04:15:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.55, 9.70, 9.49 [04:16:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.66, 3.33, 3.94 [04:17:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.38, 10.16, 9.71 [04:17:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.44, 10.20, 9.69 [04:19:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.63, 9.85, 9.24 [04:19:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.53, 9.90, 9.67 [04:19:22] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.52, 6.70, 5.64 [04:19:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.59, 9.27, 9.59 [04:21:22] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.24, 6.17, 5.58 [04:22:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.04, 3.29, 3.84 [04:22:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.74, 8.67, 8.18 [04:23:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.29, 9.66, 9.79 [04:24:34] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.75, 8.10, 8.04 [04:25:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.94, 2.60, 3.26 [04:25:22] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.96, 7.12, 6.08 [04:26:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.80, 2.60, 3.34 [04:26:04] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 7.85, 9.36, 5.01 [04:27:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.94, 9.26, 9.35 [04:27:22] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.09, 6.46, 5.97 [04:27:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.27, 9.10, 9.44 [04:27:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.54, 8.22, 8.05 [04:27:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw102 [04:29:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.31, 9.25, 9.33 [04:29:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.95, 8.30, 9.11 [04:29:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.75, 7.69, 7.87 [04:29:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [04:31:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.05, 10.15, 9.72 [04:31:58] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 1.05, 3.46, 3.69 [04:33:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.34, 9.71, 9.45 [04:33:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.75, 9.70, 9.62 [04:33:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.27, 9.39, 8.57 [04:33:56] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 0.87, 2.60, 3.34 [04:35:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.01, 9.42, 9.01 [04:36:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 3.43, 3.52 [04:37:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.68, 9.00, 8.91 [04:37:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.57, 9.28, 8.70 [04:39:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.90, 9.23, 9.33 [04:40:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.90, 3.29, 3.45 [04:42:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.85, 3.12, 3.37 [04:43:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.45, 7.93, 8.46 [04:45:33] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.59, 9.31, 8.83 [04:46:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 3.68, 3.52 [04:47:02] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.12, 8.86, 8.39 [04:47:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.26, 9.05, 8.76 [04:47:32] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.42, 9.12, 8.84 [04:48:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.38, 3.44 [04:48:59] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.34, 7.96, 8.12 [04:49:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.98, 9.96, 9.83 [04:51:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.90, 9.55, 9.12 [04:52:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.30, 3.33, 3.39 [04:53:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.74, 9.82, 9.86 [04:53:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.12, 9.76, 9.76 [04:55:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.48, 9.24, 8.93 [04:55:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.58, 9.93, 9.35 [04:56:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.46, 3.45 [04:56:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.62, 3.85, 3.34 [04:56:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.94, 8.63, 8.38 [04:57:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.03, 8.32, 8.63 [04:58:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.94, 3.83, 3.40 [04:58:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:58:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.49, 8.47, 8.34 [04:59:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.25, 7.89, 8.45 [04:59:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.79, 9.80, 9.46 [05:01:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.87, 10.38, 10.06 [05:01:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.69, 9.96, 9.92 [05:01:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.31, 9.58, 9.11 [05:02:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:03:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.99, 8.95, 8.93 [05:07:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.00, 7.37, 8.46 [05:07:28] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.42, 7.77, 8.46 [05:07:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.24, 4.00, 3.63 [05:10:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.35, 3.97, 3.48 [05:11:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.36, 9.60, 9.94 [05:11:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.72, 3.70, 3.62 [05:12:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 3.80, 3.49 [05:15:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.94, 3.76, 3.62 [05:16:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.38, 3.91, 3.58 [05:16:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.05, 4.26, 3.85 [05:17:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.66, 3.74, 3.63 [05:18:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.85, 3.56, 3.50 [05:18:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.94, 3.79 [05:20:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.28, 3.15, 3.35 [05:21:21] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.39, 3.07, 3.38 [05:24:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.35, 3.32, 3.44 [05:26:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.19, 3.38 [05:26:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.98, 2.69, 3.26 [05:29:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.50, 9.13, 8.96 [05:31:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.72, 9.09, 8.97 [05:37:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:41:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.63, 8.00, 8.46 [05:41:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.10, 8.49, 8.58 [05:43:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.43, 8.07, 8.43 [05:52:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:57:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:01:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.90, 9.08, 8.44 [06:03:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.54, 8.27, 8.24 [06:05:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.84, 8.57, 8.03 [06:09:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:11:39] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.17, 7.96, 7.94 [06:11:41] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:15:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.56, 8.51, 8.17 [06:17:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.67, 7.97, 7.99 [06:18:13] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.66, 1.83, 1.42 [06:19:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.25, 8.37, 7.61 [06:20:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.25, 9.31, 8.65 [06:20:12] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.55, 1.75, 1.44 [06:21:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.62, 8.65, 7.91 [06:21:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.15, 8.78, 8.33 [06:22:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.28, 1.55, 1.40 [06:23:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.84, 7.86, 7.70 [06:23:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.80, 8.89, 8.70 [06:25:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.50, 8.41, 7.93 [06:28:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.97, 8.05, 7.73 [06:29:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.80, 8.40, 8.35 [06:29:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:32:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.95, 7.98, 7.79 [06:33:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.75, 8.94, 8.59 [06:33:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.69, 7.86, 8.30 [06:35:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.24, 7.52, 8.14 [06:36:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:36:58] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:37:05] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:39:00] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.323 second response time [06:39:02] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.054 second response time [06:39:41] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:41:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:53:10] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.97, 1.99, 1.49 [06:54:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.68, 8.33, 7.65 [06:55:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.21, 8.49, 7.83 [06:55:59] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.69, 7.98, 7.60 [06:57:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.41, 9.09, 8.12 [06:57:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:59:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.11, 9.07, 8.24 [06:59:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.34, 8.91, 8.09 [06:59:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.49, 8.67, 7.99 [07:01:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.50, 8.50, 8.05 [07:03:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.92, 7.94, 7.88 [07:07:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.48, 8.01, 8.25 [07:07:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:10:49] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.33, 1.96, 1.97 [07:18:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.50, 8.95, 8.39 [07:18:49] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.31, 1.37, 1.67 [07:20:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.18, 8.70, 8.20 [07:21:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:22:11] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.96, 8.46, 8.17 [07:26:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:28:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.23, 8.43, 8.22 [07:29:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.17, 8.39, 8.47 [07:30:00] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.93, 8.37, 8.23 [07:33:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.32, 8.67, 8.58 [07:33:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.79, 9.26, 8.62 [07:35:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.18, 7.99, 8.34 [07:35:48] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.65, 8.44, 8.41 [07:36:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:41:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:44:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.32, 9.18, 8.69 [07:46:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.37, 9.67, 8.93 [07:48:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.95, 9.16, 8.81 [07:50:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.66, 9.00, 8.48 [07:51:49] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.60, 9.26, 8.88 [07:52:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.95, 9.13, 8.61 [07:53:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.35, 3.53, 2.73 [07:53:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:53:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.72, 9.17, 8.88 [07:54:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.09, 8.31, 8.37 [07:55:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.94, 2.90, 2.59 [07:57:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.03, 9.16, 8.90 [07:57:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.56, 8.67, 8.00 [07:58:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.28, 9.12, 8.70 [07:58:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:59:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.03, 8.63, 8.73 [08:00:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.44, 8.72, 8.17 [08:01:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.50, 8.25, 8.00 [08:03:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:03:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.43, 10.15, 9.14 [08:05:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.27, 8.80, 8.30 [08:05:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.42, 9.47, 9.02 [08:05:54] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.90, 8.20, 8.17 [08:08:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:09:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.98, 9.60, 9.12 [08:13:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.14, 9.19, 9.13 [08:15:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.12, 9.52, 9.09 [08:16:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.75, 8.10, 7.31 [08:16:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:17:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.26, 9.78, 9.39 [08:17:55] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.75, 8.39, 7.41 [08:18:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.01, 9.12, 7.78 [08:19:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.36, 8.27, 8.46 [08:20:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.58, 8.89, 7.86 [08:20:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.67, 9.35, 8.62 [08:21:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.26, 9.42, 9.28 [08:21:54] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.97, 8.24, 7.58 [08:22:20] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.38, 8.34, 7.78 [08:23:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.11, 9.98, 9.50 [08:24:17] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.35, 8.40, 8.39 [08:25:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.79, 9.85, 9.50 [08:27:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.80, 9.59, 9.51 [08:31:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:37:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:39:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.75, 7.70, 8.42 [08:40:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.59, 2.79 [08:42:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.98, 3.26, 2.76 [08:42:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:43:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.05, 8.23, 8.52 [08:47:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.82, 7.92, 8.31 [08:49:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.67, 7.56, 8.49 [08:49:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:51:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.53, 8.75, 8.60 [08:51:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.89, 8.04, 7.59 [08:53:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.13, 7.97, 7.62 [08:55:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.99, 10.40, 9.25 [08:57:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.06, 7.86, 7.28 [08:57:03] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.67, 1.84, 1.26 [08:57:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.67, 8.37, 8.48 [08:57:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.17, 9.60, 9.10 [08:59:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.23, 7.33, 7.16 [08:59:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.86, 1.55, 1.23 [08:59:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.83, 7.97, 8.31 [08:59:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:05:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:10:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:11:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.64, 7.67, 8.44 [09:14:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:19:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.52, 8.12, 7.55 [09:19:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.03, 8.55, 8.46 [09:19:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:21:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.79, 8.08, 7.61 [09:21:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.16, 8.21, 8.33 [09:25:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.48, 8.56, 8.45 [09:25:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:29:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.51, 9.40, 8.80 [09:31:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.32, 8.60, 8.58 [09:31:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.69, 8.43, 7.94 [09:33:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.07, 9.66, 8.44 [09:34:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.33, 8.51, 7.87 [09:35:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.39, 8.93, 8.12 [09:35:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.61, 9.75, 8.64 [09:36:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.68, 7.86, 7.71 [09:37:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.57, 8.35, 8.00 [09:39:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.83, 8.46, 8.40 [09:39:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.22, 9.75, 9.04 [09:40:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:43:04] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/aab22cc3d443...588c5c9d2814 [09:43:05] [02miraheze/puppet] 07JohnFLewis 03588c5c9 - Varnish: add start of line to regex [09:43:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.47, 9.15, 9.03 [09:49:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.22, 8.85, 8.48 [09:49:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:51:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.54, 8.79, 8.23 [09:51:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.24, 9.32, 8.70 [09:53:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.86, 9.14, 8.71 [10:01:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.04, 7.78, 8.13 [10:01:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.33, 7.37, 8.37 [10:03:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.16, 8.75, 8.71 [10:05:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.72, 8.91, 8.79 [10:09:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:11:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.68, 7.98, 8.37 [10:19:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.54, 9.08, 8.60 [10:21:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.63, 8.59, 8.49 [10:21:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.28, 8.96, 8.19 [10:22:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.47, 3.94, 3.04 [10:23:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.30, 8.69, 8.21 [10:24:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.47, 2.97 [10:25:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:25:51] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[postgresql-contrib-13] [10:26:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.20, 3.07, 2.89 [10:27:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.58, 9.71, 9.05 [10:27:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.06, 8.43, 8.20 [10:29:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.02, 9.40, 9.01 [10:35:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.33, 8.92, 8.39 [10:37:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.79, 9.96, 9.32 [10:37:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.59, 8.94, 8.49 [10:39:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.15, 9.30, 9.16 [10:41:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.70, 9.97, 8.93 [10:43:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.60, 9.30, 8.81 [10:45:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.36, 8.26, 7.90 [10:45:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:47:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.00, 8.05, 8.42 [10:47:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.02, 7.78, 7.77 [10:51:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.53, 8.05, 7.48 [10:51:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.57, 9.10, 8.75 [10:53:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.82, 7.94, 7.52 [10:53:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.93, 8.70, 8.62 [10:53:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:53:51] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [11:01:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.90, 9.87, 9.14 [11:05:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.51, 7.91, 8.39 [11:07:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.11, 9.79, 9.40 [11:09:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.38, 9.04, 8.76 [11:11:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.46, 2.83 [11:11:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.60, 10.80, 9.45 [11:12:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.50, 3.12, 2.78 [11:13:22] [02mw-config] 07ugochimobi synchronize pull request 03#4502: Wikibase: Adding missing setups in favour of GD & GP - 13https://github.com/miraheze/mw-config/pull/4502 [11:14:40] miraheze/mw-config - ugochimobi the build passed. [11:17:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.72, 9.60, 9.40 [11:18:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.10, 9.44, 8.48 [11:18:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:19:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.86, 10.24, 9.67 [11:19:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.65, 8.94, 8.14 [11:20:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.49, 9.42, 8.62 [11:21:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.07, 9.00, 8.26 [11:22:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.25, 9.80, 8.83 [11:22:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:23:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.08, 9.45, 8.52 [11:25:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.34, 9.44, 8.63 [11:25:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.74, 10.08, 9.50 [11:25:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.75, 8.82, 7.63 [11:27:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.81, 8.56, 7.69 [11:28:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.09, 9.82, 9.19 [11:29:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.03, 8.18, 7.65 [11:31:09] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.05, 8.94, 7.99 [11:32:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.12, 10.64, 9.64 [11:35:08] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.35, 8.43, 8.05 [11:37:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.11, 9.42, 9.43 [11:41:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.92, 9.89, 9.21 [11:42:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.55, 8.70, 8.19 [11:44:45] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.35, 9.20, 8.42 [11:45:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.68, 9.13, 9.09 [11:45:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.59, 8.71, 8.24 [11:46:41] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.66, 9.10, 8.46 [11:49:04] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.80, 8.32, 8.21 [11:51:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.45, 9.46, 9.21 [11:51:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.48, 9.66, 9.44 [11:56:24] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.56, 8.32, 8.39 [11:57:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.88, 9.79, 9.56 [11:57:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.26, 8.85, 9.27 [11:59:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.26, 10.09, 9.70 [11:59:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.19, 9.95, 9.63 [12:00:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.95, 8.79, 8.58 [12:01:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.06, 8.87, 9.29 [12:01:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.85, 9.92, 8.61 [12:01:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.02, 9.28, 9.44 [12:03:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.83, 10.04, 9.68 [12:05:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.34, 9.12, 8.58 [12:05:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.28, 9.49, 9.51 [12:05:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.63, 8.95, 9.84 [12:06:08] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.39, 7.56, 8.18 [12:07:00] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.97, 8.16, 8.28 [12:07:55] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:15] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:18] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:22] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:09:58] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.375 second response time [12:10:14] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.995 second response time [12:10:18] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.843 second response time [12:10:23] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.650 second response time [12:11:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.59, 10.01, 9.90 [12:13:51] [02mw-config] 07Reception123 reviewed pull request 03#4502 commit - 13https://github.com/miraheze/mw-config/pull/4502#discussion_r824659520 [12:13:52] [02mw-config] 07Reception123 reviewed pull request 03#4502 commit - 13https://github.com/miraheze/mw-config/pull/4502#discussion_r824659735 [12:13:54] [02mw-config] 07Reception123 reviewed pull request 03#4502 commit - 13https://github.com/miraheze/mw-config/pull/4502#discussion_r824659360 [12:13:55] [02mw-config] 07Reception123 reviewed pull request 03#4502 commit - 13https://github.com/miraheze/mw-config/pull/4502#discussion_r824659864 [12:23:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.75, 8.53, 9.76 [12:23:22] [02mw-config] 07ugochimobi reviewed pull request 03#4502 commit - 13https://github.com/miraheze/mw-config/pull/4502#discussion_r824666385 [12:24:49] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.78, 1.83, 1.41 [12:25:00] [02mw-config] 07ugochimobi reviewed pull request 03#4502 commit - 13https://github.com/miraheze/mw-config/pull/4502#discussion_r824667537 [12:26:49] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.44, 1.69, 1.41 [12:29:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.73, 8.10, 8.47 [12:29:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.91, 7.67, 8.40 [12:31:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.35, 9.27, 9.80 [12:32:06] [02mw-config] 07ugochimobi reviewed pull request 03#4502 commit - 13https://github.com/miraheze/mw-config/pull/4502#discussion_r824672280 [12:32:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:37:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.78, 8.48, 8.40 [12:37:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.14, 2.79 [12:39:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.91, 8.15, 8.28 [12:39:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.66, 3.53, 2.97 [12:39:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:41:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.48, 3.01 [12:41:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.48, 10.58, 9.95 [12:43:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.46, 3.77, 3.17 [12:43:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.50, 9.13, 8.88 [12:44:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.48, 3.12 [12:46:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.44, 3.14, 3.04 [12:49:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.04, 9.37, 8.75 [12:49:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.71, 9.82, 9.27 [12:49:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.44, 9.69, 9.77 [12:49:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.45, 8.57, 8.16 [12:51:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.78, 9.09, 8.75 [12:52:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.36, 3.89, 3.40 [12:53:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.03, 9.38, 8.88 [12:53:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.47, 10.07, 9.88 [12:53:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.76, 8.48, 8.25 [12:54:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 3.56, 3.33 [12:55:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.88, 9.50, 8.98 [12:56:23] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.50, 3.85, 3.45 [12:57:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.56, 8.53, 8.39 [12:58:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.86, 3.54, 3.39 [12:59:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.14, 9.44, 9.21 [13:01:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.70, 9.24, 9.18 [13:01:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.19, 9.02, 8.59 [13:03:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.94, 9.51, 9.26 [13:03:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.55, 9.08, 8.66 [13:04:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.22, 4.50, 3.79 [13:05:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.86, 7.87, 8.27 [13:06:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.28, 3.72, 3.60 [13:07:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.86, 9.49, 9.38 [13:09:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.96, 9.46, 9.94 [13:09:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:11:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.92, 10.16, 10.14 [13:15:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.83, 9.85, 9.48 [13:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:16:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.95, 3.14, 3.37 [13:18:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.62, 9.07, 8.62 [13:19:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.43, 8.97, 8.80 [13:19:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.75, 9.72, 9.98 [13:22:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.95, 9.88, 9.05 [13:24:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.84, 9.79, 9.12 [13:25:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.76, 9.83, 9.43 [13:26:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.90, 8.29, 7.89 [13:27:52] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.36, 9.63, 9.16 [13:29:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.44, 9.48, 9.71 [13:29:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.37, 9.29, 9.10 [13:30:50] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.72, 8.97, 8.23 [13:31:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.22, 9.28, 9.61 [13:31:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.63, 9.71, 9.27 [13:32:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.98, 8.85, 8.27 [13:33:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.05, 9.49, 9.24 [13:35:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:37:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.06, 9.69, 9.65 [13:37:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.44, 9.88, 9.45 [13:38:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.10, 7.89, 8.13 [13:38:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:39:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.90, 8.70, 9.30 [13:39:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.98, 9.61, 9.40 [13:41:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.22, 7.46, 8.35 [13:41:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.16, 9.56, 9.53 [13:43:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.92, 9.16, 9.40 [13:45:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.94, 10.04, 9.71 [13:45:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.35, 9.06, 8.56 [13:49:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.49, 9.20, 8.77 [13:49:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.27, 9.32, 9.11 [13:51:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.19, 8.78, 8.58 [13:53:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.59, 9.69, 9.33 [13:55:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.93, 7.89, 8.35 [13:55:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.10, 9.74, 9.38 [13:56:57] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [13:57:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.10, 9.61, 9.39 [13:57:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.40, 10.31, 9.25 [13:59:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.38, 9.80, 9.21 [14:00:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.91, 9.18, 8.34 [14:02:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.32, 8.81, 8.31 [14:06:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.70, 8.41, 8.31 [14:07:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.34, 9.14, 9.04 [14:08:50] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.07, 8.02, 8.16 [14:11:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.56, 9.48, 9.23 [14:12:28] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.12, 7.89, 8.12 [14:15:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.91, 7.66, 8.48 [14:16:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.58, 8.55, 8.35 [14:19:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.10, 10.01, 9.39 [14:20:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.07, 9.14, 8.63 [14:20:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.95, 8.54, 8.34 [14:21:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.05, 9.30, 9.21 [14:22:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.62, 8.86, 8.59 [14:22:30] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.48, 8.11, 8.21 [14:25:42] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [14:26:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.09, 8.96, 8.66 [14:27:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.76, 9.30, 9.16 [14:28:05] [02mw-config] 07ugochimobi opened pull request 03#4503: PropertySuggester/APH configs for GD/GP - 13https://github.com/miraheze/mw-config/pull/4503 [14:28:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.73, 9.22, 8.79 [14:29:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.33, 8.43, 8.27 [14:29:13] miraheze/mw-config - ugochimobi the build passed. [14:29:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.07, 9.42, 9.24 [14:30:23] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.82, 9.78, 9.05 [14:31:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.79, 8.03, 8.14 [14:32:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.18, 8.87, 8.82 [14:33:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.52, 9.28, 9.89 [14:34:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.36, 2.77 [14:35:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.07, 10.18, 10.16 [14:36:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.91, 3.35, 2.83 [14:37:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.69, 9.07, 8.51 [14:37:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.71, 9.68, 9.33 [14:38:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 3.57, 2.96 [14:38:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.02, 7.62, 8.35 [14:40:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.67, 3.27, 2.88 [14:41:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.60, 2.90 [14:41:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.92, 9.31, 9.31 [14:42:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.02, 3.27, 2.93 [14:43:17] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.27, 3.33, 2.88 [14:43:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.38, 9.65, 9.42 [14:45:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.64, 9.19, 9.30 [14:46:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 3.62, 3.41 [14:48:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.17, 4.43, 3.72 [14:50:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.61, 3.14 [14:50:17] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.73, 3.62, 3.20 [14:51:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.90, 9.59, 9.98 [14:52:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.79, 2.93, 2.95 [14:52:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.13, 3.41, 3.49 [14:53:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.52, 2.89, 2.98 [14:54:30] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.092 second response time [14:55:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.76, 10.14, 10.08 [14:57:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.10, 9.20, 8.93 [14:59:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.03, 9.31, 9.02 [15:00:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.70, 4.12, 3.79 [15:00:16] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.75, 9.08, 8.34 [15:00:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.88, 8.97, 8.43 [15:02:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.08, 4.30, 3.40 [15:02:16] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.71, 8.25, 8.13 [15:03:03] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.64, 3.30 [15:04:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.07, 8.37, 8.30 [15:05:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.82, 10.04, 9.39 [15:05:02] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.868 second response time [15:05:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.15, 3.13, 3.16 [15:06:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.38, 3.33, 3.22 [15:06:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.32, 3.78, 3.87 [15:07:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.63, 9.84, 9.40 [15:09:23] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.01, 8.82, 9.80 [15:09:31] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:11:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.42, 9.82, 10.06 [15:11:28] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.503 second response time [15:11:29] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.919 second response time [15:12:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.64, 3.70 [15:14:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.50, 3.29 [15:16:06] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.16, 8.72, 8.39 [15:16:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.51, 3.63, 3.68 [15:16:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.27, 3.00, 3.13 [15:17:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.78, 7.66, 8.43 [15:20:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.93, 8.07, 7.64 [15:21:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.18, 8.39, 8.58 [15:21:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.57, 8.43, 8.40 [15:22:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.87, 3.08, 3.40 [15:25:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.03, 8.92, 8.86 [15:26:10] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.06, 8.39, 7.95 [15:27:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.58, 9.06, 8.91 [15:27:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.06, 8.67, 8.50 [15:28:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.95, 3.32, 3.05 [15:29:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.28, 9.58, 9.13 [15:29:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.98, 8.01, 8.31 [15:30:07] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.29, 3.31, 3.07 [15:30:08] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.01, 8.86, 8.22 [15:31:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.58, 9.50, 9.18 [15:33:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.22, 10.00, 9.41 [15:33:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.13, 8.51, 8.47 [15:34:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.47, 9.56, 8.68 [15:35:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.80, 8.57, 9.66 [15:37:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.65, 9.12, 9.71 [15:37:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.19, 9.28, 9.29 [15:37:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.13, 7.95, 8.25 [15:38:06] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.31, 8.12, 8.27 [15:39:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.39, 9.01, 9.59 [15:41:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.01, 9.30, 8.87 [15:41:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.13, 10.25, 9.98 [15:41:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.92, 9.68, 9.39 [15:41:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.82, 9.25, 8.69 [15:42:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.33, 8.69, 8.46 [15:43:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.74, 8.72, 8.71 [15:43:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.62, 9.24, 9.26 [15:43:47] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.85, 1.80, 1.58 [15:43:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.24, 8.73, 8.56 [15:45:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.49, 2.11, 1.72 [15:45:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.30, 7.99, 8.32 [15:46:16] [02mw-config] 07ugochimobi synchronize pull request 03#4502: Wikibase: Adding missing setups in favour of GD & GP - 13https://github.com/miraheze/mw-config/pull/4502 [15:46:18] [02mw-config] 07ugochimobi synchronize pull request 03#4502: Wikibase: Adding missing setups in favour of GD & GP - 13https://github.com/miraheze/mw-config/pull/4502 [15:47:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.88, 9.31, 9.96 [15:47:20] miraheze/mw-config - ugochimobi the build passed. [15:47:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.50, 9.88, 9.46 [15:47:28] miraheze/mw-config - ugochimobi the build passed. [15:47:45] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.02, 1.74, 1.64 [15:49:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.15, 8.92, 8.74 [15:49:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.21, 9.62, 9.99 [15:49:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.34, 3.08 [15:49:45] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.85, 1.46, 1.55 [15:50:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.63, 8.31, 8.42 [15:50:50] [02mw-config] 07ugochimobi opened pull request 03#4504: Update Wikibase.php - 13https://github.com/miraheze/mw-config/pull/4504 [15:51:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.36, 8.83, 8.72 [15:51:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.69, 9.56, 9.92 [15:51:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.79, 3.12, 3.04 [15:51:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.64, 9.46, 8.84 [15:51:57] miraheze/mw-config - ugochimobi the build passed. [15:53:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.08, 9.29, 9.40 [15:59:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.00, 10.03, 9.89 [15:59:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.20, 7.92, 8.38 [16:01:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.48, 9.91, 9.43 [16:02:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.47, 8.85, 8.55 [16:03:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.48, 9.29, 9.26 [16:05:59] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.42, 8.50, 8.47 [16:06:17] [02mw-config] 07ugochimobi synchronize pull request 03#4502: Wikibase: Adding missing setups in favour of GD & GP - 13https://github.com/miraheze/mw-config/pull/4502 [16:07:25] miraheze/mw-config - ugochimobi the build passed. [16:09:22] [02mw-config] 07ugochimobi synchronize pull request 03#4502: Wikibase: Adding missing setups in favour of GD & GP - 13https://github.com/miraheze/mw-config/pull/4502 [16:10:48] miraheze/mw-config - ugochimobi the build passed. [16:12:16] [02mw-config] 07ugochimobi synchronize pull request 03#4504: Update Wikibase.php - 13https://github.com/miraheze/mw-config/pull/4504 [16:13:38] miraheze/mw-config - ugochimobi the build passed. [16:16:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.56, 3.63, 3.02 [16:18:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.78, 3.21, 2.93 [16:19:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.30, 9.57, 9.92 [16:22:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.58, 3.13 [16:23:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.86, 10.29, 10.14 [16:25:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.00, 7.45, 8.23 [16:25:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.63, 8.46, 9.45 [16:25:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.71, 9.16, 9.84 [16:25:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.37, 7.77, 8.47 [16:25:36] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:45] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:46] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:53] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:26:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 5.94, 4.22 [16:27:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.34, 5.23, 3.88 [16:27:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.90, 9.75, 9.96 [16:27:33] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.321 second response time [16:27:33] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.06, 1.53, 1.09 [16:27:42] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.020 second response time [16:27:44] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.022 second response time [16:27:49] PROBLEM - www.mwcosmos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mwcosmos.com' expires in 15 day(s) (Sun 27 Mar 2022 16:12:54 GMT +0000). [16:27:49] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.036 second response time [16:29:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.89, 1.31, 1.07 [16:30:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.76, 3.69, 3.68 [16:31:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.37, 3.48, 3.45 [16:31:14] PROBLEM - datacrondatabase.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'datacrondatabase.com' expires in 15 day(s) (Sun 27 Mar 2022 16:18:38 GMT +0000). [16:31:32] PROBLEM - mwcosmos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mwcosmos.com' expires in 15 day(s) (Sun 27 Mar 2022 16:12:54 GMT +0000). [16:33:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.72, 3.28, 3.38 [16:33:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.72, 9.33, 9.39 [16:33:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0eeb810a6c43...97b64cca0050 [16:33:47] [02miraheze/ssl] 07MirahezeSSLBot 0397b64cc - Bot: Update SSL cert for datacrondatabase.com [16:34:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.79, 3.02, 3.40 [16:35:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.26, 9.52, 9.47 [16:37:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.35, 8.89, 8.63 [16:37:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.50, 9.54, 9.91 [16:37:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/97b64cca0050...7447d731800e [16:37:34] [02miraheze/ssl] 07MirahezeSSLBot 037447d73 - Bot: Update SSL cert for mwcosmos.com [16:39:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.52, 8.25, 8.43 [16:39:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.53, 9.79, 9.95 [16:39:55] PROBLEM - bluearchive.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bluearchive.wiki' expires in 15 day(s) (Sun 27 Mar 2022 16:25:38 GMT +0000). [16:40:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.15, 3.47, 4.00 [16:42:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.52, 3.83, 4.09 [16:44:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.69, 3.39, 3.89 [16:45:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.69, 9.40, 9.18 [16:45:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7447d731800e...b294ee5e3e55 [16:45:54] [02miraheze/ssl] 07MirahezeSSLBot 03b294ee5 - Bot: Update SSL cert for bluearchive.wiki [16:47:39] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 1035 MB (10% inode=80%); [16:51:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.19, 8.71, 8.31 [16:51:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.91, 9.04, 8.59 [16:53:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.85, 8.81, 8.41 [16:54:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.93, 2.73, 3.38 [16:55:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.62, 9.01, 8.27 [16:56:37] RECOVERY - www.mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mwcosmos.com' will expire on Thu 09 Jun 2022 15:37:28 GMT +0000. [16:58:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.93, 4.42, 3.84 [16:58:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.69, 3.35, 2.96 [16:59:19] PROBLEM - www.fortis.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fortis.wiki' expires in 15 day(s) (Sun 27 Mar 2022 16:43:56 GMT +0000). [16:59:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.79, 9.14, 9.60 [16:59:25] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.49, 3.87, 3.21 [16:59:30] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.89, 8.36, 8.18 [17:00:28] RECOVERY - datacrondatabase.com - LetsEncrypt on sslhost is OK: OK - Certificate 'datacrondatabase.com' will expire on Thu 09 Jun 2022 15:33:39 GMT +0000. [17:00:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.36, 3.01 [17:01:03] RECOVERY - mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mwcosmos.com' will expire on Thu 09 Jun 2022 15:37:28 GMT +0000. [17:01:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.69, 9.78, 9.77 [17:01:21] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.69, 3.32, 3.08 [17:01:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.85, 9.44, 8.99 [17:02:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.85, 3.22, 3.00 [17:03:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.17, 9.26, 8.61 [17:03:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.82, 8.87, 7.84 [17:04:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.61, 3.84, 3.85 [17:04:25] PROBLEM - factorio-mods.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'factorio-mods.tk' expires in 15 day(s) (Sun 27 Mar 2022 16:32:53 GMT +0000). [17:05:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.12, 9.75, 9.27 [17:07:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.72, 7.67, 8.20 [17:07:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.46, 8.42, 8.40 [17:07:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.93, 9.06, 9.84 [17:07:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.78, 7.63, 7.60 [17:08:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.86, 4.04, 3.86 [17:08:51] RECOVERY - bluearchive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bluearchive.wiki' will expire on Thu 09 Jun 2022 15:45:47 GMT +0000. [17:09:00] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b294ee5e3e55...4d2ae15c3f15 [17:09:02] [02miraheze/ssl] 07MirahezeSSLBot 034d2ae15 - Bot: Update SSL cert for factorio-mods.tk [17:09:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.29, 9.84, 9.89 [17:10:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.42, 3.19 [17:11:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.84, 10.22, 10.03 [17:11:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.42, 9.34, 9.68 [17:12:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.54, 3.07, 3.08 [17:15:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.40, 7.63, 8.36 [17:15:33] PROBLEM - fortis.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fortis.wiki' expires in 15 day(s) (Sun 27 Mar 2022 16:43:56 GMT +0000). [17:18:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4d2ae15c3f15...cd0453df38be [17:18:08] [02miraheze/ssl] 07MirahezeSSLBot 03cd0453d - Bot: Update SSL cert for fortis.wiki [17:20:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.78, 3.87 [17:21:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.07, 9.69, 9.95 [17:21:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.37, 9.63, 8.97 [17:23:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.06, 9.91, 10.00 [17:25:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.30, 9.08, 8.91 [17:26:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.90, 2.85, 3.40 [17:28:09] RECOVERY - www.fortis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fortis.wiki' will expire on Thu 09 Jun 2022 16:18:00 GMT +0000. [17:29:19] PROBLEM - www.wiki.mirahezebots.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mirahezebots.org' expires in 15 day(s) (Sun 27 Mar 2022 17:25:54 GMT +0000). [17:33:50] RECOVERY - factorio-mods.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'factorio-mods.tk' will expire on Thu 09 Jun 2022 16:08:55 GMT +0000. [17:34:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.49, 3.42 [17:35:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.38, 8.98, 9.84 [17:35:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.54, 7.65, 8.42 [17:37:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.97, 9.68, 9.97 [17:38:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:39:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.68, 9.67, 9.94 [17:41:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.39, 9.89, 9.98 [17:41:44] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [17:41:49] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:05] RECOVERY - fortis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fortis.wiki' will expire on Thu 09 Jun 2022 16:18:00 GMT +0000. [17:45:56] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.857 second response time [17:46:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.65, 3.43, 3.81 [17:50:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.83, 4.18, 3.98 [17:50:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.59, 8.62, 8.41 [17:51:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.93, 9.61, 9.96 [17:51:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.59, 8.88, 8.14 [17:52:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.09, 8.39, 8.35 [17:52:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 3.59, 3.13 [17:53:40] PROBLEM - wiki.mirahezebots.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mirahezebots.org' expires in 15 day(s) (Sun 27 Mar 2022 17:25:54 GMT +0000). [17:54:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.63, 3.20 [17:55:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.67, 9.86, 9.94 [17:55:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.97, 7.73, 7.86 [17:56:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.77, 8.77, 8.52 [17:56:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.12, 3.08, 3.05 [17:59:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.46, 8.54, 8.21 [17:59:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cd0453df38be...798a434086a8 [17:59:59] [02miraheze/ssl] 07MirahezeSSLBot 03798a434 - Bot: Update SSL cert for wiki.mirahezebots.org [18:00:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.90, 9.51, 8.66 [18:01:28] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.67, 9.04, 8.41 [18:02:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.97, 3.29, 3.83 [18:02:04] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.60, 9.93, 8.91 [18:03:27] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.53, 8.37, 8.24 [18:03:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.42, 9.34, 8.88 [18:07:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.21, 9.95, 9.33 [18:07:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.98, 9.90, 9.25 [18:09:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.43, 8.97, 8.11 [18:09:43] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:11:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.78, 9.93, 9.43 [18:11:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.79, 8.68, 8.12 [18:13:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.61, 9.76, 9.43 [18:13:34] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.97, 8.42, 8.09 [18:14:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.36, 3.99, 3.84 [18:15:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.15, 9.87, 9.51 [18:15:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.55, 9.62, 9.22 [18:16:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.69, 3.75 [18:17:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.12, 9.44, 9.92 [18:17:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.81, 9.80, 9.34 [18:17:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.80, 8.82, 8.30 [18:20:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 4.33, 3.97 [18:20:49] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.76, 1.65, 1.22 [18:21:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.12, 9.91, 9.97 [18:22:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.61, 3.63, 3.76 [18:22:49] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.62, 2.02, 1.41 [18:23:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.74, 7.72, 8.05 [18:23:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.77, 9.68, 9.87 [18:24:49] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.66, 1.89, 1.44 [18:26:49] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.41, 1.70, 1.43 [18:27:19] RECOVERY - www.wiki.mirahezebots.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mirahezebots.org' will expire on Thu 09 Jun 2022 16:59:52 GMT +0000. [18:27:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.88, 10.06, 9.96 [18:27:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.74, 9.98, 9.98 [18:29:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.68, 10.04, 10.01 [18:31:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.19, 9.89, 9.97 [18:33:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.56, 9.61, 9.94 [18:34:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.75, 3.77, 3.62 [18:35:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.25, 9.50, 9.36 [18:35:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.05, 9.55, 9.87 [18:36:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.46, 3.52 [18:37:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.72, 9.44, 9.37 [18:37:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.73, 9.78, 9.92 [18:38:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.44, 3.05, 3.36 [18:39:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.51, 10.23, 10.06 [18:40:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.53, 9.07, 8.29 [18:42:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.20, 8.99, 8.36 [18:44:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.23, 3.68, 3.50 [18:45:49] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.29, 1.77, 1.36 [18:46:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 3.57, 3.48 [18:46:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.83, 4.24, 3.25 [18:46:39] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.90, 8.28, 8.24 [18:47:47] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.78, 1.45, 1.29 [18:49:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.35, 7.70, 8.44 [18:49:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.07, 7.62, 8.38 [18:51:58] RECOVERY - wiki.mirahezebots.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mirahezebots.org' will expire on Thu 09 Jun 2022 16:59:52 GMT +0000. [18:52:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.97, 3.89, 3.60 [18:53:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.72, 9.29, 8.90 [18:54:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.81, 3.60 [18:54:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.09, 8.20, 7.78 [18:54:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.97, 3.52 [18:54:49] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.79, 1.67, 1.47 [18:55:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.74, 9.92, 9.12 [18:56:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 4.14, 3.64 [18:56:15] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [18:56:15] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:56:49] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.59, 1.62, 1.47 [18:57:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.27, 9.59, 9.16 [18:58:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.70, 3.53 [18:59:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.02, 10.26, 9.46 [18:59:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.67, 9.84, 9.29 [19:00:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.61, 3.21, 3.40 [19:00:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.46, 9.44, 8.47 [19:00:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.52, 3.17, 3.35 [19:01:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.94, 9.72, 9.36 [19:01:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.20, 9.34, 9.99 [19:02:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.20, 8.70, 8.33 [19:04:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.82, 4.73, 4.04 [19:04:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.12, 7.48, 7.93 [19:05:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.29, 9.53, 9.32 [19:05:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.32, 9.77, 9.40 [19:07:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.74, 8.93, 9.13 [19:07:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.67, 9.35, 9.29 [19:09:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.73, 9.54, 9.76 [19:09:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.26, 9.72, 9.43 [19:09:45] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.66, 3.28, 2.15 [19:11:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.74, 9.24, 9.63 [19:11:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.37, 9.73, 9.48 [19:12:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.81, 8.49, 8.22 [19:12:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.77, 3.84 [19:13:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.28, 9.91, 9.58 [19:13:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.01, 8.18, 8.15 [19:14:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 3.77, 3.81 [19:15:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.44, 9.62, 9.67 [19:16:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.98, 3.05, 3.54 [19:17:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.02, 9.13, 9.02 [19:17:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.40, 9.43, 9.59 [19:17:24] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.79, 1.59, 1.77 [19:19:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.96, 1.43, 1.70 [19:20:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.31, 3.52 [19:21:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.35, 9.40, 9.19 [19:21:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.87, 9.75, 9.71 [19:22:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.01, 2.75, 3.28 [19:24:59] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [19:27:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.60, 10.12, 9.80 [19:29:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.89, 9.81, 9.74 [19:30:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.84, 4.67, 3.85 [19:31:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.49, 3.67, 2.90 [19:31:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.97, 9.17, 9.93 [19:31:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 3.72, 3.08 [19:31:45] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.86, 2.95, 2.73 [19:33:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.87, 9.87, 10.09 [19:33:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.48, 3.05 [19:33:44] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.233 second response time [19:34:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.39, 3.43, 3.54 [19:35:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.85, 9.18, 9.09 [19:35:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.69, 2.87, 2.88 [19:35:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.50, 9.33, 9.89 [19:36:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.34, 2.75, 3.27 [19:37:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.27, 9.99, 10.05 [19:38:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.25, 8.61, 8.30 [19:38:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:39:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.91, 9.08, 9.08 [19:40:02] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.92, 8.06, 8.14 [19:41:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.90, 9.54, 9.32 [19:41:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.44, 9.25, 9.31 [19:44:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:45:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.12, 9.01, 9.26 [19:47:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.95, 9.44, 9.08 [19:49:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.59, 9.99, 9.55 [19:51:03] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.05, 2.05, 1.33 [19:51:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.78, 9.77, 9.76 [19:53:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.36, 9.71, 9.35 [19:53:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.87, 1.65, 1.27 [19:53:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.09, 10.16, 9.89 [19:54:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.43, 3.25 [19:56:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.48, 3.01, 3.11 [19:59:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.32, 9.60, 9.74 [20:00:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.76, 3.64, 3.35 [20:00:57] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.42, 8.94, 8.38 [20:01:59] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.03, 1.90, 1.26 [20:02:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.76, 2.94, 3.13 [20:03:59] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.30, 1.69, 1.26 [20:05:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.29, 10.21, 9.61 [20:07:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.71, 9.69, 9.51 [20:09:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.18, 9.75, 9.54 [20:10:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.69, 4.00, 3.52 [20:10:54] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.98, 7.72, 8.13 [20:11:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.17, 9.36, 9.41 [20:11:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.37, 3.06 [20:12:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.63, 3.36, 3.34 [20:13:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.53, 3.03, 2.97 [20:13:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.15, 9.48, 9.41 [20:15:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.32, 9.37, 9.38 [20:16:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.34, 8.64, 8.41 [20:17:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.94, 3.74, 3.27 [20:17:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.15, 10.32, 9.72 [20:17:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.07, 8.86, 8.33 [20:18:52] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.23, 8.49, 8.38 [20:19:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.28, 3.17, 3.12 [20:21:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.53, 9.73, 9.49 [20:23:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.99, 9.02, 9.26 [20:24:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.85, 3.81, 3.42 [20:24:49] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.87, 1.52, 0.94 [20:25:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.86, 4.00, 3.07 [20:25:54] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:12] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:35] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:38] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:49] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.78, 1.19, 0.89 [20:26:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.98, 4.10, 3.54 [20:27:05] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.34, 3.59, 3.02 [20:27:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.65, 9.58, 9.88 [20:27:53] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.680 second response time [20:28:14] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:28:27] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:28:36] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.128 second response time [20:29:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.46, 9.51, 9.98 [20:30:16] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.104 second response time [20:30:19] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.326 second response time [20:30:23] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.869 second response time [20:30:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.73, 3.97, 3.61 [20:30:43] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.324 second response time [20:31:07] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.771 second response time [20:31:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.96, 3.37, 3.06 [20:32:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.89, 3.51, 3.56 [20:32:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 4.04, 3.67 [20:32:48] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.55, 8.83, 8.32 [20:33:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.02, 9.11, 9.13 [20:34:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.57, 3.54 [20:34:48] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.43, 8.00, 8.09 [20:35:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.82, 8.95, 9.08 [20:36:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.92, 3.08, 3.36 [20:36:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.50, 3.71, 3.58 [20:37:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.10, 9.88, 9.88 [20:39:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.51, 9.46, 8.89 [20:40:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.53, 3.81, 3.58 [20:41:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.78, 9.12, 9.64 [20:42:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.31, 3.23, 3.39 [20:42:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.17, 3.46, 3.60 [20:43:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.72, 6.75, 8.02 [20:43:50] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.19, 8.62, 8.76 [20:44:01] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:12] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 3.64, 3.63 [20:45:50] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.020 second response time [20:45:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.36, 7.89, 8.47 [20:46:01] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 2.593 second response time [20:46:12] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.971 second response time [20:46:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.31, 3.51 [20:47:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.52, 8.72, 8.54 [20:47:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.04, 10.13, 9.81 [20:48:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.43, 2.99, 3.36 [20:49:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.68, 8.84, 8.62 [20:49:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.02, 9.39, 9.56 [20:49:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.15, 9.31, 9.12 [20:49:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.81, 8.60, 8.66 [20:50:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.63, 3.37, 3.23 [20:51:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.36, 8.43, 8.48 [20:52:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.93, 3.32, 3.22 [20:52:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 3.56, 3.50 [20:53:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.84, 9.66, 9.60 [20:53:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.62, 7.88, 8.34 [20:54:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.47, 3.28, 3.41 [20:55:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.35, 9.95, 9.53 [20:56:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.73, 3.00, 3.18 [20:56:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.51, 3.17, 3.36 [20:59:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.88, 10.00, 9.88 [21:00:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.85, 3.56 [21:01:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.37, 10.23, 9.98 [21:03:40] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.95, 8.97, 8.56 [21:04:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.65, 4.06, 3.68 [21:05:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.15, 8.45, 8.27 [21:07:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.95, 8.19, 8.35 [21:08:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.67, 3.63 [21:09:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.56, 8.82, 8.84 [21:11:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.71, 9.02, 8.91 [21:11:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.66, 8.07, 8.22 [21:12:16] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/588c5c9d2814...54c928ec392f [21:12:18] [02miraheze/puppet] 07JohnFLewis 0354c928e - prometheus: fix mysql exporter, remove multi-instance [21:12:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.16, 9.63, 8.81 [21:14:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.26, 3.72, 3.67 [21:14:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.59, 9.28, 8.74 [21:14:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.81, 9.08, 8.71 [21:16:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.52, 3.58, 3.62 [21:18:38] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.18, 8.12, 8.41 [21:19:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.07, 9.82, 8.87 [21:19:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.42, 10.02, 9.36 [21:20:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.63, 2.66, 3.24 [21:21:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.49, 9.68, 9.34 [21:24:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.06, 4.18, 3.68 [21:24:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.81, 9.65, 9.27 [21:25:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.97, 9.84, 9.23 [21:25:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.97, 9.64, 9.98 [21:26:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.90, 3.63 [21:26:14] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [21:26:14] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:27:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.53, 10.49, 10.26 [21:28:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.12, 3.82, 3.63 [21:30:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.75, 3.82, 3.65 [21:34:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.59, 3.56 [21:35:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.86, 9.55, 9.25 [21:35:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.25, 9.13, 9.09 [21:36:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.66, 3.66, 3.59 [21:37:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.89, 8.99, 9.07 [21:40:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.25, 3.07, 3.37 [21:40:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.37, 7.77, 8.36 [21:41:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.41, 9.48, 9.33 [21:49:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.61, 9.11, 9.79 [21:49:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.06, 8.98, 9.05 [21:51:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.65, 8.93, 8.84 [21:51:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.42, 9.77, 9.94 [21:51:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.68, 9.24, 9.15 [21:54:59] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [21:57:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.57, 9.90, 9.39 [21:58:41] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.38, 8.66, 8.28 [22:02:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.61, 8.45, 8.27 [22:02:34] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.48, 9.46, 8.68 [22:02:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.38, 3.15 [22:04:24] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.88, 8.28, 8.24 [22:04:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.00, 8.89, 8.57 [22:04:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.12, 3.08 [22:05:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.32, 10.47, 9.66 [22:06:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.56, 8.41, 8.43 [22:07:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.32, 9.46, 9.39 [22:08:07] !log enable user_stats and install the query_response_time plugins on db101,db111 and db121 [22:08:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:08:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 3.83, 3.39 [22:09:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.32, 7.60, 8.48 [22:10:23] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.22, 3.21, 3.21 [22:17:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.79, 7.60, 8.44 [22:20:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.75, 9.95, 9.13 [22:20:53] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+1/-1/±0] 13https://github.com/miraheze/puppet/compare/b600c442726a...21f5a17eed78 [22:20:54] [02miraheze/puppet] 07JohnFLewis 0321f5a17 - prometheus: move innitscript [22:21:16] PROBLEM - db121 Puppet on db121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:22:35] PROBLEM - ircwiki.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ircwiki.cf' expires in 15 day(s) (Sun 27 Mar 2022 22:18:15 GMT +0000). [22:22:49] PROBLEM - db111 Puppet on db111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:23:43] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:24:00] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/798a434086a8...955d497f85bb [22:24:02] [02miraheze/ssl] 07MirahezeSSLBot 03955d497 - Bot: Update SSL cert for ircwiki.cf [22:25:53] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/21f5a17eed78...6a17338badf4 [22:25:54] [02miraheze/puppet] 07JohnFLewis 036a17338 - prometheus: remove unnecessary require [22:26:49] RECOVERY - db111 Puppet on db111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:27:16] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [22:27:43] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:30:00] PROBLEM - wiki.creaturathegame.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.creaturathegame.com' expires in 15 day(s) (Sun 27 Mar 2022 22:23:20 GMT +0000). [22:30:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.62, 3.29 [22:31:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.94, 3.43, 2.75 [22:31:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b86957197676 [22:31:28] [02miraheze/puppet] 07paladox 03b869571 - varnish: Fix downloading files [22:31:30] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://github.com/miraheze/puppet [22:31:31] [02puppet] 07paladox opened pull request 03#2450: varnish: Fix downloading files - 13https://github.com/miraheze/puppet/pull/2450 [22:31:45] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.18, 10.10, 9.46 [22:31:52] !log upgrade prometheus-mysqld-exporter on db* to 0.13.0 [22:31:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:33:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.83, 3.20, 2.75 [22:34:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.85, 3.32, 3.26 [22:35:19] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/955d497f85bb...d4a17836ce4f [22:35:20] [02miraheze/ssl] 07MirahezeSSLBot 03d4a1783 - Bot: Update SSL cert for wiki.creaturathegame.com [22:35:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.87, 10.00, 9.60 [22:36:58] PROBLEM - ecole.science - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ecole.science' expires in 15 day(s) (Sun 27 Mar 2022 22:31:32 GMT +0000). [22:37:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.82, 9.08, 9.98 [22:37:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.79, 10.23, 9.75 [22:39:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.75, 9.27, 9.45 [22:41:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.17, 9.05, 9.81 [22:41:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.23, 9.99, 10.08 [22:43:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.88, 9.60, 9.91 [22:43:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d4a17836ce4f...e5499049b188 [22:43:34] [02miraheze/ssl] 07MirahezeSSLBot 03e549904 - Bot: Update SSL cert for ecole.science [22:44:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.46, 3.25 [22:45:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.23, 9.62, 9.87 [22:46:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.47, 3.29 [22:47:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.26, 9.55, 9.78 [22:47:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.63, 8.84, 9.55 [22:47:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.67, 8.96, 8.33 [22:48:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.51, 3.34, 3.28 [22:49:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.82, 9.79, 9.85 [22:49:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.53, 9.65, 9.76 [22:49:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.46, 9.29, 9.27 [22:49:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.70, 8.84, 8.36 [22:51:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.81, 8.57, 8.22 [22:51:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.72, 10.24, 10.02 [22:51:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.41, 9.40, 9.33 [22:51:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.51, 8.31, 8.22 [22:52:10] RECOVERY - ircwiki.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'ircwiki.cf' will expire on Thu 09 Jun 2022 21:23:54 GMT +0000. [22:53:11] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.87, 8.06, 8.09 [22:53:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.71, 9.58, 9.74 [22:55:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.42, 3.34 [22:55:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.54, 9.96, 9.86 [22:55:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.19, 9.72, 9.45 [22:56:18] PROBLEM - rct.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rct.wiki' expires in 15 day(s) (Sun 27 Mar 2022 22:26:00 GMT +0000). [22:57:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.56, 2.98, 3.18 [22:57:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.10, 9.03, 9.22 [22:59:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.63, 9.62, 9.86 [22:59:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.06, 9.84, 9.87 [22:59:30] RECOVERY - wiki.creaturathegame.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.creaturathegame.com' will expire on Thu 09 Jun 2022 21:35:14 GMT +0000. [23:00:01] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:02] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:41] PROBLEM - wiki.beergeeks.co.il - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.beergeeks.co.il' expires in 15 day(s) (Sun 27 Mar 2022 23:00:03 GMT +0000). [23:02:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e5499049b188...3c7afceb6051 [23:02:47] [02miraheze/ssl] 07MirahezeSSLBot 033c7afce - Bot: Update SSL cert for rct.wiki [23:03:09] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.72, 8.11, 7.89 [23:03:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.09, 9.47, 9.66 [23:05:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.98, 9.45, 9.64 [23:05:56] RECOVERY - ecole.science - LetsEncrypt on sslhost is OK: OK - Certificate 'ecole.science' will expire on Thu 09 Jun 2022 21:43:27 GMT +0000. [23:06:57] PROBLEM - wiki.airyx.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.airyx.org' expires in 15 day(s) (Sun 27 Mar 2022 22:58:36 GMT +0000). [23:07:05] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3c7afceb6051...fd3b660fbfe1 [23:07:07] [02miraheze/ssl] 07MirahezeSSLBot 03fd3b660 - Bot: Update SSL cert for wiki.beergeeks.co.il [23:07:08] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.43, 7.66, 7.79 [23:07:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.62, 9.72, 9.70 [23:07:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.03, 7.45, 8.29 [23:09:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.25, 9.07, 9.49 [23:10:19] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fd3b660fbfe1...9c0ebf900eeb [23:10:21] [02miraheze/ssl] 07MirahezeSSLBot 039c0ebf9 - Bot: Update SSL cert for wiki.airyx.org [23:11:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.25, 9.28, 9.49 [23:11:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.81, 9.08, 8.76 [23:11:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.87, 3.40 [23:13:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.23, 9.59, 9.43 [23:13:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.73, 9.51, 9.56 [23:13:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.68, 9.05, 8.80 [23:13:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.92, 3.47 [23:13:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.12, 8.41, 8.10 [23:15:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.55, 9.52, 9.43 [23:15:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.56, 8.31, 8.11 [23:15:42] PROBLEM - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.bikinibottomdiscord.org' expires in 15 day(s) (Sun 27 Mar 2022 22:59:46 GMT +0000). [23:17:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.09, 9.14, 8.84 [23:18:31] PROBLEM - www.connorjwat.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.connorjwat.net' expires in 15 day(s) (Sun 27 Mar 2022 22:57:17 GMT +0000). [23:19:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.66, 4.00, 3.67 [23:19:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.50, 9.64, 8.70 [23:20:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9c0ebf900eeb...18813d3aae75 [23:20:04] [02miraheze/ssl] 07MirahezeSSLBot 0318813d3 - Bot: Update SSL cert for wiki.bikinibottomdiscord.org [23:20:05] PROBLEM - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikipedia.yogfront.ooo' expires in 15 day(s) (Sun 27 Mar 2022 22:58:53 GMT +0000). [23:20:18] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/18813d3aae75...c51b797cb2eb [23:20:20] [02miraheze/ssl] 07MirahezeSSLBot 03c51b797 - Bot: Update SSL cert for www.connorjwat.net [23:21:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.68, 3.57 [23:21:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.76, 8.99, 8.57 [23:21:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.27, 9.93, 9.60 [23:21:40] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 2.00, 1.67, 1.19 [23:23:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.95, 3.38, 3.48 [23:23:35] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.35, 1.91, 1.34 [23:24:34] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1472s [23:24:35] PROBLEM - www.ambientguitar.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ambientguitar.org' expires in 15 day(s) (Sun 27 Mar 2022 22:59:11 GMT +0000). [23:24:39] PROBLEM - wiki.timewisely.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.timewisely.net' expires in 15 day(s) (Sun 27 Mar 2022 22:59:28 GMT +0000). [23:24:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:24:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.77, 3.07, 3.35 [23:25:10] PROBLEM - threedomwiki.pcast.site - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'threedomwiki.pcast.site' expires in 15 day(s) (Sun 27 Mar 2022 23:00:46 GMT +0000). [23:25:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.18, 9.44, 9.34 [23:25:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.14, 9.88, 9.58 [23:25:30] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.80, 1.84, 1.38 [23:26:13] PROBLEM - zh.internetpedia.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'zh.internetpedia.tk' expires in 15 day(s) (Sun 27 Mar 2022 23:00:22 GMT +0000). [23:26:18] RECOVERY - rct.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rct.wiki' will expire on Thu 09 Jun 2022 22:02:40 GMT +0000. [23:26:29] PROBLEM - wiki.fraterniterebellion.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fraterniterebellion.com' expires in 15 day(s) (Sun 27 Mar 2022 23:23:46 GMT +0000). [23:27:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.42, 9.35, 9.34 [23:27:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.07, 9.91, 9.84 [23:27:35] PROBLEM - ambientguitar.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ambientguitar.org' expires in 15 day(s) (Sun 27 Mar 2022 22:59:11 GMT +0000). [23:28:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c51b797cb2eb...4f300e24058f [23:28:27] [02miraheze/ssl] 07MirahezeSSLBot 034f300e2 - Bot: Update SSL cert for wikipedia.yogfront.ooo [23:29:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.65, 7.62, 8.14 [23:29:15] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:29:16] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:29:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.15, 4.05, 3.22 [23:29:24] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:29:28] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:29:38] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:29:40] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:29:44] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:29:45] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:30:21] PROBLEM - wikileague.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikileague.cf' expires in 15 day(s) (Sun 27 Mar 2022 22:58:22 GMT +0000). [23:30:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 4.42, 3.87 [23:30:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.53, 3.18 [23:31:14] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.989 second response time [23:31:15] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.52, 2.07, 1.61 [23:31:16] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.013 second response time [23:31:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.99, 3.63, 3.16 [23:31:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.18, 9.14, 9.22 [23:31:24] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.227 second response time [23:31:31] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 4.022 second response time [23:31:38] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.432 second response time [23:31:41] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.306 second response time [23:31:41] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.030 second response time [23:31:42] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.019 second response time [23:31:52] RECOVERY - wiki.beergeeks.co.il - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.beergeeks.co.il' will expire on Thu 09 Jun 2022 22:07:00 GMT +0000. [23:32:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4f300e24058f...faf598b2fa30 [23:32:29] PROBLEM - wikidiffs.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikidiffs.ga' expires in 15 day(s) (Sun 27 Mar 2022 23:01:26 GMT +0000). [23:32:30] [02miraheze/ssl] 07MirahezeSSLBot 03faf598b - Bot: Update SSL cert for threedomwiki.pcast.site [23:32:56] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/faf598b2fa30...78d0a8fc72da [23:32:57] [02miraheze/ssl] 07MirahezeSSLBot 0378d0a8f - Bot: Update SSL cert for wikileague.cf [23:33:13] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.47, 3.33, 3.11 [23:33:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.18, 9.09, 9.19 [23:33:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.35, 9.04, 9.07 [23:34:07] PROBLEM - mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Sun 27 Mar 2022 23:30:36 GMT +0000). [23:34:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.20, 3.47, 3.64 [23:34:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 2.93, 3.03 [23:35:14] PROBLEM - mediawiki.ecfin.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mediawiki.ecfin.eu' expires in 15 day(s) (Sun 27 Mar 2022 23:24:02 GMT +0000). [23:35:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.26, 8.31, 8.79 [23:35:34] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 2132s [23:35:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/78d0a8fc72da...e32df351b633 [23:35:40] [02miraheze/ssl] 07MirahezeSSLBot 03e32df35 - Bot: Update SSL cert for wikidiffs.ga [23:35:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:54] RECOVERY - wiki.airyx.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.airyx.org' will expire on Thu 09 Jun 2022 22:09:15 GMT +0000. [23:35:57] PROBLEM - nenawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nenawiki.org' expires in 15 day(s) (Sun 27 Mar 2022 23:32:57 GMT +0000). [23:36:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e32df351b633...424eed90b9bb [23:36:09] [02miraheze/ssl] 07MirahezeSSLBot 03424eed9 - Bot: Update SSL cert for wiki.timewisely.net [23:36:27] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mrlove.wiki' expires in 15 day(s) (Sun 27 Mar 2022 23:24:17 GMT +0000). [23:37:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.30, 9.12, 8.73 [23:38:44] PROBLEM - miraheze.ml - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.ml' expires in 15 day(s) (Sun 27 Mar 2022 23:22:19 GMT +0000). [23:38:55] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.77, 1.61, 1.66 [23:38:57] PROBLEM - wiki.ssij.se - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ssij.se' expires in 15 day(s) (Sun 27 Mar 2022 23:31:09 GMT +0000). [23:39:11] PROBLEM - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ricochetuniverse.com' expires in 15 day(s) (Sun 27 Mar 2022 23:24:45 GMT +0000). [23:39:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/424eed90b9bb...8c03bb45f6d6 [23:39:15] [02miraheze/ssl] 07MirahezeSSLBot 038c03bb4 - Bot: Update SSL cert for zh.internetpedia.tk [23:39:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.51, 9.56, 9.48 [23:39:48] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'speleo.wiki' expires in 15 day(s) (Sun 27 Mar 2022 23:34:22 GMT +0000). [23:40:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.25, 2.85, 3.33 [23:41:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.41, 9.74, 9.39 [23:41:28] PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'adadevelopersacademy.wiki' expires in 15 day(s) (Sun 27 Mar 2022 23:34:52 GMT +0000). [23:42:10] PROBLEM - www.fraterniterebellion.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fraterniterebellion.com' expires in 15 day(s) (Sun 27 Mar 2022 23:23:46 GMT +0000). [23:43:04] PROBLEM - data.nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'data.nonbinary.wiki' expires in 15 day(s) (Sun 27 Mar 2022 23:40:03 GMT +0000). [23:43:24] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8c03bb45f6d6...f4b431b8564a [23:43:25] [02miraheze/ssl] 07MirahezeSSLBot 03f4b431b - Bot: Update SSL cert for wiki.ssij.se [23:44:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:45:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.55, 9.69, 9.06 [23:45:14] PROBLEM - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'encyclopediaofastrobiology.org' expires in 15 day(s) (Sun 27 Mar 2022 23:42:52 GMT +0000). [23:45:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.85, 9.00, 9.19 [23:45:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.20, 9.34, 9.55 [23:45:31] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f4b431b8564a...940651c3c082 [23:45:33] [02miraheze/ssl] 07MirahezeSSLBot 03940651c - Bot: Update SSL cert for ambientguitar.org [23:45:57] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Sun 27 Mar 2022 23:38:58 GMT +0000). [23:46:00] PROBLEM - vitriol.spoon.army - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vitriol.spoon.army' expires in 15 day(s) (Sun 27 Mar 2022 23:24:31 GMT +0000). [23:46:32] PROBLEM - archive.a2b2.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'archive.a2b2.org' expires in 15 day(s) (Sun 27 Mar 2022 23:34:37 GMT +0000). [23:46:34] PROBLEM - franchise.franchising.org.ua - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'franchise.franchising.org.ua' expires in 15 day(s) (Sun 27 Mar 2022 23:41:55 GMT +0000). [23:46:39] PROBLEM - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'cyberlaw.ccdcoe.org' expires in 15 day(s) (Sun 27 Mar 2022 23:43:49 GMT +0000). [23:46:49] PROBLEM - wiki.tulpa.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.tulpa.info' expires in 15 day(s) (Sun 27 Mar 2022 23:25:17 GMT +0000). [23:47:15] PROBLEM - wiki.danvs.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.danvs.net' expires in 15 day(s) (Sun 27 Mar 2022 23:34:06 GMT +0000). [23:47:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.40, 10.06, 9.78 [23:47:31] PROBLEM - wiki.thunis.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.thunis.eu' expires in 15 day(s) (Sun 27 Mar 2022 23:38:16 GMT +0000). [23:48:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.15, 3.36, 3.47 [23:48:02] PROBLEM - nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonbinary.wiki' expires in 15 day(s) (Sun 27 Mar 2022 23:33:20 GMT +0000). [23:48:45] PROBLEM - archive.stellurgists.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'archive.stellurgists.wiki' expires in 15 day(s) (Sun 27 Mar 2022 23:39:35 GMT +0000). [23:48:49] PROBLEM - dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Sun 27 Mar 2022 23:38:58 GMT +0000). [23:49:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.45, 9.56, 9.19 [23:49:07] PROBLEM - wiki.cdntennis.ca - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cdntennis.ca' expires in 15 day(s) (Sun 27 Mar 2022 23:22:36 GMT +0000). [23:49:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.46, 9.50, 9.34 [23:49:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.40, 9.69, 9.68 [23:50:40] PROBLEM - ndg.nenawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nenawiki.org' expires in 15 day(s) (Sun 27 Mar 2022 23:32:57 GMT +0000). [23:50:41] PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.grottocenter.org' expires in 15 day(s) (Sun 27 Mar 2022 23:23:16 GMT +0000). [23:50:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.60, 8.94, 7.94 [23:50:57] PROBLEM - wiki.mineland.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mineland.eu' expires in 15 day(s) (Sun 27 Mar 2022 23:33:52 GMT +0000). [23:51:19] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'm.miraheze.org' expires in 15 day(s) (Sun 27 Mar 2022 23:47:45 GMT +0000). [23:51:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.36, 10.48, 9.98 [23:51:31] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/940651c3c082...6f2ceaae558b [23:51:32] [02miraheze/ssl] 07MirahezeSSLBot 036f2ceaa - Bot: Update SSL cert for encyclopediaofastrobiology.org [23:51:35] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 11.83, 5.42, 3.00 [23:51:37] PROBLEM - zamlung.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'zamlung.tk' expires in 15 day(s) (Sun 27 Mar 2022 23:48:30 GMT +0000). [23:52:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.20, 2.98, 3.31 [23:52:38] PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'permanentfuturelab.wiki' expires in 15 day(s) (Sun 27 Mar 2022 23:30:53 GMT +0000). [23:52:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.27, 8.88, 8.03 [23:53:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.31, 9.38, 9.18 [23:54:00] PROBLEM - wiki.graalmilitary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.graalmilitary.com' expires in 15 day(s) (Sun 27 Mar 2022 23:22:51 GMT +0000). [23:55:40] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'antiguabarbudacalypso.com' expires in 15 day(s) (Sun 27 Mar 2022 23:35:09 GMT +0000). [23:55:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.63, 8.35, 7.83 [23:56:09] RECOVERY - ambientguitar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ambientguitar.org' will expire on Thu 09 Jun 2022 22:45:26 GMT +0000. [23:56:29] PROBLEM - monarchists.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'monarchists.wiki' expires in 15 day(s) (Sun 27 Mar 2022 23:25:01 GMT +0000). [23:57:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.48, 9.44, 9.32 [23:57:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.35, 9.91, 9.98 [23:57:33] PROBLEM - guia.esporo.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guia.esporo.net' expires in 15 day(s) (Sun 27 Mar 2022 23:44:33 GMT +0000). [23:57:44] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'publictestwiki.com' expires in 15 day(s) (Sun 27 Mar 2022 23:41:23 GMT +0000). [23:57:55] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.61, 7.79, 7.69 [23:58:23] PROBLEM - ao90.pinho.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ao90.pinho.org' expires in 15 day(s) (Sun 27 Mar 2022 23:39:20 GMT +0000). [23:58:39] PROBLEM - www.mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Sun 27 Mar 2022 23:30:36 GMT +0000). [23:59:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.28, 10.52, 10.19 [23:59:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6f2ceaae558b...33589a310d0b [23:59:42] [02miraheze/ssl] 07MirahezeSSLBot 0333589a3 - Bot: Update SSL cert for wiki.grottocenter.org