[00:00:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.69, 3.94, 3.48 [00:02:18] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.17, 1.98, 1.38 [00:03:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.72, 11.25, 9.86 [00:03:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 3.82, 3.64 [00:03:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.91, 3.69, 3.56 [00:04:44] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.88, 1.63, 1.95 [00:05:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.61, 4.62, 3.83 [00:06:19] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.99, 1.53, 1.33 [00:06:32] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.77, 2.79, 1.63 [00:07:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.50, 10.30, 9.80 [00:07:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.08, 3.90, 3.65 [00:09:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.41, 10.03, 9.77 [00:10:27] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.98, 3.03, 2.01 [00:11:46] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [00:11:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.62, 3.17, 3.34 [00:13:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.95, 3.87 [00:14:44] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.96, 1.25, 1.64 [00:14:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.72, 4.68, 4.14 [00:17:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.63, 3.78 [00:19:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.25, 4.03, 3.91 [00:21:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.75, 3.72, 3.80 [00:27:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.31, 3.57 [00:31:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.52, 3.30, 3.53 [00:32:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.86, 3.38, 3.87 [00:34:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.67, 3.96, 4.05 [00:35:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.29, 10.49, 9.85 [00:35:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.70, 3.87, 3.70 [00:36:43] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.47, 1.83, 1.67 [00:37:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.90, 3.75 [00:39:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.95, 11.72, 10.55 [00:39:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.04, 4.53, 4.00 [00:40:43] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.53, 1.60, 1.61 [00:41:16] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:41:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.57, 12.46, 10.96 [00:42:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.63, 3.11, 3.72 [00:43:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.12, 11.90, 10.96 [00:43:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.95, 3.36, 3.66 [00:47:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.83, 2.76, 3.36 [00:49:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.93, 3.86 [00:51:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.78, 3.81 [00:51:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.53, 3.53 [00:53:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.81, 3.63 [00:53:43] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 1.85, 1.69 [00:55:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.46, 3.64 [00:55:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.62, 3.58 [00:55:43] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.04, 1.57, 1.60 [00:57:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.68, 3.18, 3.52 [00:59:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.87, 3.16, 3.40 [01:00:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.93, 3.38, 4.00 [01:02:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.12, 3.81, 4.06 [01:03:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.92, 9.33, 10.17 [01:04:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.92, 3.44, 3.90 [01:06:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 3.77, 3.96 [01:07:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.98, 2.89, 3.31 [01:07:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.52, 10.34, 10.40 [01:08:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.86, 3.37, 3.79 [01:11:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.48, 10.84, 10.54 [01:13:27] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.24, 3.50, 2.60 [01:13:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.93, 10.76, 10.54 [01:16:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.34, 2.80, 3.38 [01:17:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 11.58, 10.93 [01:20:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.30, 3.46 [01:21:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.67, 11.87, 11.20 [01:22:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.69, 3.18, 3.40 [01:25:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.42, 12.06, 11.42 [01:26:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.61, 3.54 [01:27:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.74, 11.06, 11.12 [01:28:36] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.45, 3.45, 3.15 [01:32:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.68, 3.95, 3.42 [01:32:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.40, 4.31, 3.80 [01:33:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.65, 10.90, 10.93 [01:34:21] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.31, 3.04, 3.12 [01:34:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.66, 3.37 [01:34:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.97, 4.04, 3.44 [01:34:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.33, 3.86, 3.69 [01:36:23] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.45, 3.29 [01:37:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.80, 10.52, 10.81 [01:38:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.83, 3.29, 3.25 [01:40:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.50, 3.11, 3.23 [01:41:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.90, 9.29, 8.19 [01:41:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.57, 11.56, 11.15 [01:42:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.93, 2.87, 3.35 [01:43:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.35, 9.22, 8.30 [01:43:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.32, 11.01, 11.02 [01:46:19] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.92, 1.59, 1.37 [01:47:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.69, 11.24, 11.03 [01:48:19] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.24, 1.47, 1.35 [01:48:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.52, 4.08, 3.66 [01:49:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.17, 9.89, 10.58 [01:49:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.83, 3.42 [01:50:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.72, 3.80, 3.61 [01:51:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.72, 3.37, 3.30 [01:53:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.51, 1.86, 1.52 [01:55:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.11, 1.58, 1.46 [02:00:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.77, 3.22, 3.40 [02:07:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.74, 3.29 [02:09:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.55, 3.28 [02:11:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.17, 10.97, 10.67 [02:11:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.02, 4.48, 3.72 [02:12:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.50, 3.68, 3.50 [02:13:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.75, 3.89, 3.46 [02:13:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.72, 10.90, 10.69 [02:17:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.78, 3.78, 3.57 [02:17:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.85, 3.73 [02:18:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.99, 3.07, 3.34 [02:21:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.55, 3.75, 3.68 [02:22:50] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.74, 1.86, 1.48 [02:22:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.93, 3.54, 3.47 [02:23:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 2.95, 3.26 [02:23:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.32, 12.10, 11.21 [02:23:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.67, 3.51, 3.58 [02:24:41] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.09, 1.83, 1.51 [02:24:45] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.70, 1.48, 1.39 [02:24:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.22, 3.01, 3.28 [02:25:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.24, 11.91, 11.26 [02:25:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.63, 3.76, 3.63 [02:26:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.24, 1.61, 1.46 [02:27:27] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.75, 2.98, 3.91 [02:27:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.27, 11.95, 11.34 [02:28:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.51, 3.43 [02:29:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.88, 11.46, 11.24 [02:31:27] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.56, 2.18, 3.36 [02:31:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.86, 3.77 [02:32:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.74, 3.30, 3.38 [02:34:18] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.27, 1.71, 1.51 [02:36:18] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.59, 1.36, 1.41 [02:39:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.39, 3.59, 3.47 [02:41:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 3.75, 3.54 [02:42:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.75, 10.35, 9.37 [02:43:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.25, 3.38, 3.44 [02:43:41] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.02, 1.60, 1.40 [02:44:02] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.98, 9.96, 9.33 [02:44:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.50, 3.36 [02:45:31] MacFan4000: puppet still failing. [02:45:35] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.72, 3.20, 3.37 [02:46:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.43, 2.99, 3.19 [02:46:37] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.21, 6.56, 5.82 [02:47:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.47, 1.59, 1.43 [02:47:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.01, 9.17, 10.06 [02:47:49] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+1/-1/±0] 13https://github.com/miraheze/ssl/compare/f82ecb63fdb4...dbb800e07233 [02:47:49] [url] Comparing f82ecb63fdb4...dbb800e07233 · miraheze/ssl · GitHub | github.com [02:47:50] [02miraheze/ssl] 07Universal-Omega 03dbb800e - Fix cert name [02:47:57] MacFan4000: ^ [02:48:13] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.84, 3.20, 2.76 [02:48:30] Ty [02:48:37] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.77, 6.63, 5.94 [02:49:17] No problem. [02:49:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.78, 2.73, 3.24 [02:50:13] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.16, 3.61, 2.97 [02:51:13] RECOVERY - mw112 Puppet on mw112 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [02:51:49] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [02:52:13] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.63, 3.42, 2.97 [02:52:37] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.03, 7.32, 6.37 [02:53:19] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [02:54:03] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [02:54:13] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.93, 3.07, 2.88 [02:54:37] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.06, 6.95, 6.33 [02:55:41] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.28, 2.08, 1.75 [02:56:03] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [02:56:37] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.12, 6.37, 6.20 [02:56:44] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:57:09] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [02:57:41] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.69, 1.99, 1.75 [02:57:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.31, 3.44, 3.38 [02:59:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.72, 3.10, 3.26 [03:01:40] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.21, 1.56, 1.63 [03:02:18] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [03:04:50] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [03:06:24] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [03:07:02] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.73, 3.61, 2.31 [03:07:24] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:08:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.47, 4.32, 3.55 [03:09:40] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.80, 1.83, 1.68 [03:10:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.92, 3.50 [03:11:40] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.03, 1.55, 1.60 [03:12:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.21, 3.37, 3.35 [03:12:45] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.26, 1.76, 1.84 [03:12:55] RECOVERY - mw102 Puppet on mw102 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:13:13] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:14:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.26, 3.70, 3.36 [03:16:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.60, 3.35, 3.27 [03:18:28] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.81, 1.24, 1.60 [03:20:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.82, 3.48 [03:21:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.63, 3.87, 3.37 [03:21:40] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.61, 1.84, 1.61 [03:22:13] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.92, 3.23, 2.86 [03:23:40] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.66, 1.68, 1.58 [03:24:13] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.39, 2.86, 2.76 [03:25:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.40, 3.76, 3.46 [03:25:47] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.53, 3.22, 3.34 [03:27:25] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.67, 3.83, 3.51 [03:29:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.93, 3.72, 3.52 [03:29:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.25, 3.91, 3.56 [03:31:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.26, 3.24, 3.35 [03:34:40] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 2.00, 1.82, 1.70 [03:35:10] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.36, 3.08, 3.33 [03:36:40] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.70, 1.67, 1.65 [03:46:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.96, 3.79, 3.42 [03:50:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 4.04, 3.58 [03:51:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.82, 3.35, 3.25 [03:52:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.97, 3.86, 3.56 [03:52:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.34, 2.26, 1.92 [03:53:41] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.19, 2.91, 3.10 [03:54:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.84, 1.78, 1.79 [03:56:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.71, 1.43, 1.66 [03:58:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.76, 3.20, 3.37 [04:02:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.61, 3.29 [04:04:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.92, 3.57, 3.31 [04:05:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 3.54, 3.36 [04:06:23] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.36, 3.35, 3.25 [04:07:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.27, 10.34, 9.39 [04:07:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.63, 3.11, 3.22 [04:09:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.54, 9.68, 9.27 [04:13:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.35, 10.66, 9.79 [04:17:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.49, 10.03, 9.74 [04:23:22] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.15, 6.73, 5.97 [04:24:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.92, 3.42, 3.04 [04:25:17] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.22, 6.51, 5.98 [04:28:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.04, 3.43, 3.14 [04:30:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.43, 9.13, 8.81 [04:30:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.36, 3.37, 3.14 [04:32:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.42, 9.09, 8.83 [04:32:57] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.11, 7.38, 6.47 [04:34:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.04, 2.92 [04:34:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.62, 9.99, 9.70 [04:34:52] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.80, 7.68, 6.70 [04:36:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.67, 3.65, 3.14 [04:37:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.93, 3.28, 3.06 [04:42:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.94, 9.43, 9.68 [04:43:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.31, 9.96, 9.33 [04:44:37] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.33, 6.74, 6.75 [04:48:37] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.68, 6.81, 6.79 [04:49:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.83, 9.39, 9.34 [04:50:37] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.10, 6.33, 6.62 [04:56:19] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.63, 1.75, 1.44 [04:58:18] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.75, 1.36, 1.34 [05:06:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.13, 11.22, 10.20 [05:08:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.15, 10.53, 10.08 [05:10:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.33, 10.15, 9.99 [05:15:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.75, 3.25 [05:16:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.54, 10.35, 9.51 [05:17:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.31, 10.68, 10.15 [05:18:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.25, 2.96 [05:18:49] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.27, 9.54, 9.32 [05:19:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.56, 2.85, 3.03 [05:19:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.36, 10.05, 9.96 [05:20:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.01, 3.02, 2.90 [05:23:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.84, 1.54 [05:24:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.89, 3.47, 3.14 [05:25:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.95, 1.55, 1.48 [05:25:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.32, 11.17, 10.45 [05:25:57] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.82, 2.91, 2.98 [05:27:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.27, 9.43, 8.36 [05:29:09] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.59, 8.90, 8.31 [05:29:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.47, 11.20, 10.59 [05:37:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.19, 9.95, 10.20 [05:42:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.72, 10.47, 9.78 [05:44:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.93, 9.58, 9.55 [05:49:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.97, 10.62, 10.10 [05:51:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.92, 11.29, 10.43 [05:56:02] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.49, 10.39, 9.63 [05:57:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.41, 9.74, 9.49 [06:03:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.39, 11.27, 10.90 [06:09:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.99, 11.79, 11.34 [06:11:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.60, 9.96, 9.46 [06:11:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.60, 12.07, 11.49 [06:13:06] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.39, 6.92, 6.50 [06:13:37] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.94, 9.08, 9.19 [06:13:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.09, 11.31, 11.29 [06:15:01] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.68, 7.49, 6.75 [06:16:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.30, 7.44, 6.82 [06:20:45] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.29, 8.13, 7.21 [06:27:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.87, 9.23, 10.15 [06:29:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.23, 3.60, 3.07 [06:30:37] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.86, 7.44, 7.41 [06:31:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.39, 3.27, 3.00 [06:34:37] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.97, 8.04, 7.59 [06:36:34] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.03, 3.77, 3.24 [06:37:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.35, 10.01, 9.87 [06:38:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.43, 3.17 [06:38:37] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.03, 7.99, 7.73 [06:39:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.15, 8.70, 9.41 [06:40:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.89, 3.16, 3.09 [06:50:37] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.60, 5.96, 6.74 [06:59:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.66, 3.22, 2.87 [07:03:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.76, 2.74, 2.78 [07:15:36] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.88, 1.69, 1.33 [07:17:36] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.17, 1.57, 1.34 [07:20:46] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.34, 7.51, 6.46 [07:22:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.05, 2.80 [07:22:40] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.80, 7.13, 6.44 [07:24:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.06, 2.82, 2.73 [07:26:37] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.53, 6.40, 6.31 [07:36:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.41, 10.13, 9.17 [07:38:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.94, 10.08, 9.27 [07:42:47] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.07, 2.52, 1.72 [07:44:47] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.22, 1.95, 1.61 [07:46:47] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.59, 1.47, 1.47 [07:54:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.68, 3.19, 2.76 [07:59:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 3.76, 3.05 [07:59:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.74, 3.02 [08:01:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.43, 3.37, 3.00 [08:01:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.63, 3.32, 2.96 [08:04:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.47, 3.71, 3.51 [08:06:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.24, 3.22, 3.36 [08:37:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.77, 3.00, 2.76 [08:37:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.80, 3.20, 2.87 [08:38:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.45, 3.41, 2.99 [08:40:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.03, 3.03, 2.91 [08:41:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.86, 2.72, 2.73 [08:41:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.04, 2.68, 2.73 [09:11:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.26, 9.72, 8.55 [09:11:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.48, 10.00, 8.83 [09:13:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.52, 9.11, 8.49 [09:13:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.00, 9.49, 8.78 [09:23:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.62, 10.36, 9.51 [09:25:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.35, 9.75, 9.40 [09:29:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.15, 10.30, 9.74 [09:37:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.50, 9.65, 9.76 [09:46:40] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.90, 6.93, 6.23 [09:46:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.31, 3.34, 2.87 [09:47:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.18, 3.83, 3.09 [09:48:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.18, 3.05, 2.81 [09:49:04] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.64, 3.25, 2.88 [09:49:52] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.54, 3.18, 2.93 [09:50:37] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.14, 7.14, 6.45 [09:50:59] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.62, 2.98, 2.83 [09:51:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.29, 9.91, 9.76 [09:52:37] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.81, 6.97, 6.47 [09:53:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.96, 9.73, 9.72 [09:56:37] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.41, 6.25, 6.29 [10:08:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.91, 3.43, 3.07 [10:10:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 3.23, 2.92 [10:10:34] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.22, 3.31, 3.07 [10:12:23] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.47, 2.96, 2.86 [10:13:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.02, 10.36, 9.43 [10:16:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 3.81, 3.24 [10:17:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.53, 10.03, 9.51 [10:18:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.56, 3.21 [10:20:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.17, 3.11, 3.09 [10:28:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.60, 3.31 [10:30:05] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.79, 2.91, 3.09 [10:51:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.95, 3.42, 3.06 [10:53:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.02, 3.34, 3.08 [10:57:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.48, 3.06 [10:57:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.21, 3.72, 3.31 [10:59:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.77, 2.92, 2.91 [10:59:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.71, 2.98, 3.09 [11:07:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.06, 10.10, 9.06 [11:09:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.20, 10.14, 9.20 [11:10:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.57, 3.30 [11:12:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.57, 3.08, 3.14 [11:16:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.72, 3.41 [11:20:06] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.37, 3.19, 3.28 [11:24:29] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [11:24:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.02, 10.26, 9.81 [11:26:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.17, 9.92, 9.74 [11:26:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.21, 3.04 [11:26:44] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/61b7c599d892...d3f809bbc909 [11:26:45] [url] Comparing 61b7c599d892...d3f809bbc909 · miraheze/mediawiki · GitHub | github.com [11:26:46] [02miraheze/mediawiki] 07Reception123 03d3f809b - Update CentralAuth [11:27:47] !log [reception@mwtask111] starting deploy of {'world': True} to all [11:27:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:28:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.45, 3.27, 2.94 [11:28:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.59, 2.93, 2.96 [11:30:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.61, 3.02, 2.89 [11:30:58] Reception123: you need --l10n [11:31:04] There are i18n changes [11:34:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.71, 3.65, 3.19 [11:36:35] !log [reception@mwtask111] finished deploy of {'world': True} to all - SUCCESS in 527s [11:36:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:38:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.28, 2.72, 2.96 [11:48:52] ah [11:48:59] !log [reception@mwtask111] starting deploy of {'world': True, 'l10n': True} to all [11:49:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:49:47] Reception123: you don't need --world now [11:49:59] --l10n works on its own [11:50:42] I thought so but wasn't 100% sure [11:52:38] Reception123: from the next version, if you're unsure, just run with --noop [11:52:50] That will tell you exactly what it does [11:53:02] cool [11:53:04] Might be worth you locally testing [11:53:10] To be used to the output [11:53:35] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [11:53:37] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.72, 1.94, 1.38 [11:53:39] yeah [11:54:16] Reception123: a noop deploy injects itself right before a deploy [11:54:21] And after the prep stage [11:54:35] Showing you the json output [11:54:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.56, 3.78, 3.30 [11:55:36] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.24, 1.61, 1.33 [11:56:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.18, 3.09, 3.10 [12:00:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.56, 3.91, 3.42 [12:04:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.59, 3.77, 3.51 [12:08:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.43, 3.04, 3.26 [12:21:25] !log [reception@mwtask111] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw111.miraheze.org [12:21:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.39, 10.50, 9.53 [12:23:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.20, 10.34, 9.57 [12:23:55] uh what now? publictestwiki? [12:24:29] Reception123: that's been for ages [12:24:47] It should tell you why it failed [12:25:02] https://www.irccloud.com/pastebin/i7NPR4L1/ [12:25:36] Reception123: that tells you mw111 is down [12:25:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.95, 11.37, 10.05 [12:26:25] And I can tell you it has no workers left [12:26:48] hmm [12:26:49] Status: 500 means it returned 500 as the response code [12:27:05] Reception123: what does sudo service php7.4-fpm status say on mw111 [12:27:40] it's up [12:27:46] nothing particularly relevant, just some warnings/notices [12:27:49] https://www.irccloud.com/pastebin/k8eNZq9a/ [12:28:01] Reception123: higher up [12:28:12] It should say how many workers are active / idle [12:28:27] oh just got this [12:28:29] I have a feeling it's locked up [12:28:29] Jul 08 12:27:29 mw111 php7.4-fpm[394]: [NOTICE] [pool www] child 804892 exited with code 0 after 1364.781132 seconds from start [12:28:29] Jul 08 12:27:29 mw111 php7.4-fpm[394]: [NOTICE] [pool www] child 830831 started [12:28:35] Reception123: no [12:28:38] Not what I'm asking [12:28:49] https://www.irccloud.com/pastebin/a5lHok8X/ [12:29:08] Reception123: idle: 0 [12:29:19] Restart php7.4-fpm [12:29:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.39, 3.19 [12:29:38] I imagine icinga will tell you rendering is broken [12:29:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.94, 11.99, 10.64 [12:30:01] !log sudo service php7.4-fpm restart on mw111 [12:30:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:30:43] Reception123: now does mw111 work [12:31:14] you'll be best doing --servers=mw111,mw112,mw121,mw122 [12:31:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.94, 3.28, 3.17 [12:31:20] To save redoing finished servers [12:32:46] Reception123: can you please review/possibly merge https://github.com/miraheze/puppet/pull/2691 when you have a chance? [12:32:47] [url] Manage /home/ssl-admins recursively + ensure existence and permissions for /etc/puppetlabs/puppet/ssl-keys by MacFan4000 · Pull Request #2691 · miraheze/puppet · GitHub | github.com [12:33:54] MacFan4000: sure [12:34:18] !log [reception@mwtask111] finished deploy of {'l10n': True, 'servers': 'mw111,mw112,mw121,mw122'} to mw111,mw112,mw121,mw122 - SUCCESS in 5s [12:34:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:34:37] That's not supposed to list servers twice [12:34:44] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c01955e39852...031f3157c877 [12:34:45] [url] Comparing c01955e39852...031f3157c877 · miraheze/puppet · GitHub | github.com [12:34:46] [02miraheze/puppet] 07MacFan4000 03031f315 - Manage /home/ssl-admins recursively + ensure existence and permissions for /etc/puppetlabs/puppet/ssl-keys (#2691) [12:34:47] [02puppet] 07Reception123 closed pull request 03#2691: Manage /home/ssl-admins recursively + ensure existence and permissions for /etc/puppetlabs/puppet/ssl-keys - 13https://github.com/miraheze/puppet/pull/2691 [12:34:47] [url] Page not found · GitHub · GitHub | github.com [12:34:55] But I don't care because v3 redoes that logic [12:36:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.52, 1.68, 1.94 [12:36:28] There was no start either [12:36:37] Reception123: 5s is fast! [12:36:46] That's why you should only deploy what you need [12:37:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.28, 11.11, 10.58 [12:37:45] probably because i18n ran everywhere before I guess [12:37:51] well except mw111 [12:38:03] Reception123: that's not how it works at all [12:38:18] i18n only runs on mwtask [12:38:26] Then it syncs out to the rest [12:39:01] By doing --servers without task, you're telling it to just copy what's on task to them servers [12:39:28] And no --world means it only looks in 1 directory and picks up changed files [12:39:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.37, 10.41, 9.04 [12:39:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.21, 11.63, 10.87 [12:40:04] ah so that's why it was so fast [12:40:33] Reception123: not using --world means the local i18n files won't match but we ignore them anyway [12:40:50] And there's a task to delete the i18n/*.json anyway from the live copy [12:40:59] And default to using staging [12:41:13] ah [12:41:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.50, 9.40, 8.84 [12:42:45] PROBLEM - biblestrength.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query biblestrength.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:43:33] Reception123: that's complex though [12:44:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.91, 1.98, 1.93 [12:45:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.35, 3.79, 3.27 [12:47:37] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.24, 11.06, 9.22 [12:47:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.47, 12.06, 11.30 [12:47:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 3.33, 3.16 [12:49:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.91, 10.86, 9.58 [12:49:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.92, 10.23, 9.15 [12:51:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.75, 10.22, 9.52 [12:51:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.95, 10.00, 9.21 [12:53:09] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.39, 9.16, 9.22 [12:55:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.81, 3.15 [12:55:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.39, 11.64, 11.57 [12:59:22] Look like my change worked which means that permissions should never again be an issue [12:59:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.49, 3.17 [13:00:20] PROBLEM - gluster111 Puppet on gluster111 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[glusterfs.key],File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:00:59] great :) [13:01:14] PROBLEM - reports121 Puppet on reports121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:01:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.02, 3.29, 3.13 [13:01:53] PROBLEM - mon111 Puppet on mon111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:02:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.97, 1.88, 2.00 [13:03:22] Hmm [13:03:26] Error: /Stage[main]/Ssl::Hiera/Ssl::Hiera::Certs[wwwrothwellleedscouk]/File[/etc/ssl/private/www.rothwell-leeds.co.uk.key]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///ssl-keys/www.rothwell-leeds.co.uk.key [13:03:42] Reception123: not sure why we are getting this for every private key [13:04:06] PROBLEM - mem121 Puppet on mem121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:04:25] it’s the cause of the mass puppet failures [13:04:35] MacFan4000: check the dir [13:04:42] make sure they are there [13:04:48] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:05:21] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 132 failures. Last run 2 minutes ago with 132 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [13:05:52] I'm going to have no access in ~5 minutes so not the best timing [13:06:08] paladox: ^ [13:06:11] RhinosF1: everything is there [13:06:22] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 426 failures. Last run 3 minutes ago with 426 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [13:06:23] MacFan4000: what perms [13:06:37] although puppet should be root so always have access [13:06:38] The only change I made is that ssl-admins have access instead of just root [13:06:56] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 426 failures. Last run 2 minutes ago with 426 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [13:07:01] this happens before when we fixed perms [13:07:05] happend [13:07:22] paladox did something to fix I don’t know what [13:07:27] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Puppet has 426 failures. Last run 2 minutes ago with 426 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [13:07:39] PROBLEM - bast101 Puppet on bast101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:08:33] Currently everything is root:ssl-admins and 0770 [13:08:39] ok [13:09:24] before it was root:root in most cases and in at least one case it was root:ssl-admins [13:10:03] surely ssl-admins:ssl-admins would make more sense [13:10:34] ssl-admins is a group not a user [13:10:42] ah [13:10:45] Only actual users can own stuff [13:11:02] which is why we are only ensuring that it is the group [13:11:06] that rules out root not reading stuff [13:11:58] PROBLEM - db111 Puppet on db111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:12:07] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/miraheze.wiki.key] [13:12:21] Reception123: can you at least stop pppet gloablly [13:12:44] RECOVERY - biblestrength.net - reverse DNS on sslhost is OK: SSL OK - biblestrength.net reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [13:12:51] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:13:09] PROBLEM - ldap111 Puppet on ldap111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:13:13] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:13:16] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:14:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.10, 1.74, 1.81 [13:14:25] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.45, 3.60, 3.09 [13:15:01] PROBLEM - mw102 Puppet on mw102 is CRITICAL: CRITICAL: Puppet has 426 failures. Last run 2 minutes ago with 426 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [13:15:15] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:15:18] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 426 failures. Last run 3 minutes ago with 426 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [13:15:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.75, 3.37, 3.13 [13:15:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.67, 8.70, 10.01 [13:17:00] PROBLEM - es121 Puppet on es121 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/elasticsearch/ssl/wildcard.miraheze.org-2020-2.key] [13:17:01] Hmm? [13:17:10] What’s happened? [13:17:27] paladox: Reception123 merged https://github.com/miraheze/puppet/compare/c01955e39852...031f3157c877 [13:17:27] [url] Comparing c01955e39852...031f3157c877 · miraheze/puppet · GitHub | github.com [13:17:36] PROBLEM - gluster101 Puppet on gluster101 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[glusterfs.key],File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:17:38] puppet now failing everywhere [13:17:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.54, 3.06, 3.05 [13:17:54] What’s the error [13:18:15] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.67, 3.70, 3.26 [13:18:17] paladox: i only have what icinga-miraheze says. I don't have access. MacFan4000 ? [13:18:50] PROBLEM - jobchron121 Puppet on jobchron121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:19:23] 09:03 Error: /Stage[main]/Ssl::Hiera/Ssl::Hiera::Certs[wwwrothwellleedscouk]/File[/etc/ssl/private/www.rothwell-leeds.co.uk.key]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///ssl-keys/www.rothwell-leeds.co.uk.key [13:19:27] paladox: ^^ [13:19:37] hmm [13:19:39] Happens for all private keys [13:19:42] so [13:19:51] you managed to delete the keys? [13:19:54] PROBLEM - es101 Puppet on es101 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/elasticsearch/ssl/wildcard.miraheze.org-2020-2.key] [13:19:58] No they are there [13:20:05] huh [13:20:11] but the error suggests it's not [13:20:13] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.48, 3.90, 3.38 [13:20:35] oh hold on [13:20:38] you changed the perms [13:20:42] This happened last time we fixed permissions and changing them back didn’t help [13:20:52] You then did something, don’t know what [13:20:59] yeh puppet cannot read it [13:21:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.27, 9.81, 8.76 [13:21:29] PROBLEM - db121 Puppet on db121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:21:46] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:22:13] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.42, 3.27, 3.21 [13:22:19] paladox: before only root could read/write (root:root) and now ssl-admins can as well (root:ssl-admins) [13:22:32] So don’t know why it is suddenly the case [13:22:47] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:23:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.25, 10.35, 9.10 [13:23:13] PROBLEM - mem101 Puppet on mem101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:23:16] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:23:25] PROBLEM - prometheus101 Puppet on prometheus101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:23:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.59, 3.30 [13:23:41] i really don't know how this broke... [13:23:43] PROBLEM - mw112 Puppet on mw112 is CRITICAL: CRITICAL: Puppet has 426 failures. Last run 2 minutes ago with 426 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [13:23:51] i'm out of ideas [13:24:19] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:24:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.22, 10.67, 10.43 [13:25:01] paladox: can we revert to what it was before [13:25:06] no [13:25:07] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 426 failures. Last run 3 minutes ago with 426 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [13:25:09] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.72, 9.91, 9.10 [13:25:12] because i don't know what it was before [13:25:16] PROBLEM - es111 Puppet on es111 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/elasticsearch/ssl/wildcard.miraheze.org-2020-2.key] [13:25:26] MacFan4000: what 770 still the same [13:25:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.58, 3.12, 3.16 [13:25:47] > 2022-07-08T13:25:32.754Z INFO [qtp1071587293-33018] [puppetserver] Puppet File does not exist or is not accessible: /etc/puppetlabs/puppet/ssl-keys/dragdown.wiki.key [13:25:51] paladox: does puppet log changes [13:25:54] perm issues [13:25:55] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [13:25:58] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 426 failures. Last run 2 minutes ago with 426 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [13:26:06] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 426 failures. Last run 2 minutes ago with 426 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [13:26:18] Notice: /Stage[main]/Ssl/File[/etc/puppetlabs/puppet/ssl-keys/wiki.wisrail.de.key]/mode: mode changed '0644' to '0770' [13:26:19] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.52, 10.69, 9.39 [13:26:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.12, 11.09, 10.60 [13:26:40] Notice: /Stage[main]/Ssl/File[/home/ssl-admins/ssl-keys/.git/objects/d4/006ec6f9e0665ef6ad40ef951d250d6ca2877e]/owner: owner changed 'puppet' to 'root' [13:27:17] seems like this is f*cked up. different perms etc [13:27:29] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 3 minutes ago with 20 failures. Failed resources (up to 3 shown): File[/etc/postgresql/13/main/ssl/wildcard.miraheze.org.key],File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key] [13:27:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-2691-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/86004d48476c [13:27:31] [url] Page not found · GitHub · GitHub | github.com [13:27:32] [02miraheze/puppet] 07paladox 0386004d4 - Revert "Manage /home/ssl-admins recursively + ensure existence and permissions for /etc/puppetlabs/puppet/ssl-keys (#2691)" [13:27:33] [02puppet] 07paladox created branch 03revert-2691-patch-9 - 13https://github.com/miraheze/puppet [13:27:34] [url] Page not found · GitHub · GitHub | github.com [13:27:35] [02puppet] 07paladox opened pull request 03#2692: Revert "Manage /home/ssl-admins recursively + ensure existence and permissions for /etc/puppetlabs/puppet/ssl-keys" - 13https://github.com/miraheze/puppet/pull/2692 [13:27:35] [url] Page not found · GitHub · GitHub | github.com [13:27:38] [02puppet] 07paladox closed pull request 03#2692: Revert "Manage /home/ssl-admins recursively + ensure existence and permissions for /etc/puppetlabs/puppet/ssl-keys" - 13https://github.com/miraheze/puppet/pull/2692 [13:27:38] [url] Page not found · GitHub · GitHub | github.com [13:27:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/031f3157c877...7dd55acc2d8c [13:27:40] [url] Comparing 031f3157c877...7dd55acc2d8c · miraheze/puppet · GitHub | github.com [13:27:41] [02miraheze/puppet] 07paladox 037dd55ac - Revert "Manage /home/ssl-admins recursively + ensure existence and permissions for /etc/puppetlabs/puppet/ssl-keys (#2691)" (#2692) [13:27:42] [02miraheze/puppet] 07paladox deleted branch 03revert-2691-patch-9 [13:27:44] [02puppet] 07paladox deleted branch 03revert-2691-patch-9 - 13https://github.com/miraheze/puppet [13:27:44] ... [13:27:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:28:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.46, 10.83, 9.59 [13:28:18] https://www.irccloud.com/pastebin/ds5o4Vwl/ [13:28:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.85, 11.16, 10.70 [13:28:28] why is it trying to remove /home/puppet-users????? [13:28:59] paladox: group was renamed that dir is no longer used [13:29:12] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 426 failures. Last run 2 minutes ago with 426 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [13:29:18] oh [13:29:29] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 426 failures. Last run 3 minutes ago with 426 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [13:30:38] PROBLEM - graylog121 Puppet on graylog121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:30:42] https://www.irccloud.com/pastebin/KDbCCFJf/ [13:30:44] MacFan4000: are we sure it should be root:ssl-admins and not puppet:ssl-admins [13:30:54] all the messages paladox gives show that [13:31:04] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.62, 3.59, 3.32 [13:31:18] Most things before were root:root [13:31:27] PROBLEM - gluster121 Puppet on gluster121 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[glusterfs.key],File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [13:31:30] Not sure [13:31:31] Notice: /Stage[main]/Ssl/File[/etc/puppetlabs/puppet/ssl-keys/dreamsit.com.br.key]/mode: mode changed '0644' to '0770' [13:31:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.18, 11.20, 10.13 [13:32:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.87, 11.74, 10.23 [13:32:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.02, 8.83, 9.88 [13:32:12] MacFan4000: I'm not sure the change was well thought out [13:32:27] 0644 would have let anyone read [13:32:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 3.91, 3.47 [13:32:35] which is why puppet could before [13:32:40] !log root@puppet111:/home# chown -R puppet:root /etc/puppetlabs/puppet/ssl-keys [13:32:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:32:57] i hate puppet errors [13:32:59] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.25, 3.19, 3.22 [13:33:02] it would need to be puppet:ssl-admins I think and 0660 likely better [13:33:07] never really telling me what exactly is the issue [13:33:52] [02puppet] 07RhinosF1 opened pull request 03#2693: Revert "Revert "Manage /home/ssl-admins recursively + ensure existence and permissions for /etc/puppetlabs/puppet/ssl-keys"" - 13https://github.com/miraheze/puppet/pull/2693 [13:33:52] [url] Page not found · GitHub · GitHub | github.com [13:34:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.05, 10.97, 10.15 [13:34:28] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.82, 3.32, 3.29 [13:34:42] [02puppet] 07RhinosF1 synchronize pull request 03#2693: Revert "Revert "Manage /home/ssl-admins recursively + ensure existence and permissions for /etc/puppetlabs/puppet/ssl-keys"" - 13https://github.com/miraheze/puppet/pull/2693 [13:34:42] ... [13:35:04] paladox: ^ should work [13:35:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.30, 10.14, 9.93 [13:35:38] I would disable puppet fleet wide though to deploy [13:35:54] seems to work now [13:35:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.28, 9.73, 9.79 [13:36:10] paladox: yeah but we need to do what MacFan4000 was trying right [13:36:33] Sorry for causing such a mess [13:36:35] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.47, 3.81, 2.16 [13:36:51] [02puppet] 07paladox closed pull request 03#2693: Revert "Revert "Manage /home/ssl-admins recursively + ensure existence and permissions for /etc/puppetlabs/puppet/ssl-keys"" - 13https://github.com/miraheze/puppet/pull/2693 [13:36:52] [url] Page not found · GitHub · GitHub | github.com [13:36:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7dd55acc2d8c...7e2d6ad231b6 [13:36:53] [url] Comparing 7dd55acc2d8c...7e2d6ad231b6 · miraheze/puppet · GitHub | github.com [13:36:54] [02miraheze/puppet] 07RhinosF1 037e2d6ad - Revert "Revert "Manage /home/ssl-admins recursively + ensure existence and permissions for /etc/puppetlabs/puppet/ssl-keys"" (#2693) [13:37:24] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:37:29] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [13:39:02] paladox: can you confirm it works [13:39:16] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [13:39:32] does this mean we can delete /home/puppet-users and also the user on puppet? [13:39:57] RECOVERY - db111 Puppet on db111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:40:06] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:40:23] > Warning: The directory '/home/ssl-admins' contains 12119 entries, which exceeds the default soft limit 1000 and may cause excessive resource consumption and degraded performance. To remove this warning set a value for `max_files` parameter or consider using an alternate method to manage large directory trees [13:40:51] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:41:12] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [13:41:19] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [13:42:26] paladox: this doesn't but that should be done if the migration is done [13:42:31] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.33, 2.00, 1.85 [13:42:39] and that warning is not good [13:42:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.68, 3.22 [13:42:55] RECOVERY - mw102 Puppet on mw102 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [13:42:56] I don't want to blindly change it [13:43:03] but why did it work before [13:43:09] RECOVERY - ldap111 Puppet on ldap111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:43:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.75, 3.47 [13:43:13] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [13:43:14] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:44:17] paladox: please delete /home/ssl-admins/ssl-keys-writable - that was me trying things, and I am all done with it now [13:44:28] ok [13:44:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.37, 3.15 [13:45:00] RECOVERY - es121 Puppet on es121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:45:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.62, 3.01, 3.23 [13:45:34] MacFan4000: please don't just try things when we're fixing stuff. It makes it harder to see why things error if the base is changing. [13:45:36] RECOVERY - gluster101 Puppet on gluster101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:46:31] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.83, 1.35, 1.62 [13:46:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.61, 3.06, 3.06 [13:46:50] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [13:47:53] RECOVERY - es101 Puppet on es101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:49:29] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [13:50:19] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:50:47] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:51:13] RECOVERY - mem101 Puppet on mem101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:51:26] RECOVERY - prometheus101 Puppet on prometheus101 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:51:43] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:51:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.07, 4.64, 3.61 [13:51:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 10.11, 5.87, 4.25 [13:52:12] RECOVERY - mw112 Puppet on mw112 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [13:52:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.36, 4.96, 3.79 [13:53:14] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [13:53:15] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:53:16] RECOVERY - es111 Puppet on es111 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [13:53:26] RhinosF1: this was last night [13:53:35] Ah ok [13:53:49] MacFan4000: does it look to be working now [13:54:02] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:54:46] The chance pulled in just now and we aren’t seeing puppet fails yet [13:54:57] That's good [13:55:01] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:55:13] Maybe force a run somewhere MacFan4000 [13:55:47] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [13:55:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:56:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.68, 10.59, 10.09 [13:57:09] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:58:13] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.77, 3.57, 3.26 [13:58:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.43, 9.59, 9.80 [13:59:14] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:59:26] RECOVERY - gluster121 Puppet on gluster121 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [13:59:51] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [14:00:13] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 1.63, 2.88, 3.05 [14:00:18] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:00:20] RECOVERY - gluster111 Puppet on gluster111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:00:31] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.86, 2.64, 1.90 [14:00:32] RhinosF1: just ran on mw101 without issues [14:00:38] RECOVERY - graylog121 Puppet on graylog121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:00:48] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [14:00:52] MacFan4000: \o/ [14:01:09] I assume you can access the files with the new perms fine [14:02:05] RECOVERY - mem121 Puppet on mem121 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [14:03:39] RECOVERY - bast101 Puppet on bast101 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:04:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.70, 1.84, 1.98 [14:04:31] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.15, 1.67, 1.66 [14:06:50] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:12:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.94, 10.12, 9.30 [14:14:14] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.07, 9.72, 9.26 [14:14:39] Yes [14:18:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.40, 2.02, 1.93 [14:21:16] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:19] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:40] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:48] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:13] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.316 second response time [14:23:13] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [14:23:24] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:23:38] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [14:23:48] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.700 second response time [14:24:07] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [14:24:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 2.00, 1.98 [14:29:10] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.05, 1.85, 1.42 [14:31:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.48, 1.68, 1.41 [14:31:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.71, 9.99, 9.12 [14:33:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.01, 10.36, 9.34 [14:33:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [14:33:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.59, 9.72, 9.12 [14:35:26] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.20, 9.40, 9.13 [14:38:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [14:42:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.08, 9.81, 9.37 [14:44:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.83, 1.84 [14:46:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.98, 1.86, 1.85 [14:47:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.23, 10.27, 9.86 [14:47:51] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.28, 10.15, 9.69 [14:48:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.49, 2.11, 1.94 [14:49:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.92, 10.11, 9.85 [14:54:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.78, 1.72, 1.38 [14:56:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.96, 1.98 [14:56:54] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.45, 1.68, 1.42 [14:57:29] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.87, 3.74, 3.30 [14:58:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.06, 1.90, 1.95 [14:59:24] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.74, 3.40, 3.23 [15:03:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.58, 9.85, 9.21 [15:03:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.98, 10.21, 10.08 [15:04:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.73, 1.85, 1.91 [15:05:26] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.50, 9.29, 9.07 [15:06:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [15:09:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.26, 9.45, 9.84 [15:18:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.26, 2.17, 1.93 [15:21:31] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.17, 4.01, 3.47 [15:22:19] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:23:26] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.38, 3.67, 3.40 [15:25:21] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.52, 3.29, 3.29 [15:26:31] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.89, 1.88, 1.34 [15:26:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [15:28:31] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.23, 2.04, 1.44 [15:30:31] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.02, 1.57, 1.34 [15:33:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.73, 10.43, 9.39 [15:35:32] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.53, 9.64, 9.21 [15:51:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [16:03:29] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.57, 4.69, 2.83 [16:04:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.80, 2.00 [16:11:15] [02puppet] 07Universal-Omega opened pull request 03#2694: Remove /home/puppet-users definition - 13https://github.com/miraheze/puppet/pull/2694 [16:11:16] [url] Page not found · GitHub · GitHub | github.com [16:15:22] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.45, 1.65, 2.00 [16:15:45] paladox: ^ and also /home/puppet-users can be manually deleted, that was not doing it recursively so absenting it did nothing, but can just be manual. [16:15:59] ok [16:16:11] [02puppet] 07paladox closed pull request 03#2694: Remove /home/puppet-users definition - 13https://github.com/miraheze/puppet/pull/2694 [16:16:11] [url] Page not found · GitHub · GitHub | github.com [16:16:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7e2d6ad231b6...9e9b530b340c [16:16:13] [url] Comparing 7e2d6ad231b6...9e9b530b340c · miraheze/puppet · GitHub | github.com [16:16:14] [02miraheze/puppet] 07Universal-Omega 039e9b530 - Remove /home/puppet-users definition (#2694) [16:18:07] https://github.com/miraheze/puppet/blob/943035e0e119995a09d29360306a4ecd9255bd9c/modules/users/data/data.yaml#L32 should be removed [16:18:07] [url] puppet/data.yaml at 943035e0e119995a09d29360306a4ecd9255bd9c · miraheze/puppet · GitHub | github.com [16:18:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.93, 2.13, 1.94 [16:18:50] paladox: I think John said it had to be left or something because it exists. [16:18:59] oh [16:19:30] !log puppet111: rm -rf /home/puppet-users [16:19:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:20:01] Thanks paladox! [16:20:17] yw [16:20:31] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [16:23:18] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.03, 1.25, 1.65 [16:24:00] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.27, 3.36, 2.98 [16:25:55] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.49, 3.02, 2.90 [16:26:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.42, 1.86, 1.94 [16:33:03] [02puppet] 07Universal-Omega reviewed pull request 03#2456 commit - 13https://github.com/miraheze/puppet/pull/2456#discussion_r916975910 [16:33:04] [url] mw-python-tools: major improvements by RhinosF1 · Pull Request #2456 · miraheze/puppet · GitHub | github.com [16:34:17] [02puppet] 07RhinosF1 reviewed pull request 03#2456 commit - 13https://github.com/miraheze/puppet/pull/2456#discussion_r916977340 [16:34:19] [url] mw-python-tools: major improvements by RhinosF1 · Pull Request #2456 · miraheze/puppet · GitHub | github.com [16:34:32] CosmicAlpha: can you try it? [16:35:31] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.86, 3.08, 2.85 [16:36:20] RhinosF1: it did not work when I ran just that with env after php. I did it on accident once, it did not work right. Unless I am thinking of something else. I am mobile now, can in about 30 minutes. [16:36:59] CosmicAlpha: ok, I'll tweak WikiCommand to support injecting it [16:38:16] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.43, 1.51, 1.70 [16:38:20] Shouldn't be hard [16:39:21] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.94, 3.00, 2.87 [16:41:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.31, 10.01, 9.07 [16:43:13] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.89, 3.70, 3.20 [16:43:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.76, 9.16, 8.87 [16:44:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.65, 1.68 [16:45:08] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 5.61, 4.27, 3.46 [16:46:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.13, 1.76, 1.72 [16:52:49] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.08, 3.94, 3.66 [16:55:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.86, 3.26, 3.89 [16:58:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.64, 1.94, 1.98 [17:01:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.03, 9.73, 9.15 [17:03:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.98, 9.20, 9.04 [17:05:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.02, 3.72, 3.81 [17:06:13] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.23, 3.68, 3.58 [17:07:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.81, 3.79, 3.82 [17:08:13] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.37, 3.52, 3.53 [17:08:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.24, 1.98, 1.96 [17:09:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.50, 10.17, 9.47 [17:09:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.81, 3.82 [17:11:36] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.84, 11.02, 9.85 [17:12:13] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.56, 3.18, 3.40 [17:13:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.69, 11.31, 10.12 [17:13:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.72, 3.80 [17:14:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.11, 10.49, 9.39 [17:15:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.95, 4.11, 3.92 [17:15:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 10.19, 10.01, 9.33 [17:17:36] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.37, 11.29, 10.34 [17:18:13] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.97, 3.43, 3.40 [17:19:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.61, 11.27, 10.44 [17:21:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 4.00, 3.97 [17:22:13] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.10, 3.71, 3.52 [17:22:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.53, 10.80, 9.92 [17:23:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 4.06, 3.99 [17:24:13] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.58, 3.29, 3.39 [17:24:42] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.92, 9.99, 9.72 [17:25:37] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.98, 9.90, 10.16 [17:42:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.80, 1.96 [17:46:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.15, 1.94, 1.99 [17:50:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/69330820ec01 [17:50:28] [02miraheze/mw-config] 07Universal-Omega 036933082 - Set $smwgNamespacesWithSemanticLinks for constantnoblewiki [17:50:30] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config [17:50:30] [url] Page not found · GitHub · GitHub | github.com [17:50:31] [02mw-config] 07Universal-Omega opened pull request 03#4810: Set $smwgNamespacesWithSemanticLinks for constantnoblewiki - 13https://github.com/miraheze/mw-config/pull/4810 [17:50:32] [url] Page not found · GitHub · GitHub | github.com [17:51:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.48, 9.67, 8.60 [17:51:41] miraheze/mw-config - Universal-Omega the build passed. [17:53:51] PROBLEM - cloud11 Current Load on cloud11 is CRITICAL: CRITICAL - load average: 56.33, 38.85, 24.69 [17:54:38] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:28] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.05, 9.53, 8.90 [17:55:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.99, 10.65, 9.50 [17:55:51] RECOVERY - cloud11 Current Load on cloud11 is OK: OK - load average: 22.75, 31.83, 23.83 [17:56:38] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.580 second response time [17:56:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/69330820ec01...bf7c9704bc68 [17:56:42] [url] Comparing 69330820ec01...bf7c9704bc68 · miraheze/mw-config · GitHub | github.com [17:56:43] [02miraheze/mw-config] 07Universal-Omega 03bf7c970 - Update SemanticMediaWiki.php [17:56:44] [02mw-config] 07Universal-Omega synchronize pull request 03#4810: Set $smwgNamespacesWithSemanticLinks for constantnoblewiki - 13https://github.com/miraheze/mw-config/pull/4810 [17:56:45] [url] Page not found · GitHub · GitHub | github.com [17:56:46] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [17:57:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.92, 10.03, 9.41 [17:57:54] miraheze/mw-config - Universal-Omega the build passed. [17:58:06] CosmicAlpha: any other thoughts [17:58:44] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [17:58:56] I'm more concerned it's only really been unit tested by me [17:59:33] and a very brief smoke end to end test on my laptop [17:59:37] RhinosF1: still need to do full review. Just noticed that. I'll admit I dont understand everything you did, as I am not the best in python, but I get it enough to at least point out some things if I find them. I can probably try and test later also. [18:00:12] CosmicAlpha: the best thing to do is pull it locally on test101, try a lot in no-op and then if not try live [18:00:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.83, 1.98 [18:02:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.98, 2.02 [18:02:48] If it works in no-op, I'm fairly confident it should live [18:02:59] as long as I've not been an idiot and missed a stage off [18:03:07] Because it's type checked & linted heavily [18:04:12] It would be good to know what you've done to test [18:04:14] With logs [18:04:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.91, 1.95, 2.00 [18:04:28] Because I can try and add tests for no-ops you've checked [18:04:41] I added mwscript without wiki too [18:04:54] Both are fully tested as far as my knowledge allows [18:05:04] Without setting up full end to end test environments [18:05:57] I would like to consider adding the config to a separate file [18:06:05] Maybe stored in mw-config [18:06:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.73, 2.17, 2.07 [18:06:27] So you don't need puppet changes to do things like add new repos, change the servers etc [18:07:41] RhinosF1: will let you know when I test. [18:07:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.92, 10.61, 10.05 [18:08:21] CosmicAlpha: does moving config out sound good? [18:08:37] I mainly thinking too because I don't want to update the tests every time we change servers or repos [18:08:59] RhinosF1: not just yet, maybe eventually, but for now I would rather get this one done. [18:09:07] CosmicAlpha: ye sure [18:09:38] It would need some thought [18:09:52] As I think we'd need packages to deploy to config CI to ensure it's valid [18:10:15] Just make sure we do it before new servers come [18:10:44] Reception123, paladox: any idea when ^ will be [18:11:00] I want to expand configs scope anyway though, like moving some hooks to config from MirahezeMagic that could go in MirahezeFunctions also, easier to modify that way if can. Also want to see if we can get i18n working in config, so that i18n there can be used for ManageWiki etc... but that is another day. [18:11:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.50, 10.05, 9.99 [18:11:51] [02mw-config] 07Universal-Omega closed pull request 03#4810: Set $smwgNamespacesWithSemanticLinks for constantnoblewiki - 13https://github.com/miraheze/mw-config/pull/4810 [18:11:52] [url] Page not found · GitHub · GitHub | github.com [18:11:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7b5f64fcfb2c...5fbe7c4adb94 [18:11:54] [url] Comparing 7b5f64fcfb2c...5fbe7c4adb94 · miraheze/mw-config · GitHub | github.com [18:11:54] [02miraheze/mw-config] 07Universal-Omega 035fbe7c4 - Set $smwgNamespacesWithSemanticLinks for constantnoblewiki (#4810) [18:11:56] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config [18:11:56] [url] Page not found · GitHub · GitHub | github.com [18:11:57] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 [18:12:45] I'm not sure, but I think John and Paladox have discussed things [18:12:48] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.60, 3.20, 2.95 [18:13:06] miraheze/mw-config - Universal-Omega the build passed. [18:14:12] CosmicAlpha: making mw deploy configurable would be just remove the constants from the top of the file, add it to the config repo and then import that in the script [18:14:18] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.26, 2.19, 1.30 [18:14:43] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.11, 2.92, 2.89 [18:18:19] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.39, 1.80, 1.35 [18:20:00] Reception123: I almost thought Grafana emails were broken there for a bit until I got some today, I had not gotten any in 2 days, usually at least one a day, but now it seems not even that sometimes. They are a lot rarer now. [18:20:17] RhinosF1: cool. We can wait, but yeah eventually good idea. [18:20:18] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.94, 1.54, 1.31 [18:20:54] heh [18:22:33] Reception123: before the PHP rewrite they were given sometimes multiple times an hour also, like 80 per day, now not even 1 per day sometimes, so that is great! Easier to navigate mail also. [18:22:55] Depends on the day, but a lot better regardless. [18:24:07] !log [@mwtask111] starting deploy of {'config': True} to all [18:24:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 10.99, 10.30 [18:24:31] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 24s [18:24:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:45] !log [@test101] starting deploy of {'config': True} to all [18:24:47] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 2s [18:25:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:00] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/b77c9364116e [18:25:02] [02miraheze/ssl] 07Universal-Omega 03b77c936 - Remove ircwiki.cf cert [18:25:03] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [18:25:04] [url] Page not found · GitHub · GitHub | github.com [18:25:05] [02ssl] 07Universal-Omega opened pull request 03#563: Remove ircwiki.cf cert - 13https://github.com/miraheze/ssl/pull/563 [18:25:06] ... [18:25:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.84, 9.63, 8.53 [18:25:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:40] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b77c9364116e...49f74c1b690c [18:25:41] [url] Comparing b77c9364116e...49f74c1b690c · miraheze/ssl · GitHub | github.com [18:25:42] [02miraheze/ssl] 07Universal-Omega 0349f74c1 - Update certs.yaml [18:25:43] [02ssl] 07Universal-Omega synchronize pull request 03#563: Remove ircwiki.cf cert - 13https://github.com/miraheze/ssl/pull/563 [18:25:44] [url] Page not found · GitHub · GitHub | github.com [18:26:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.54, 10.86, 10.34 [18:26:40] CosmicAlpha: yeah, my main concern is redoing all the tests every time we update comfig [18:26:54] It would be ideal if the script was completely customisable [18:27:11] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.00, 2.39, 1.55 [18:27:12] So anyone can just setup basic config [18:27:31] Far reach but it would be nice if it wasn't too Miraheze specfic [18:30:04] [02ssl] 07Universal-Omega closed pull request 03#563: Remove ircwiki.cf cert - 13https://github.com/miraheze/ssl/pull/563 [18:30:05] [url] Page not found · GitHub · GitHub | github.com [18:30:06] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/dbb800e07233...dc59e39b15c5 [18:30:07] [url] Comparing dbb800e07233...dc59e39b15c5 · miraheze/ssl · GitHub | github.com [18:30:07] [02miraheze/ssl] 07Universal-Omega 03dc59e39 - Remove ircwiki.cf cert [18:30:09] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [18:30:10] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [18:30:11] [url] Page not found · GitHub · GitHub | github.com [18:31:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.04, 1.68, 1.46 [18:31:09] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.04, 8.95, 8.69 [18:33:02] RhinosF1: ? [18:33:29] paladox: when are we expecting new servers [18:33:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.47, 10.68, 10.39 [18:33:59] Oh I guess next month? (John mentioned a month or two)o [18:34:19] RhinosF1: John said software upgrades in a month or two I believe. I don't know if he meant current servers or new ones though. [18:35:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.46, 11.06, 10.57 [18:36:13] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.83, 3.63, 3.15 [18:40:13] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.78, 3.43, 3.20 [18:42:13] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.89, 3.31, 3.18 [18:47:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.44, 9.21, 9.99 [18:50:13] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.17, 3.84, 3.43 [18:51:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.80, 10.48, 9.72 [18:51:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.61, 3.29, 3.93 [18:52:13] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.92, 3.81, 3.47 [18:53:00] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:53:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.20, 10.16, 9.71 [18:54:13] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.83, 4.01, 3.57 [18:56:54] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.84, 1.89, 1.47 [18:57:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.49, 10.89, 10.46 [18:58:13] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.07, 3.57, 3.49 [19:00:57] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.80, 4.00, 3.99 [19:01:08] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.49, 1.48, 1.38 [19:01:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.47, 9.63, 10.05 [19:02:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.26, 3.74, 3.90 [19:07:34] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.25, 3.10, 3.30 [19:07:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.05, 10.98, 10.55 [19:10:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.75, 4.76, 4.20 [19:12:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.13, 11.02, 10.13 [19:13:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.91, 9.37, 10.04 [19:14:22] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.55, 2.06, 1.34 [19:14:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.92, 11.01, 10.21 [19:14:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.77, 2.97, 3.73 [19:15:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.55, 3.43, 3.91 [19:16:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.57, 3.38, 3.81 [19:16:19] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.74, 1.57, 1.24 [19:18:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.00, 4.65, 4.21 [19:18:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.00, 10.02, 10.00 [19:19:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 3.83, 3.94 [19:21:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.25, 3.85, 3.98 [19:22:06] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [19:22:31] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.26, 2.48, 1.71 [19:22:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.52, 4.39, 4.02 [19:24:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.01, 1.91, 1.59 [19:26:29] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.66, 1.41, 1.44 [19:27:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.36, 4.47, 4.17 [19:29:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.75, 3.89 [19:33:03] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.09, 1.96, 1.55 [19:33:03] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 11.48, 7.87, 5.95 [19:33:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.42, 4.28, 4.08 [19:33:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5fbe7c4adb94...4d29575258ad [19:33:34] [url] Comparing 5fbe7c4adb94...4d29575258ad · miraheze/mw-config · GitHub | github.com [19:33:35] [02miraheze/mw-config] 07Universal-Omega 034d29575 - Re-enable EditNotify [19:33:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.55, 3.85 [19:34:44] miraheze/mw-config - Universal-Omega the build passed. [19:34:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.33, 9.75, 9.52 [19:34:57] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.50, 1.49, 1.43 [19:35:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.51, 3.97, 3.98 [19:38:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.46, 3.63, 3.95 [19:39:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.77, 3.76, 3.85 [19:39:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.72, 3.96, 3.86 [19:40:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.41, 3.91, 4.02 [19:41:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.77, 3.86 [19:41:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.49, 3.37, 3.66 [19:44:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.60, 3.84 [19:45:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.89, 4.34, 4.00 [19:45:49] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.95, 1.75, 1.29 [19:46:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.94, 3.94 [19:47:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.79, 3.85 [19:47:43] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.38, 1.61, 1.29 [19:48:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.42, 3.74 [19:49:54] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 5.07, 3.92, 3.36 [19:50:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.81, 10.20, 10.09 [19:50:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.46, 3.70 [19:51:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.69, 3.76 [19:51:51] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.58, 3.87, 3.41 [19:51:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.98, 4.04, 3.75 [19:52:00] !log [@mwtask111] starting deploy of {'config': True} to all [19:52:11] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 11s [19:52:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.64, 3.74 [19:53:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.00, 3.66, 3.64 [19:54:51] !log [@test101] starting deploy of {'config': True} to all [19:54:53] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [19:55:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.96, 3.76 [19:57:42] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.46, 3.10, 3.23 [19:58:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.12, 4.60, 4.09 [20:02:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.08, 10.12, 9.28 [20:03:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.98, 3.96 [20:04:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.21, 3.74, 3.92 [20:04:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.35, 10.00, 9.33 [20:04:43] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.29, 6.39, 7.84 [20:05:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 3.95, 3.93 [20:06:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.74, 10.89, 9.71 [20:06:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.69, 4.16, 4.05 [20:07:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.58, 3.81 [20:08:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.64, 10.54, 9.72 [20:08:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.25, 3.74, 3.90 [20:09:21] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.60, 3.66, 3.33 [20:10:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.50, 3.95, 3.95 [20:10:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.59, 9.33, 9.34 [20:11:15] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.49, 10.99, 10.08 [20:11:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.53, 9.87, 9.63 [20:12:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.73, 3.58, 3.82 [20:13:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.15, 10.17, 9.88 [20:16:35] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.19, 7.57, 7.57 [20:16:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.06, 3.54, 3.73 [20:17:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.42, 3.45, 3.86 [20:17:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.75, 3.00, 3.40 [20:18:34] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.70, 7.25, 7.46 [20:19:06] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.74, 3.90, 3.64 [20:19:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.21, 4.02, 4.01 [20:20:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.32, 3.65, 3.76 [20:21:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.36, 3.45 [20:23:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.81, 3.95 [20:25:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.50, 4.09, 4.03 [20:26:29] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.84, 5.13, 6.40 [20:26:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.97, 4.26, 3.97 [20:27:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.44, 3.49, 3.82 [20:29:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.49, 4.41, 3.85 [20:31:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.66, 3.83, 3.71 [20:32:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4d29575258ad...396ec7300311 [20:32:41] [url] Comparing 4d29575258ad...396ec7300311 · miraheze/mw-config · GitHub | github.com [20:32:42] [02miraheze/mw-config] 07Universal-Omega 03396ec73 - Disable wgVectorSkinMigrationMode [20:33:37] JohnLewis: when do we expect new MWs? [20:33:48] new MWs? [20:33:49] miraheze/mw-config - Universal-Omega the build passed. [20:33:54] Planning in my head when to make deploy-mediawiki properly configurable [20:33:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 3.96, 3.76 [20:34:09] JohnLewis: assuming new cloud servers come and we've raised enough money [20:34:34] Whenever the MW team ask for them when resources are available [20:34:43] Currently the resource problem is up for debate on how to solve [20:35:15] fair [20:35:27] JohnLewis: is there a rough I should plan for [20:35:45] Because the tests are awful without splitting config up and are going to be debt for later [20:35:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.14, 3.46, 3.62 [20:36:11] Until we decide how to move forward, no timeline can be estimated [20:36:39] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.01, 3.76, 3.61 [20:38:36] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.81, 3.78, 3.64 [20:38:41] Is there a not before [20:39:08] I can't see it being this month [20:40:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.20, 3.32, 3.86 [20:41:39] I can plan for August then [20:41:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.53, 3.55 [20:41:56] It's not difficult [20:42:30] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.25, 2.95, 3.33 [20:43:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.64, 3.66 [20:43:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.30, 3.39, 3.49 [20:44:36] It's just when to do it [20:44:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.50, 3.95, 3.98 [20:45:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.99, 3.80 [20:45:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.73, 3.18, 3.40 [20:47:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 4.11, 3.87 [20:49:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.67, 4.20, 3.77 [20:53:33] !log [@mwtask111] starting deploy of {'config': True} to all [20:53:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:53:43] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 9s [20:53:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:53:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.39, 3.60, 3.66 [20:54:17] !log [@test101] starting deploy of {'config': True} to all [20:54:18] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [20:54:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:54:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:55:46] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [20:55:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.64, 4.27, 3.91 [21:01:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.04, 3.92, 3.93 [21:03:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.88, 3.91 [21:04:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.54, 3.22, 3.80 [21:05:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.10, 3.59, 3.80 [21:06:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.18, 4.55, 4.21 [21:07:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 3.83, 3.86 [21:09:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.51, 3.38, 3.69 [21:10:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.50, 3.63, 3.93 [21:14:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 4.14, 4.08 [21:16:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.07, 3.79, 3.96 [21:19:54] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.54, 3.38, 3.08 [21:19:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.33, 3.68, 3.62 [21:22:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.65, 4.00, 3.96 [21:23:48] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.82, 3.21, 3.08 [21:23:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.79, 3.70 [21:24:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.10, 3.70, 3.86 [21:25:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.93, 4.04, 3.81 [21:26:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.13, 3.98, 3.95 [21:29:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.13, 3.43, 3.65 [21:30:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.61, 3.59, 3.83 [21:33:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.41, 3.33, 3.89 [21:33:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.35, 2.85, 3.36 [21:38:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.17, 2.83, 3.37 [21:42:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.74, 3.38, 3.48 [21:43:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.28, 3.99, 3.81 [21:43:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 3.74, 3.48 [21:44:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.51, 4.52, 3.87 [21:45:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.62, 3.48 [21:47:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.50, 3.62, 3.76 [21:47:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.36, 3.96, 3.62 [21:49:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.81, 3.61 [21:50:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.65, 3.62, 3.70 [21:51:35] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:53:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 3.83, 3.65 [21:55:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.66, 3.61 [21:58:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.91, 4.03, 3.81 [21:59:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.62, 2.94, 3.39 [21:59:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.90, 2.00 [22:00:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.83, 3.92, 3.80 [22:01:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.65, 4.05, 3.74 [22:03:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.78, 3.68 [22:04:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.30, 9.47, 7.90 [22:06:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.63, 3.75, 3.72 [22:06:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.00, 8.91, 7.89 [22:08:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.81, 3.31, 3.56 [22:10:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 3.76, 3.54 [22:10:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.14, 4.11, 3.80 [22:11:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.15, 3.52, 3.47 [22:12:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.93, 3.96, 3.79 [22:13:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 3.66, 3.52 [22:13:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.29, 4.01, 3.82 [22:14:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.84, 4.17, 3.88 [22:16:22] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.54, 3.31, 3.03 [22:16:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.95, 3.84 [22:17:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.35, 1.96, 1.85 [22:17:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.90, 3.85 [22:20:21] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.55, 3.16, 3.04 [22:20:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.04, 4.03, 3.86 [22:21:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 4.02, 3.89 [22:31:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.29, 3.69, 3.98 [22:35:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.83, 4.20, 4.10 [22:37:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.81, 3.96 [22:43:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.85, 1.99 [22:45:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.88, 3.51, 3.88 [22:45:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.22, 3.61, 3.76 [22:47:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.49, 2.01, 2.01 [22:47:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.27, 3.61 [22:49:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.73, 4.98, 4.30 [22:49:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.59, 4.57, 4.07 [22:51:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.98, 3.91 [22:53:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.83, 3.98 [22:53:35] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [22:55:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.82, 4.39, 4.17 [22:57:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.90, 3.70, 3.93 [22:59:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.87, 4.12, 4.05 [23:01:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.70, 1.84, 1.99 [23:03:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.72, 3.48, 3.80 [23:04:36] PROBLEM - prometheus101 Current Load on prometheus101 is WARNING: WARNING - load average: 1.29, 1.74, 1.02 [23:06:34] RECOVERY - prometheus101 Current Load on prometheus101 is OK: OK - load average: 0.47, 1.27, 0.93 [23:06:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.82, 3.72, 3.97 [23:09:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.44, 3.46, 3.53 [23:10:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.47, 3.98, 4.00 [23:11:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.68, 3.61 [23:13:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.34, 3.68, 3.61 [23:13:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.24, 1.83, 1.84 [23:13:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 4.27, 3.85 [23:17:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.28, 3.93, 3.88 [23:18:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.49, 3.55, 3.86 [23:19:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.67, 3.94, 3.88 [23:21:10] [02mw-config] 07Universal-Omega closed pull request 03#4806: Disable `$wgCosmosSearchUseActionAPI` by default - 13https://github.com/miraheze/mw-config/pull/4806 [23:21:10] [url] Page not found · GitHub · GitHub | github.com [23:21:12] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [23:21:13] [url] Page not found · GitHub · GitHub | github.com [23:21:14] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [23:21:28] [02mw-config] 07Universal-Omega commented on pull request 03#4806: Disable `$wgCosmosSearchUseActionAPI` by default - 13https://github.com/miraheze/mw-config/pull/4806#issuecomment-1179424910 [23:21:28] [url] Disable `$wgCosmosSearchUseActionAPI` by default by Universal-Omega · Pull Request #4806 · miraheze/mw-config · GitHub | github.com [23:21:35] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:21:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/396ec7300311...289dbb1a5f2d [23:21:59] [url] Comparing 396ec7300311...289dbb1a5f2d · miraheze/mw-config · GitHub | github.com [23:22:00] [02miraheze/mw-config] 07Universal-Omega 03289dbb1 - Add initSiteStats.php to ImportDumpScriptCommand (#4801) [23:22:02] [02mw-config] 07Universal-Omega closed pull request 03#4801: Add initSiteStats.php to ImportDumpScriptCommand - 13https://github.com/miraheze/mw-config/pull/4801 [23:22:02] [url] Page not found · GitHub · GitHub | github.com [23:22:03] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [23:22:05] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [23:22:05] [url] Page not found · GitHub · GitHub | github.com [23:22:24] !log [@mwtask111] starting deploy of {'config': True} to all [23:22:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 3.69, 3.81 [23:22:40] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 15s [23:22:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:56] miraheze/mw-config - Universal-Omega the build passed. [23:23:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.34, 4.03, 3.88 [23:23:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.76, 4.17, 3.91 [23:24:56] [02mw-config] 07Universal-Omega closed pull request 03#4802: Remove (centralauth-)usermerge rights - 13https://github.com/miraheze/mw-config/pull/4802 [23:24:56] [url] Page not found · GitHub · GitHub | github.com [23:24:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/289dbb1a5f2d...61b4f2cdda3e [23:24:58] [url] Comparing 289dbb1a5f2d...61b4f2cdda3e · miraheze/mw-config · GitHub | github.com [23:24:59] [02miraheze/mw-config] 07Universal-Omega 0361b4f2c - Remove (centralauth-)usermerge rights (#4802) [23:25:00] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 [23:25:02] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://github.com/miraheze/mw-config [23:25:02] ... [23:25:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.55, 1.91, 1.97 [23:25:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/61b4f2cdda3e...d92c03f273c5 [23:25:52] [url] Comparing 61b4f2cdda3e...d92c03f273c5 · miraheze/mw-config · GitHub | github.com [23:25:53] [02miraheze/mw-config] 07Universal-Omega 03d92c03f - Update FlaggedRevs defaults to extension defaults (#4789) [23:25:55] [02mw-config] 07Universal-Omega closed pull request 03#4789: Update FlaggedRevs defaults to extension defaults - 13https://github.com/miraheze/mw-config/pull/4789 [23:25:55] [url] Page not found · GitHub · GitHub | github.com [23:25:56] miraheze/mw-config - Universal-Omega the build passed. [23:25:56] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [23:25:58] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [23:25:58] [url] Page not found · GitHub · GitHub | github.com [23:26:51] miraheze/mw-config - Universal-Omega the build passed. [23:28:50] [02mw-config] 07Universal-Omega edited pull request 03#4807: Remove the `$wgCosmosUseWVUISearch` config - 13https://github.com/miraheze/mw-config/pull/4807 [23:28:50] [url] Page not found · GitHub · GitHub | github.com [23:29:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.82, 3.93 [23:30:36] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/cosmos.json /srv/mediawiki/w/maintenance/populateWikiSettings.php --wgsetting=wgCosmosUseWVUISearch --sourcelist=false --remove (START) [23:30:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:46] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/cosmos.json /srv/mediawiki/w/maintenance/populateWikiSettings.php --wgsetting=wgCosmosUseWVUISearch --sourcelist=false --remove (END - exit=2) [23:30:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.26, 4.40, 4.13 [23:31:20] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/cosmos.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgCosmosUseWVUISearch --sourcelist=false --remove (START) [23:31:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:37] [02mw-config] 07Universal-Omega edited pull request 03#4807: Remove the `$wgCosmosUseWVUISearch` config - 13https://github.com/miraheze/mw-config/pull/4807 [23:31:38] [url] Page not found · GitHub · GitHub | github.com [23:31:54] [02mw-config] 07Universal-Omega closed pull request 03#4807: Remove the `$wgCosmosUseWVUISearch` config - 13https://github.com/miraheze/mw-config/pull/4807 [23:31:55] ... [23:31:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/d92c03f273c5...6b400eb11345 [23:31:56] [url] Comparing d92c03f273c5...6b400eb11345 · miraheze/mw-config · GitHub | github.com [23:31:57] [02miraheze/mw-config] 07Universal-Omega 036b400eb - Remove the `$wgCosmosUseWVUISearch` config (#4807) [23:31:59] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7 - 13https://github.com/miraheze/mw-config [23:31:59] [url] Page not found · GitHub · GitHub | github.com [23:32:00] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7 [23:33:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.39, 3.55, 3.84 [23:33:10] miraheze/mw-config - Universal-Omega the build passed. [23:33:19] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.45, 1.77, 1.38 [23:33:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.25, 3.66, 3.96 [23:35:18] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.51, 1.33 [23:36:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.92, 3.79, 3.99 [23:36:46] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/cosmos.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgCosmosUseWVUISearch --sourcelist=false --remove (END - exit=0) [23:36:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:36:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.56, 10.31, 9.75 [23:36:59] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [23:37:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:37:09] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [23:37:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:38:51] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.04, 9.23, 9.42 [23:38:53] !log [universalomega@test101] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgCosmosUseWVUISearch --sourcelist=false --remove (START) [23:38:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:39:00] !log [universalomega@test101] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgCosmosUseWVUISearch --sourcelist=false --remove (END - exit=0) [23:39:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:39:13] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [23:39:17] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 3s [23:39:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:39:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:40:21] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.16, 3.25, 2.87 [23:40:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6b400eb11345...c2e7452eadda [23:40:25] [url] Comparing 6b400eb11345...c2e7452eadda · miraheze/mw-config · GitHub | github.com [23:40:26] [02miraheze/mw-config] 07Universal-Omega 03c2e7452 - GlobalLogging: add all `use` statements (#4800) [23:40:27] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [23:40:27] [02mw-config] 07Universal-Omega closed pull request 03#4800: GlobalLogging: add all `use` statements - 13https://github.com/miraheze/mw-config/pull/4800 [23:40:28] ... [23:40:29] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [23:40:30] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [23:40:31] [url] Page not found · GitHub · GitHub | github.com [23:40:31] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 3s [23:40:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:40:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:41:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.15, 2.78, 3.39 [23:41:13] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [23:41:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:41:22] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [23:41:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:41:31] miraheze/mw-config - Universal-Omega the build passed. [23:41:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.53, 1.94, 1.83 [23:41:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.12, 2.70, 3.39 [23:42:21] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.92, 3.54, 3.03 [23:45:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.58, 1.91, 1.85 [23:45:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.17, 3.44 [23:46:21] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.68, 3.14, 2.99 [23:46:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.67, 2.68, 3.35 [23:47:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.65, 2.13, 1.93 [23:47:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.25, 3.17, 3.40 [23:50:21] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.90, 3.48, 3.16 [23:52:21] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.88, 3.15, 3.07 [23:55:22] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 3.30, 3.31 [23:55:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.36, 1.91, 1.98 [23:58:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.71, 4.73, 3.91 [23:59:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.34, 4.02, 3.63