[00:01:07] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.07, 6.98, 7.72 [00:02:34] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.23, 6.83, 7.09 [00:02:50] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.51, 6.91, 7.04 [00:03:26] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown): File[/etc/sudoers],File[/home/salt-user],File[/etc/nginx/mime.types],File[/etc/nginx/fastcgi_params] [00:04:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.64, 20.73, 18.06 [00:08:10] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.28, 5.39, 6.56 [00:09:13] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.62, 22.66, 19.62 [00:10:16] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.27, 7.56, 7.59 [00:11:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.62, 22.46, 19.98 [00:11:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.60, 3.11 [00:12:26] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.88, 7.34, 7.51 [00:15:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 34.64, 26.50, 22.03 [00:16:13] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.91, 3.90, 3.34 [00:16:51] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.06, 8.38, 7.88 [00:17:57] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.32, 6.16, 6.70 [00:18:19] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [00:18:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.50, 3.43, 3.25 [00:20:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.79, 3.41 [00:21:35] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.70, 7.24, 7.57 [00:22:16] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (2 critical updates). [00:23:54] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.37, 8.16, 7.87 [00:25:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.97, 5.68, 4.37 [00:25:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.52, 3.53, 3.43 [00:27:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.07, 3.26, 3.33 [00:27:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.38, 7.10, 7.98 [00:28:22] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.60, 6.68, 7.77 [00:29:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.38, 22.99, 23.88 [00:29:19] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.07, 4.56, 4.25 [00:30:08] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.42, 6.92, 7.78 [00:30:43] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [00:31:57] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.48, 7.07, 6.63 [00:33:52] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.58, 7.11, 6.77 [00:34:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.11, 6.72, 7.37 [00:34:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.50, 7.50, 7.79 [00:34:49] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.22, 7.83, 7.80 [00:38:17] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.04, 6.43, 6.55 [00:38:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.08, 6.56, 7.22 [00:38:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.39, 7.53, 7.81 [00:40:12] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.01, 7.80, 7.17 [00:40:48] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.82, 8.44, 8.09 [00:41:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.39, 3.25 [00:42:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.45, 7.78, 7.25 [00:42:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.62, 7.60, 7.82 [00:43:10] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.63, 7.64, 7.88 [00:43:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.59, 3.35 [00:43:56] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.09, 17.27, 19.93 [00:47:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.20, 5.92, 6.71 [00:47:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 3.03, 3.18 [00:47:57] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.56, 7.82, 7.10 [00:50:02] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.45, 7.26, 6.98 [00:50:46] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.54, 8.00, 7.51 [00:51:33] If you're still using XP and are relatively secure and truly are using it to its potential, then out of date root signing certificates in theory should not a problem for you :p [00:52:50] Otherwise you should probably leave it well back in the dust. The only viable holdout of obsolete windows versions these days is 7, which is still vaguely achievable for everyday use with work. [00:52:50] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.68, 7.21, 7.29 [00:53:04] Or 8, but I think that's in support still [00:54:17] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.36, 8.27, 7.39 [00:54:53] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.99, 8.32, 7.68 [01:00:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.22, 7.75, 7.51 [01:01:05] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.35, 7.70, 7.72 [01:02:09] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.44, 6.01, 6.77 [01:04:11] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.09, 7.12, 7.10 [01:06:02] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.83, 6.81, 6.59 [01:06:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.54, 6.62, 6.93 [01:06:28] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.37, 6.61, 6.88 [01:08:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.98, 5.87, 6.28 [01:08:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.92, 6.12, 6.71 [01:08:30] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.33, 6.28, 6.73 [01:08:43] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.20, 6.04, 6.74 [01:09:24] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.39, 7.22, 7.37 [01:12:25] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.56, 7.47, 7.98 [01:12:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.26, 6.57, 6.81 [01:13:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.52, 6.96, 7.26 [01:14:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.84, 5.93, 6.55 [01:22:36] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.14, 7.27, 7.40 [01:22:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.13, 3.47, 3.24 [01:23:47] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.04, 6.99, 7.00 [01:24:04] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JgCEt [01:24:05] [02miraheze/MirahezeMagic] 07Universal-Omega 0337f09ac - Remove unused variable [01:24:47] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.57, 3.29 [01:25:02] miraheze/MirahezeMagic - Universal-Omega the build passed. [01:27:16] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.06, 7.20, 6.77 [01:27:53] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.42, 7.61, 7.30 [01:30:51] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.14, 7.42, 7.55 [01:30:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.37, 3.58, 3.42 [01:31:18] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.48, 7.23, 6.84 [01:32:00] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.87, 7.98, 7.46 [01:32:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.77, 3.50 [01:33:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.30, 6.81, 6.74 [01:34:58] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.45, 7.91, 7.71 [01:34:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.50, 3.43 [01:35:15] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.93, 8.02, 6.71 [01:35:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.61, 6.30, 6.56 [01:35:24] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.91, 8.46, 7.03 [01:35:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.33, 7.24, 6.51 [01:36:06] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.06, 7.68, 7.46 [01:37:00] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.86, 2.96, 3.24 [01:37:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.33, 7.90, 7.75 [01:37:18] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.86, 7.29, 6.61 [01:37:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.46, 7.46, 6.70 [01:39:20] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.97, 6.75, 6.52 [01:39:31] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.39, 7.71, 7.10 [01:39:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.63, 6.42, 6.41 [01:41:07] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.71, 8.08, 7.79 [01:41:33] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.00, 6.68, 6.79 [01:43:08] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.80, 7.25, 7.52 [01:44:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.52, 3.40 [01:48:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.16, 3.22, 3.29 [01:51:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.85, 6.62, 6.34 [01:53:28] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.80, 7.14, 7.11 [01:54:01] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.29, 7.18, 6.59 [01:55:12] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.25, 7.31, 6.86 [01:55:33] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.46, 7.27, 7.17 [01:56:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.40, 6.56, 6.43 [01:56:42] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.91, 6.36, 6.72 [01:57:11] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.93, 6.72, 6.69 [01:57:33] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.75, 6.24, 6.80 [02:01:41] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.83, 6.85, 6.95 [02:03:39] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.30, 5.94, 6.61 [02:17:30] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.91, 7.54, 6.86 [02:19:33] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.48, 7.57, 6.96 [02:19:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.17, 7.00, 6.12 [02:20:45] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.24, 7.48, 6.27 [02:21:03] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.36, 7.42, 6.61 [02:21:47] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 13.71, 9.61, 7.77 [02:21:48] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.59, 7.25, 6.33 [02:22:57] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.81, 7.14, 6.34 [02:23:00] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.17, 7.91, 6.94 [02:23:12] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.05, 6.84, 6.49 [02:23:54] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.34, 7.39, 6.48 [02:25:17] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.54, 6.50, 6.42 [02:26:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.52, 3.27 [02:28:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 3.15, 3.15 [02:30:02] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.45, 7.98, 7.83 [02:31:22] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.05, 7.53, 7.38 [02:31:24] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.99, 6.13, 6.28 [02:32:08] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.49, 8.50, 8.05 [02:34:12] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.20, 7.59, 7.77 [02:35:35] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.34, 7.88, 7.54 [02:35:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.57, 7.61, 6.83 [02:35:48] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.19, 7.78, 6.95 [02:35:49] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.01, 7.86, 7.00 [02:36:11] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.77, 7.75, 7.51 [02:36:21] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.54, 8.15, 7.97 [02:37:42] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.95, 7.87, 7.59 [02:37:50] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.47, 7.65, 6.96 [02:37:52] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.27, 6.95, 6.77 [02:37:54] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.63, 7.04, 6.81 [02:39:43] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.03, 7.91, 7.63 [02:39:56] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.79, 6.65, 6.68 [02:39:59] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.01, 6.70, 6.71 [02:41:45] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.18, 7.60, 7.55 [02:44:04] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.28, 6.83, 6.78 [02:44:05] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.15, 5.99, 6.52 [02:44:32] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.87, 7.29, 7.75 [02:46:10] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.16, 6.29, 6.60 [02:50:42] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.49, 7.29, 7.50 [02:52:29] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.74, 5.68, 6.59 [02:52:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.54, 7.25, 7.45 [02:56:51] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.26, 7.84, 7.62 [02:58:55] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.05, 7.73, 7.62 [02:59:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.64, 6.89, 6.45 [03:00:21] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.23, 6.85, 6.93 [03:00:50] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.58, 6.97, 6.90 [03:00:59] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.58, 7.97, 7.69 [03:01:58] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.70, 6.76, 6.46 [03:02:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.02, 6.90, 6.94 [03:03:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.42, 3.28 [03:06:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.42, 3.00, 3.14 [03:06:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.97, 8.20, 7.12 [03:06:30] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.22, 7.22, 6.61 [03:06:45] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.34, 8.08, 7.39 [03:07:08] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.36, 7.61, 7.21 [03:08:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.58, 8.30, 7.19 [03:08:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.11, 7.89, 7.16 [03:08:35] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.30, 6.40, 6.40 [03:08:49] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.89, 7.37, 7.24 [03:09:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.42, 7.18, 7.09 [03:09:24] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.48, 7.82, 7.89 [03:10:20] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.55, 7.27, 6.95 [03:12:31] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.55, 6.02, 6.55 [03:14:20] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.62, 6.64, 6.78 [03:16:57] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.08, 7.17, 7.13 [03:17:31] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.41, 7.75, 7.74 [03:19:03] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.08, 6.49, 6.89 [03:19:37] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.69, 7.49, 7.64 [03:25:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.09, 6.49, 6.79 [03:30:40] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.69, 6.92, 6.76 [03:32:44] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.40, 6.45, 6.59 [03:36:55] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.15, 8.75, 7.49 [03:37:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.31, 6.95, 6.79 [03:37:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.10, 7.30, 7.03 [03:37:40] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.37, 7.89, 6.55 [03:39:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.96, 7.25, 7.05 [03:39:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.74, 6.78, 6.74 [03:39:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.25, 7.19, 6.46 [03:40:53] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.94, 7.81, 7.39 [03:43:45] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.10, 6.56, 6.36 [03:44:02] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.60, 7.27, 7.05 [03:46:02] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.18, 7.18, 7.04 [03:46:54] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.47, 5.78, 6.57 [03:47:32] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.49, 6.31, 6.71 [03:52:11] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.39, 5.92, 6.58 [03:55:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.35, 3.14 [03:57:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.44, 6.97, 6.74 [03:57:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.10, 3.32, 3.16 [04:01:32] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.87, 6.47, 6.60 [04:06:29] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.22, 6.93, 6.49 [04:08:30] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.96, 6.27, 6.31 [04:27:51] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.86, 7.28, 6.69 [04:31:49] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.06, 7.47, 6.93 [04:33:53] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.73, 6.57, 6.68 [04:44:57] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.04, 6.91, 6.77 [04:46:58] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.29, 7.68, 7.06 [04:48:13] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 14.72, 10.51, 7.60 [04:49:10] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.54, 6.90, 6.24 [04:49:28] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.41, 7.08, 6.14 [04:51:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.70, 7.31, 7.07 [04:51:11] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.78, 7.49, 6.54 [04:51:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.88, 7.21, 6.32 [04:52:59] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.57, 6.50, 6.79 [04:53:13] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.98, 6.48, 6.28 [04:53:32] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.99, 6.60, 6.19 [04:54:16] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.07, 7.79, 7.42 [04:56:56] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.51, 6.73, 6.84 [04:58:58] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.02, 6.31, 6.69 [05:02:18] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.77, 5.91, 6.66 [05:22:14] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 3.61, 3.03 [05:26:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.63, 3.18 [05:27:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.57, 6.58, 5.99 [05:28:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.61, 3.30, 3.12 [05:29:23] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.65, 7.17, 6.28 [05:29:50] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.93, 6.74, 6.29 [05:30:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.74, 7.07, 6.43 [05:31:53] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.85, 6.37, 6.21 [05:34:51] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 187.01 ms [05:35:59] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.59, 7.32, 6.66 [05:36:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.03, 7.17, 5.98 [05:37:02] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 91.47 ms [05:37:21] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.73, 6.98, 6.35 [05:37:38] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.94, 7.70, 7.03 [05:38:03] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.55, 7.75, 6.90 [05:38:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.42, 6.86, 6.03 [05:39:21] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.27, 6.35, 6.20 [05:40:03] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.48, 7.03, 6.73 [05:41:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.73, 6.25, 5.89 [05:44:26] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.58, 6.66, 6.75 [05:45:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.33, 3.19 [05:46:12] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.44, 6.25, 6.56 [05:47:50] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.34, 3.21, 3.15 [05:53:59] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.61, 5.95, 6.52 [06:05:50] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.39, 7.29, 6.55 [06:05:55] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.33, 6.89, 6.36 [06:07:52] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.44, 6.63, 6.41 [06:07:54] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.17, 6.07, 6.12 [06:32:25] you are not secure if you're using XP [06:51:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.23, 2.96 [06:51:53] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.07, 6.46, 5.98 [06:53:42] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.27, 3.18, 2.97 [06:53:54] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.43, 6.43, 6.03 [08:04:24] Not typically, but I will not exclude the very rare madman minority who is able to tweak XP in such a way that it's not the environment you're thinking of security wise. [08:04:45] Though obviously that is rare and didn't apply in the above case. [08:05:34] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.31, 7.56, 6.32 [08:07:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.72, 6.92, 6.23 [08:07:39] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.10, 6.87, 6.22 [08:09:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.61, 6.58, 6.19 [08:09:40] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.83, 6.09, 6.01 [08:11:25] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/Jg0ck [08:11:27] [02miraheze/mw-config] 07Reception123 036721a06 - set $wgDplSettings['functionalRichness'] = 0; temporarily [08:11:28] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [08:11:34] [02mw-config] 07Reception123 opened pull request 03#4121: set $wgDplSettings['functionalRichness'] = 0; temporarily - 13https://git.io/Jg0cR [08:12:11] [02mw-config] 07Universal-Omega reviewed pull request 03#4121 commit - 13https://git.io/Jg0CD [08:12:37] miraheze/mw-config - Reception123 the build passed. [08:13:22] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/Jg08G [08:13:23] [02miraheze/mw-config] 07Reception123 0348c391d - Update Defines.php [08:13:25] [02mw-config] 07Reception123 synchronize pull request 03#4121: set $wgDplSettings['functionalRichness'] = 0; temporarily - 13https://git.io/Jg0cR [08:14:32] miraheze/mw-config - Reception123 the build passed. [08:14:48] [02mw-config] 07Reception123 closed pull request 03#4121: set $wgDplSettings['functionalRichness'] = 0; temporarily - 13https://git.io/Jg0cR [08:14:50] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jg0Rg [08:14:51] [02miraheze/mw-config] 07Reception123 03be1eb32 - set $wgDplSettings['functionalRichness'] = 0; temporarily (#4121) [08:14:53] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [08:14:54] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [08:15:55] miraheze/mw-config - Reception123 the build passed. [08:17:10] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.46, 7.01, 6.42 [08:19:54] PROBLEM - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.bikinibottomdiscord.org' expires in 7 day(s) (Sat 09 Oct 2021 08:14:29 GMT +0000). [08:21:08] Reception123: ^ [08:21:15] Can we check the renew scripts work [08:21:21] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.28, 6.40, 6.34 [08:21:52] RhinosF1: I think it doesn't work because of that mistake I made when reimaging mwtask1 [08:22:19] Ok [08:23:36] !log [@test3] starting deploy of {'config': True} to skip [08:23:37] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [08:23:43] !log [@mw11] starting deploy of {'config': True} to all [08:23:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:23:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:23:56] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 13s [08:24:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:24:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:24:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.57, 6.86, 5.86 [08:25:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.65, 7.11, 6.69 [08:25:58] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.39, 6.87, 6.11 [08:26:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.73, 6.87, 6.00 [08:26:22] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.04, 7.41, 6.63 [08:26:44] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.13, 6.55, 5.87 [08:28:03] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.37, 6.43, 6.03 [08:28:22] PROBLEM - puardian.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'puardian.xyz' expires in 7 day(s) (Sat 09 Oct 2021 08:22:40 GMT +0000). [08:30:24] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.13, 6.55, 6.09 [08:31:48] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.18, 6.35, 6.57 [08:32:28] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.07, 7.38, 6.84 [08:36:30] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.35, 6.75, 6.73 [08:37:04] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.09, 7.81, 6.96 [08:39:05] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.77, 7.08, 6.80 [08:41:04] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.23, 6.54, 6.63 [09:05:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.16, 6.65, 6.31 [09:33:10] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.78, 6.85, 6.33 [09:35:12] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.58, 6.59, 6.28 [09:36:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.60, 7.18, 6.55 [09:38:48] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.68, 6.30, 6.30 [09:44:43] Reception123 please the domain redirect of famepedia.org to en.famepedia.org still doesn't work. It shows the security error stuff, I think you need to install SSL [09:52:36] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.43, 2.98 [09:54:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 3.29, 3.00 [09:57:38] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.20, 7.10, 6.61 [09:58:53] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.67, 7.17, 6.43 [09:59:01] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.91, 6.97, 6.39 [10:00:52] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.56, 6.42, 6.26 [10:00:58] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.00, 6.21, 6.18 [10:01:38] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.65, 6.35, 6.43 [10:19:56] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.41, 6.37, 5.47 [10:21:17] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.23, 6.97, 6.06 [10:21:21] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.08, 7.44, 6.20 [10:21:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.77, 6.32, 5.27 [10:22:01] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.63, 6.60, 5.66 [10:23:16] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.69, 7.22, 6.27 [10:23:22] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.00, 7.10, 6.22 [10:23:35] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.73, 6.27, 5.40 [10:24:08] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.01, 6.46, 5.74 [10:25:32] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.00, 8.23, 6.76 [10:26:20] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.65, 7.12, 6.27 [10:27:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.95, 8.02, 6.82 [10:28:29] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 12.21, 9.19, 7.17 [10:28:29] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.14, 7.18, 6.22 [10:33:03] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.48, 7.37, 6.35 [10:33:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.53, 7.57, 7.07 [10:34:35] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.07, 8.39, 7.00 [10:34:36] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.95, 7.43, 6.15 [10:35:08] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.35, 8.62, 6.95 [10:35:33] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.54, 8.40, 7.44 [10:36:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.97, 7.55, 6.38 [10:38:39] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.71, 8.00, 7.50 [10:38:41] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.24, 7.72, 6.58 [10:39:08] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.19, 7.60, 6.94 [10:40:44] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.70, 7.15, 6.50 [10:41:51] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.66, 7.56, 7.66 [10:42:40] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.25, 7.51, 7.33 [10:43:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.21, 6.61, 6.72 [10:45:33] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.71, 7.90, 7.83 [10:45:50] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.02, 7.62, 7.66 [10:47:49] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.17, 7.36, 7.55 [10:48:43] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.34, 7.01, 7.14 [10:48:55] @Joseph which task was that again? [10:49:52] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.59, 7.52, 7.56 [10:50:42] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.05, 7.43, 7.25 [10:50:43] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.08, 6.65, 6.99 [10:50:45] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.60, 6.20, 6.37 [10:51:23] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.44, 7.46, 6.95 [10:51:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.44, 7.76, 7.67 [10:52:42] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.74, 6.90, 7.09 [10:53:24] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.42, 6.79, 6.78 [10:53:56] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.17, 8.35, 7.87 [10:54:45] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.86, 7.61, 7.32 [10:54:47] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.87, 7.03, 6.66 [10:56:01] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.35, 7.90, 7.77 [10:56:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.81, 7.46, 7.29 [10:56:48] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.07, 6.50, 6.53 [10:57:26] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.55, 6.10, 6.79 [10:58:45] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.71, 7.46, 7.28 [11:00:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.10, 6.63, 7.01 [11:02:42] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.80, 6.06, 6.76 [11:02:46] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.93, 6.38, 6.73 [11:07:18] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.86, 6.88, 6.75 [11:07:59] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.43, 5.98, 6.74 [11:09:15] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.74, 6.21, 6.53 [11:18:49] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.94, 6.35, 6.40 [11:24:50] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.18, 6.22, 6.43 [11:24:51] Reception123 I can't remember the task exactly [11:25:48] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.04, 7.10, 6.64 [11:27:46] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.37, 6.66, 6.56 [11:31:27] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.07, 7.10, 6.43 [11:31:48] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.50, 7.93, 7.06 [11:32:08] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.47, 7.01, 6.57 [11:33:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.81, 7.27, 6.57 [11:35:06] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.02, 8.75, 7.33 [11:35:29] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.07, 7.82, 6.85 [11:36:09] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.93, 7.69, 6.77 [11:36:10] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.09, 7.68, 6.92 [11:37:27] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.81, 7.47, 6.85 [11:37:52] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.70, 7.61, 7.25 [11:38:08] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.48, 6.96, 6.76 [11:39:08] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.67, 7.27, 7.06 [11:39:56] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.39, 8.03, 7.44 [11:40:12] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.66, 6.79, 6.73 [11:40:12] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.07, 6.59, 6.53 [11:41:26] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.75, 6.69, 6.67 [11:41:57] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.12, 7.05, 7.13 [11:42:53] PROBLEM - wiki.beergeeks.co.il - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.beergeeks.co.il' expires in 7 day(s) (Sat 09 Oct 2021 11:38:50 GMT +0000). [11:45:09] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.74, 6.41, 6.73 [11:47:58] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.43, 6.20, 6.80 [11:52:04] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.73, 7.16, 7.09 [11:55:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.41, 6.72, 6.44 [11:56:03] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.76, 7.71, 7.31 [11:57:24] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.26, 7.68, 6.80 [11:59:36] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.86, 7.14, 6.86 [12:00:32] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.92, 8.39, 7.25 [12:01:37] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.75, 7.11, 6.45 [12:02:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.51, 7.88, 7.19 [12:03:27] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.66, 7.93, 7.30 [12:03:39] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.41, 7.46, 6.65 [12:03:40] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.28, 7.56, 7.11 [12:03:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.86, 7.51, 6.66 [12:04:13] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.34, 7.95, 7.77 [12:05:46] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.80, 7.94, 6.96 [12:07:45] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.42, 7.57, 7.29 [12:07:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.06, 7.75, 6.95 [12:09:29] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.20, 6.29, 6.74 [12:09:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.10, 6.82, 6.72 [12:11:48] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.92, 6.10, 6.49 [12:11:58] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.70, 6.40, 6.58 [12:12:36] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.87, 5.67, 6.56 [12:13:43] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.99, 6.13, 6.71 [12:14:19] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.18, 5.49, 6.63 [12:15:45] Reception123: https://phabricator.miraheze.org/T7852? [12:15:46] [url] ⚓ T7852 Domain Redirection | phabricator.miraheze.org [12:20:35] CosmicAlpha: thanks! [12:37:03] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.32, 7.15, 6.37 [12:39:59] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.42, 7.36, 6.74 [12:41:06] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.64, 6.96, 6.53 [12:43:04] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.83, 6.74, 6.51 [12:44:05] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.16, 6.53, 6.57 [12:49:04] well, now that chameleon is working again for me, can someone point the fixedheader.xml at my wiki (like it used to before this mess happened)? [12:52:49] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.32, 6.76, 6.07 [12:54:49] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.88, 6.98, 6.22 [12:56:54] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.05, 6.27, 6.06 [13:04:27] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.56, 6.95, 6.54 [13:06:34] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.82, 7.88, 6.93 [13:07:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.94, 7.22, 6.53 [13:07:57] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.22, 7.33, 6.62 [13:08:33] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: paladox - disabled for migration to new server, last run 13 minutes ago with 0 failures [13:08:35] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.20, 7.23, 6.82 [13:10:36] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.04, 8.18, 7.23 [13:11:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.52, 6.45, 6.39 [13:12:20] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JgKUG [13:12:22] [02miraheze/dns] 07paladox 03d30fb5e - Update ns1 ip [13:12:23] [02dns] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbQXl [13:12:25] [02dns] 07paladox opened pull request 03#220: Update ns1 ip - 13https://git.io/JgKU8 [13:12:31] [02dns] 07paladox closed pull request 03#220: Update ns1 ip - 13https://git.io/JgKU8 [13:12:33] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JgKUi [13:12:34] [02miraheze/dns] 07paladox 0302099f0 - Update ns1 ip (#220) [13:12:36] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-2 [13:12:37] [02dns] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbQXl [13:14:05] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.13, 6.65, 6.59 [13:14:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.54, 7.36, 7.12 [13:16:51] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.54, 7.71, 7.26 [13:19:12] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.72, 8.33, 7.19 [13:19:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JgKc5 [13:19:16] [02miraheze/puppet] 07paladox 032d38059 - base:recursor: Update ns1 ip addresses [13:19:18] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [13:19:19] [02puppet] 07paladox opened pull request 03#1997: base:recursor: Update ns1 ip addresses - 13https://git.io/JgKch [13:19:24] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:19:55] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.32, 8.71, 7.41 [13:20:24] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 14.04, 9.66, 7.38 [13:21:16] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.47, 9.02, 7.24 [13:21:19] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.35, 8.92, 7.16 [13:22:27] PROBLEM - ns1 APT on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:23:33] PROBLEM - ns1 NTP time on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:23:48] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JgKuF [13:23:49] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:23:50] [02miraheze/dns] 07paladox 038f75d7a - Revert "Update ns1 ip (#220)" [13:24:10] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:24:35] PROBLEM - ns1 Disk Space on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:24:49] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:28:46] [02mw-config] 07paladox commented on pull request 03#4119: Cleanup request timeout code - 13https://git.io/JgKPs [14:07:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.53, 3.26 [14:09:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.47, 4.19, 3.54 [14:10:05] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jgiq0 [14:10:07] [02miraheze/mw-config] 07RhinosF1 031b0a1a2 - revert 2aa46048 [14:10:24] @Lake ^ [14:11:12] miraheze/mw-config - RhinosF1 the build passed. [14:11:59] should deploy in ~20 minutes [14:16:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.80, 3.66 [14:28:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.35, 3.10, 3.39 [14:33:07] !log [@test3] starting deploy of {'config': True} to skip [14:33:08] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [14:33:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:33:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.78, 3.25, 3.42 [14:33:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:35:43] !log [@mw11] starting deploy of {'config': True} to all [14:35:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:36:09] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 26s [14:36:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:44:24] RhinosF1: thanks! [14:49:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.35, 2.96, 3.28 [14:59:24] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 54.38.211.199/cpweb, 167.114.2.161/cpweb, 2607:5300:201:3100::1d3/cpweb [15:01:54] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:10:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.68, 3.42 [15:13:24] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.88, 4.08, 3.60 [15:15:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.82, 3.57 [15:20:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 3.22, 3.38 [15:32:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.33, 3.31 [15:37:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.06, 3.27, 3.31 [16:05:37] PROBLEM - cp13 Stunnel Http for mw10 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:05:43] PROBLEM - cp15 Stunnel Http for mw10 on cp15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:06:38] PROBLEM - cp12 Stunnel Http for mw10 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:06:43] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.38.69.175/cpweb, 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [16:07:16] I guess CosmicAlpha's idea is not the cause [16:07:36] PROBLEM - cp3 Stunnel Http for mw10 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:07:57] PROBLEM - cp3 Stunnel Http for mw11 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:08:34] RECOVERY - cp13 Stunnel Http for mw10 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 6.561 second response time [16:08:43] RECOVERY - cp15 Stunnel Http for mw10 on cp15 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 8.316 second response time [16:09:38] RECOVERY - cp12 Stunnel Http for mw10 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 7.725 second response time [16:10:09] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:30] RECOVERY - cp3 Stunnel Http for mw10 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 7.716 second response time [16:13:34] RECOVERY - cp3 Stunnel Http for mw11 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 7.046 second response time [16:15:28] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20018 bytes in 5.233 second response time [16:20:52] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 26 minutes ago with 0 failures [16:21:06] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:21:23] PROBLEM - cp3 Stunnel Http for mw13 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:21:45] PROBLEM - cp15 Stunnel Http for mw13 on cp15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:21:51] RECOVERY - ns1 Disk Space on ns1 is OK: DISK OK - free space: / 5148 MB (71% inode=85%); [16:22:02] PROBLEM - cp13 Stunnel Http for mw13 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:22:08] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -9.468197823e-05 secs [16:22:31] PROBLEM - cp14 Stunnel Http for mw13 on cp14 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:22:36] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:53] PROBLEM - cp12 Stunnel Http for mw13 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:23:16] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.01, 0.00, 0.00 [16:23:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JgDvF [16:23:46] [02miraheze/puppet] 07paladox 034dc07d3 - base:ufw: Simplify rules [16:23:47] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [16:23:49] [02puppet] 07paladox opened pull request 03#1998: base:ufw: Simplify rules - 13https://git.io/JgDfe [16:24:09] [02puppet] 07paladox closed pull request 03#1998: base:ufw: Simplify rules - 13https://git.io/JgDfe [16:24:10] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [16:24:12] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JgDfH [16:24:13] [02miraheze/puppet] 07paladox 034dc07d3 - base:ufw: Simplify rules [16:24:15] [02miraheze/puppet] 07paladox 03c48a86c - Merge pull request #1998 from miraheze/paladox-patch-4 [16:24:16] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [16:26:34] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.260 seconds response time. miraheze.org returns 2001:41d0:800:170b::5,2001:41d0:801:2000::58af,51.38.69.175,54.38.211.199 [16:33:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:37:34] RECOVERY - cp15 Stunnel Http for mw13 on cp15 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 9.833 second response time [16:38:11] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20017 bytes in 9.988 second response time [16:38:14] RECOVERY - cp14 Stunnel Http for mw13 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 9.088 second response time [16:39:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb [16:39:09] PROBLEM - cp12 Stunnel Http for mw12 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:40:03] PROBLEM - cp3 Stunnel Http for mw12 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:40:10] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:41:48] RECOVERY - cp12 Stunnel Http for mw13 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 9.862 second response time [16:41:51] RECOVERY - cp12 Stunnel Http for mw12 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 0.397 second response time [16:42:38] RECOVERY - cp3 Stunnel Http for mw12 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 1.417 second response time [16:42:44] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20017 bytes in 0.886 second response time [16:43:57] PROBLEM - cp15 Stunnel Http for mw13 on cp15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:44:18] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:27] PROBLEM - cp14 Stunnel Http for mw13 on cp14 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:45:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:46:02] RECOVERY - cp13 Stunnel Http for mw13 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 8.330 second response time [16:48:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:51:15] PROBLEM - cp12 Stunnel Http for mw13 on cp12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.305 second response time [16:51:25] PROBLEM - cp13 Stunnel Http for mw13 on cp13 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.023 second response time [16:57:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.222.25.132/cpweb, 167.114.2.161/cpweb [16:57:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:800:170b::5/cpweb, 167.114.2.161/cpweb, 2607:5300:201:3100::1d3/cpweb [17:04:24] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:06:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:08:09] RECOVERY - cp3 Stunnel Http for mw13 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 7.910 second response time [17:08:32] RECOVERY - cp15 Stunnel Http for mw13 on cp15 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 4.472 second response time [17:08:34] RECOVERY - cp12 Stunnel Http for mw13 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15758 bytes in 3.857 second response time [17:08:35] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20018 bytes in 6.263 second response time [17:08:41] RECOVERY - cp13 Stunnel Http for mw13 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 2.861 second response time [17:08:58] RECOVERY - cp14 Stunnel Http for mw13 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 2.601 second response time [17:09:54] PROBLEM - gluster3 Gluster Disk Space on gluster3 is WARNING: DISK WARNING - free space: /mnt/mediawiki-static 237739 MB (10% inode=80%); [17:46:30] paladox: you around? [17:55:53] Reception123: ^ [18:49:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.13, 2.85 [18:51:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.68, 3.58, 3.05 [18:53:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.73, 3.17, 2.96 [18:56:43] RhinosF1: hi [18:56:58] paladox: see -security [18:57:03] ok [19:03:03] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JgQKU [19:03:05] [02miraheze/CreateWiki] 07Universal-Omega 03f45859d - Cache extension-list instead of loading each page load [19:03:06] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [19:03:08] [02CreateWiki] 07Universal-Omega opened pull request 03#242: Cache extension-list instead of loading each page load - 13https://git.io/JgQKs [19:03:14] [02CreateWiki] 07Universal-Omega edited pull request 03#242: Cache extension list instead of loading each page load - 13https://git.io/JgQKs [19:04:18] CosmicAlpha: please get out asap [19:05:05] [02miraheze/mw-config] 07paladox pushed 034 commits to 03revert-automatic-loading [+1/-0/±8] 13https://git.io/JgQXO [19:05:07] [02miraheze/mw-config] 07paladox 039b4b6a3 - Revert "set $wgDplSettings['functionalRichness'] = 0; temporarily (#4121)" [19:05:08] [02miraheze/mw-config] 07paladox 0371e4bca - Revert "Convert LocalWiki.php to use a `switch` (#4118)" [19:05:10] [02miraheze/mw-config] 07paladox 030254cfd - Revert "Merge LocalExtensions.php with Defines.php (#4115)" [19:05:11] [02miraheze/mw-config] 07paladox 03e9bcce0 - Revert "Implement automatic loading of extensions (#3991)" [19:05:13] [02mw-config] 07paladox created branch 03revert-automatic-loading - 13https://git.io/vbvb3 [19:05:25] [02mw-config] 07paladox opened pull request 03#4122: Revert automatic loading - 13https://git.io/JgQ1q [19:06:02] paladox: you've reverted too much [19:06:15] The DPL is needed [19:06:20] But CosmicAlpha has a fix [19:06:35] Let's get that file on the disk cache list as well [19:06:39] miraheze/mw-config - paladox the build passed. [19:07:39] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JgQSA [19:07:41] [02miraheze/CreateWiki] 07Universal-Omega 039f4509e - Update WikiInitialise.php [19:07:42] [02CreateWiki] 07Universal-Omega synchronize pull request 03#242: Cache extension list instead of loading each page load - 13https://git.io/JgQKs [19:08:42] [02puppet] 07RhinosF1 opened pull request 03#1999: Vmtouch: add another file - 13https://git.io/JgQQZ [19:08:51] paladox: ^ [19:10:36] Waiting for CI paladox [19:10:39] [02CreateWiki] 07paladox closed pull request 03#242: Cache extension list instead of loading each page load - 13https://git.io/JgQKs [19:10:41] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JgQFV [19:10:42] [02miraheze/CreateWiki] 07Universal-Omega 039a153b6 - Cache extension list instead of loading each page load (#242) [19:10:44] [02CreateWiki] 07paladox deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [19:10:45] [02miraheze/CreateWiki] 07paladox deleted branch 03Universal-Omega-patch-1 [19:12:24] [02puppet] 07paladox closed pull request 03#1999: Vmtouch: add another file - 13https://git.io/JgQQZ [19:12:26] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JgQxt [19:12:27] [02miraheze/puppet] 07RhinosF1 038eb2ac3 - Vmtouch: add another file [19:12:29] [02miraheze/puppet] 07paladox 03a4ed0af - Merge pull request #1999 from RhinosF1/patch-95 [19:13:20] I triggered bot for update PRs [19:13:48] [02mediawiki] 07dependabot[bot] opened pull request 03#3205: Bump extensions/PageForms from `f3dabf1` to `a7f59e9` - 13https://git.io/JgQjO [19:13:49] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/PageForms-a7f59e9 [+0/-0/±1] 13https://git.io/JgQjs [19:13:51] [02miraheze/mediawiki] 07dependabot[bot] 032329697 - Bump extensions/PageForms from `f3dabf1` to `a7f59e9` [19:13:52] [02mediawiki] 07dependabot[bot] labeled pull request 03#3205: Bump extensions/PageForms from `f3dabf1` to `a7f59e9` - 13https://git.io/JgQjO [19:13:54] [02mediawiki] 07dependabot[bot] labeled pull request 03#3205: Bump extensions/PageForms from `f3dabf1` to `a7f59e9` - 13https://git.io/JgQjO [19:13:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/PageForms-a7f59e9 - 13https://git.io/vbL5b [19:13:57] [02mediawiki] 07dependabot[bot] commented on pull request 03#3187: Bump extensions/PageForms from `f3dabf1` to `865b070` - 13https://git.io/JgQj8 [19:13:58] [02mediawiki] 07dependabot[bot] closed pull request 03#3187: Bump extensions/PageForms from `f3dabf1` to `865b070` - 13https://git.io/JzAGR [19:14:00] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/PageForms-865b070 - 13https://git.io/vbL5b [19:14:01] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/PageForms-865b070 [19:14:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-9545287 [+0/-0/±1] 13https://git.io/JgQj0 [19:14:04] [02miraheze/mediawiki] 07dependabot[bot] 0312f5552 - Bump extensions/UniversalLanguageSelector from `7374540` to `9545287` [19:14:06] [02mediawiki] 07dependabot[bot] opened pull request 03#3206: Bump extensions/UniversalLanguageSelector from `7374540` to `9545287` - 13https://git.io/JgQjz [19:14:07] [02mediawiki] 07dependabot[bot] labeled pull request 03#3206: Bump extensions/UniversalLanguageSelector from `7374540` to `9545287` - 13https://git.io/JgQjz [19:14:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-9545287 - 13https://git.io/vbL5b [19:14:10] [02mediawiki] 07dependabot[bot] labeled pull request 03#3206: Bump extensions/UniversalLanguageSelector from `7374540` to `9545287` - 13https://git.io/JgQjz [19:14:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/Medik-c52dc4d [+0/-0/±1] 13https://git.io/JgQjo [19:14:13] [02miraheze/mediawiki] 07dependabot[bot] 03fd61008 - Bump skins/Medik from `4cd707d` to `c52dc4d` [19:14:15] [02mediawiki] 07dependabot[bot] opened pull request 03#3207: Bump skins/Medik from `4cd707d` to `c52dc4d` - 13https://git.io/JgQjK [19:14:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/Medik-c52dc4d - 13https://git.io/vbL5b [19:14:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#3207: Bump skins/Medik from `4cd707d` to `c52dc4d` - 13https://git.io/JgQjK [19:14:19] [02mediawiki] 07dependabot[bot] labeled pull request 03#3207: Bump skins/Medik from `4cd707d` to `c52dc4d` - 13https://git.io/JgQjK [19:14:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/TabberNeue-3358e2e [+0/-0/±1] 13https://git.io/JgQj9 [19:14:22] [02miraheze/mediawiki] 07dependabot[bot] 03da1b483 - Bump extensions/TabberNeue from `0dc1b34` to `3358e2e` [19:14:24] [02mediawiki] 07dependabot[bot] opened pull request 03#3208: Bump extensions/TabberNeue from `0dc1b34` to `3358e2e` - 13https://git.io/JgQjH [19:14:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/TabberNeue-3358e2e - 13https://git.io/vbL5b [19:14:27] [02mediawiki] 07dependabot[bot] labeled pull request 03#3208: Bump extensions/TabberNeue from `0dc1b34` to `3358e2e` - 13https://git.io/JgQjH [19:14:28] [02mediawiki] 07dependabot[bot] labeled pull request 03#3208: Bump extensions/TabberNeue from `0dc1b34` to `3358e2e` - 13https://git.io/JgQjH [19:14:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/foreground-471eaff [+0/-0/±1] 13https://git.io/Jg7eJ [19:14:31] [02miraheze/mediawiki] 07dependabot[bot] 031e7fce8 - Bump skins/foreground from `d8fe013` to `471eaff` [19:14:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/foreground-471eaff - 13https://git.io/vbL5b [19:14:34] [02mediawiki] 07dependabot[bot] opened pull request 03#3209: Bump skins/foreground from `d8fe013` to `471eaff` - 13https://git.io/Jg7eU [19:14:36] [02mediawiki] 07dependabot[bot] labeled pull request 03#3209: Bump skins/foreground from `d8fe013` to `471eaff` - 13https://git.io/Jg7eU [19:14:37] [02mediawiki] 07dependabot[bot] labeled pull request 03#3209: Bump skins/foreground from `d8fe013` to `471eaff` - 13https://git.io/Jg7eU [19:14:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/ApprovedRevs-2503370 [+0/-0/±1] 13https://git.io/Jg7ek [19:14:40] [02miraheze/mediawiki] 07dependabot[bot] 033b596a4 - Bump extensions/ApprovedRevs from `aaddd36` to `2503370` [19:14:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/ApprovedRevs-2503370 - 13https://git.io/vbL5b [19:14:43] [02mediawiki] 07dependabot[bot] commented on pull request 03#2955: Bump skins/foreground from `d8fe013` to `ab4aa18` - 13https://git.io/Jg7eI [19:14:45] [02mediawiki] 07dependabot[bot] opened pull request 03#3210: Bump extensions/ApprovedRevs from `aaddd36` to `2503370` - 13https://git.io/Jg7eL [19:14:46] [02mediawiki] 07dependabot[bot] closed pull request 03#2955: Bump skins/foreground from `d8fe013` to `ab4aa18` - 13https://git.io/Ju13s [19:14:48] [02mediawiki] 07dependabot[bot] labeled pull request 03#3210: Bump extensions/ApprovedRevs from `aaddd36` to `2503370` - 13https://git.io/Jg7eL [19:14:49] [02mediawiki] 07dependabot[bot] labeled pull request 03#3210: Bump extensions/ApprovedRevs from `aaddd36` to `2503370` - 13https://git.io/Jg7eL [19:14:50] miraheze/CreateWiki - Universal-Omega the build passed. [19:14:51] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/foreground-ab4aa18 [19:14:52] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/foreground-ab4aa18 - 13https://git.io/vbL5b [19:14:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Translate-0f9876e [+0/-0/±1] 13https://git.io/Jg7e6 [19:14:55] [02miraheze/mediawiki] 07dependabot[bot] 039c47f9e - Bump extensions/Translate from `a77087e` to `0f9876e` [19:14:57] [02mediawiki] 07dependabot[bot] opened pull request 03#3211: Bump extensions/Translate from `a77087e` to `0f9876e` - 13https://git.io/Jg7eX [19:14:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Translate-0f9876e - 13https://git.io/vbL5b [19:15:00] [02mediawiki] 07dependabot[bot] labeled pull request 03#3211: Bump extensions/Translate from `a77087e` to `0f9876e` - 13https://git.io/Jg7eX [19:15:01] [02mediawiki] 07dependabot[bot] labeled pull request 03#3211: Bump extensions/Translate from `a77087e` to `0f9876e` - 13https://git.io/Jg7eX [19:15:03] [02mediawiki] 07dependabot[bot] commented on pull request 03#3172: Bump extensions/Translate from `a77087e` to `0735b03` - 13https://git.io/Jg7eH [19:15:04] [02mediawiki] 07dependabot[bot] closed pull request 03#3172: Bump extensions/Translate from `a77087e` to `0735b03` - 13https://git.io/JzAsR [19:15:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/ManageWiki-9d18d80 [+0/-0/±1] 13https://git.io/Jg7e5 [19:15:07] [02miraheze/mediawiki] 07dependabot[bot] 03d38412a - Bump extensions/ManageWiki from `c231b88` to `9d18d80` [19:15:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/ManageWiki-9d18d80 - 13https://git.io/vbL5b [19:15:10] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Translate-0735b03 - 13https://git.io/vbL5b [19:15:12] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Translate-0735b03 [19:15:13] [02mediawiki] 07dependabot[bot] opened pull request 03#3212: Bump extensions/ManageWiki from `c231b88` to `9d18d80` - 13https://git.io/Jg7eF [19:15:15] [02mediawiki] 07dependabot[bot] labeled pull request 03#3212: Bump extensions/ManageWiki from `c231b88` to `9d18d80` - 13https://git.io/Jg7eF [19:15:16] [02mediawiki] 07dependabot[bot] labeled pull request 03#3212: Bump extensions/ManageWiki from `c231b88` to `9d18d80` - 13https://git.io/Jg7eF [19:15:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/MirahezeMagic-37f09ac [+0/-0/±1] 13https://git.io/Jg7vv [19:15:19] [02miraheze/mediawiki] 07dependabot[bot] 0361becb4 - Bump extensions/MirahezeMagic from `1fccf46` to `37f09ac` [19:15:21] [02mediawiki] 07dependabot[bot] opened pull request 03#3213: Bump extensions/MirahezeMagic from `1fccf46` to `37f09ac` - 13https://git.io/Jg7vU [19:15:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/MirahezeMagic-37f09ac - 13https://git.io/vbL5b [19:15:24] [02mediawiki] 07dependabot[bot] labeled pull request 03#3213: Bump extensions/MirahezeMagic from `1fccf46` to `37f09ac` - 13https://git.io/Jg7vU [19:15:25] [02mediawiki] 07dependabot[bot] labeled pull request 03#3213: Bump extensions/MirahezeMagic from `1fccf46` to `37f09ac` - 13https://git.io/Jg7vU [19:15:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/Cosmos-318d3dd [+0/-0/±1] 13https://git.io/Jg7vC [19:15:28] [02miraheze/mediawiki] 07dependabot[bot] 03b952204 - Bump skins/Cosmos from `9805ebb` to `318d3dd` [19:15:30] [02mediawiki] 07dependabot[bot] opened pull request 03#3214: Bump skins/Cosmos from `9805ebb` to `318d3dd` - 13https://git.io/Jg7vl [19:15:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/Cosmos-318d3dd - 13https://git.io/vbL5b [19:15:33] [02mediawiki] 07dependabot[bot] labeled pull request 03#3214: Bump skins/Cosmos from `9805ebb` to `318d3dd` - 13https://git.io/Jg7vl [19:15:34] [02mediawiki] 07dependabot[bot] labeled pull request 03#3214: Bump skins/Cosmos from `9805ebb` to `318d3dd` - 13https://git.io/Jg7vl [19:15:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/DiscordNotifications-faacede [+0/-0/±1] 13https://git.io/Jg7vz [19:15:37] [02miraheze/mediawiki] 07dependabot[bot] 03e4617f3 - Bump extensions/DiscordNotifications from `b19d11b` to `faacede` [19:15:39] [02mediawiki] 07dependabot[bot] opened pull request 03#3215: Bump extensions/DiscordNotifications from `b19d11b` to `faacede` - 13https://git.io/Jg7vg [19:15:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/DiscordNotifications-faacede - 13https://git.io/vbL5b [19:15:42] [02mediawiki] 07dependabot[bot] labeled pull request 03#3215: Bump extensions/DiscordNotifications from `b19d11b` to `faacede` - 13https://git.io/Jg7vg [19:15:43] [02mediawiki] 07dependabot[bot] labeled pull request 03#3215: Bump extensions/DiscordNotifications from `b19d11b` to `faacede` - 13https://git.io/Jg7vg [19:15:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/WikiForum-2ccdb80 [+0/-0/±1] 13https://git.io/Jg7v1 [19:15:46] [02miraheze/mediawiki] 07dependabot[bot] 0370f7e3c - Bump extensions/WikiForum from `ab21d72` to `2ccdb80` [19:15:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/WikiForum-2ccdb80 - 13https://git.io/vbL5b [19:15:49] [02mediawiki] 07dependabot[bot] opened pull request 03#3216: Bump extensions/WikiForum from `ab21d72` to `2ccdb80` - 13https://git.io/Jg7v9 [19:15:51] [02mediawiki] 07dependabot[bot] labeled pull request 03#3216: Bump extensions/WikiForum from `ab21d72` to `2ccdb80` - 13https://git.io/Jg7v9 [19:15:52] [02mediawiki] 07dependabot[bot] labeled pull request 03#3216: Bump extensions/WikiForum from `ab21d72` to `2ccdb80` - 13https://git.io/Jg7v9 [19:15:54] [02mediawiki] 07dependabot[bot] commented on pull request 03#3181: Bump extensions/WikiForum from `ab21d72` to `6b8c16d` - 13https://git.io/Jg7vN [19:15:56] [02mediawiki] 07dependabot[bot] closed pull request 03#3181: Bump extensions/WikiForum from `ab21d72` to `6b8c16d` - 13https://git.io/JzAGv [19:15:57] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/WikiForum-6b8c16d - 13https://git.io/vbL5b [19:15:58] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/WikiForum-6b8c16d [19:16:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/CreateWiki-9a153b6 [+0/-0/±1] 13https://git.io/Jg7fU [19:16:01] [02miraheze/mediawiki] 07dependabot[bot] 03d4a2279 - Bump extensions/CreateWiki from `cf39ee6` to `9a153b6` [19:16:03] [02mediawiki] 07dependabot[bot] opened pull request 03#3217: Bump extensions/CreateWiki from `cf39ee6` to `9a153b6` - 13https://git.io/Jg7fT [19:16:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/CreateWiki-9a153b6 - 13https://git.io/vbL5b [19:16:06] [02mediawiki] 07dependabot[bot] labeled pull request 03#3217: Bump extensions/CreateWiki from `cf39ee6` to `9a153b6` - 13https://git.io/Jg7fT [19:16:07] [02mediawiki] 07dependabot[bot] labeled pull request 03#3217: Bump extensions/CreateWiki from `cf39ee6` to `9a153b6` - 13https://git.io/Jg7fT [19:16:09] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/Jg7ft [19:16:10] [02miraheze/mediawiki] 07paladox 035242288 - Update CreateWiki [19:16:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/GlobalNewFiles-6c75f13 [+0/-0/±1] 13https://git.io/Jg7fs [19:16:13] [02miraheze/mediawiki] 07dependabot[bot] 03e3b9a7e - Bump extensions/GlobalNewFiles from `5a0fd36` to `6c75f13` [19:16:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/GlobalNewFiles-6c75f13 - 13https://git.io/vbL5b [19:16:16] [02mediawiki] 07dependabot[bot] opened pull request 03#3218: Bump extensions/GlobalNewFiles from `5a0fd36` to `6c75f13` - 13https://git.io/Jg7fZ [19:16:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#3218: Bump extensions/GlobalNewFiles from `5a0fd36` to `6c75f13` - 13https://git.io/Jg7fZ [19:16:19] [02mediawiki] 07dependabot[bot] labeled pull request 03#3218: Bump extensions/GlobalNewFiles from `5a0fd36` to `6c75f13` - 13https://git.io/Jg7fZ [19:16:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/liberty-3687268 [+0/-0/±1] 13https://git.io/Jg7fQ [19:16:22] [02miraheze/mediawiki] 07dependabot[bot] 03c121f4a - Bump skins/liberty from `4d614e9` to `3687268` [19:16:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/liberty-3687268 - 13https://git.io/vbL5b [19:16:25] [02mediawiki] 07dependabot[bot] opened pull request 03#3219: Bump skins/liberty from `4d614e9` to `3687268` - 13https://git.io/Jg7f5 [19:16:27] [02mediawiki] 07dependabot[bot] labeled pull request 03#3219: Bump skins/liberty from `4d614e9` to `3687268` - 13https://git.io/Jg7f5 [19:16:28] [02mediawiki] 07dependabot[bot] labeled pull request 03#3219: Bump skins/liberty from `4d614e9` to `3687268` - 13https://git.io/Jg7f5 [19:16:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/Evelution-55362e4 [+0/-0/±1] 13https://git.io/Jg7fj [19:16:31] [02miraheze/mediawiki] 07dependabot[bot] 036ba8101 - Bump skins/Evelution from `53c7721` to `55362e4` [19:16:33] [02mediawiki] 07dependabot[bot] opened pull request 03#3220: Bump skins/Evelution from `53c7721` to `55362e4` - 13https://git.io/Jg7Je [19:16:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/Evelution-55362e4 - 13https://git.io/vbL5b [19:16:36] [02mediawiki] 07dependabot[bot] labeled pull request 03#3220: Bump skins/Evelution from `53c7721` to `55362e4` - 13https://git.io/Jg7Je [19:16:37] [02mediawiki] 07dependabot[bot] labeled pull request 03#3220: Bump skins/Evelution from `53c7721` to `55362e4` - 13https://git.io/Jg7Je [19:16:39] [02mediawiki] 07RhinosF1 commented on pull request 03#3217: Bump extensions/CreateWiki from `cf39ee6` to `9a153b6` - 13https://git.io/Jg7J3 [19:16:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/vendor-1027aa7 [+0/-0/±1] 13https://git.io/Jg7J8 [19:16:42] [02miraheze/mediawiki] 07dependabot[bot] 036d27877 - Bump vendor from `8740687` to `1027aa7` [19:16:43] [02mediawiki] 07dependabot[bot] opened pull request 03#3221: Bump vendor from `8740687` to `1027aa7` - 13https://git.io/Jg7JB [19:16:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/vendor-1027aa7 - 13https://git.io/vbL5b [19:16:46] [02mediawiki] 07dependabot[bot] labeled pull request 03#3221: Bump vendor from `8740687` to `1027aa7` - 13https://git.io/Jg7JB [19:16:48] [02mediawiki] 07dependabot[bot] labeled pull request 03#3221: Bump vendor from `8740687` to `1027aa7` - 13https://git.io/Jg7JB [19:16:49] [02mediawiki] 07RhinosF1 closed pull request 03#3217: Bump extensions/CreateWiki from `cf39ee6` to `9a153b6` - 13https://git.io/Jg7fT [19:16:51] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/CreateWiki-9a153b6 - 13https://git.io/vbL5b [19:16:52] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_36 [+0/-0/±0] 13https://git.io/Jg7Ju [19:16:54] [02miraheze/mediawiki] 07dependabot[bot] 03cb93136 - Bump extensions/CreateWiki from `cf39ee6` to `9a153b6` (#3217) [19:16:55] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/CreateWiki-9a153b6 [19:16:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Cargo-14c497b [+0/-0/±1] 13https://git.io/Jg7JV [19:16:58] [02miraheze/mediawiki] 07dependabot[bot] 03862e2a8 - Bump extensions/Cargo from `19b7b42` to `14c497b` [19:17:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Cargo-14c497b - 13https://git.io/vbL5b [19:17:01] [02mediawiki] 07dependabot[bot] opened pull request 03#3222: Bump extensions/Cargo from `19b7b42` to `14c497b` - 13https://git.io/Jg7Jw [19:17:03] [02mediawiki] 07dependabot[bot] labeled pull request 03#3222: Bump extensions/Cargo from `19b7b42` to `14c497b` - 13https://git.io/Jg7Jw [19:17:04] [02mediawiki] 07dependabot[bot] labeled pull request 03#3222: Bump extensions/Cargo from `19b7b42` to `14c497b` - 13https://git.io/Jg7Jw [19:17:06] [02mediawiki] 07dependabot[bot] commented on pull request 03#3178: Bump extensions/Cargo from `19b7b42` to `f3a5036` - 13https://git.io/Jg7Ji [19:17:07] [02mediawiki] 07dependabot[bot] closed pull request 03#3178: Bump extensions/Cargo from `19b7b42` to `f3a5036` - 13https://git.io/JzAs7 [19:17:09] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Cargo-f3a5036 - 13https://git.io/vbL5b [19:17:10] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Cargo-f3a5036 [19:17:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/PageSchemas-a698ba2 [+0/-0/±1] 13https://git.io/Jg7JS [19:17:13] [02miraheze/mediawiki] 07dependabot[bot] 03e6a42af - Bump extensions/PageSchemas from `5e10d6a` to `a698ba2` [19:17:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/PageSchemas-a698ba2 - 13https://git.io/vbL5b [19:17:16] [02mediawiki] 07dependabot[bot] opened pull request 03#3223: Bump extensions/PageSchemas from `5e10d6a` to `a698ba2` - 13https://git.io/Jg7JH [19:17:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#3223: Bump extensions/PageSchemas from `5e10d6a` to `a698ba2` - 13https://git.io/Jg7JH [19:17:19] [02mediawiki] 07dependabot[bot] labeled pull request 03#3223: Bump extensions/PageSchemas from `5e10d6a` to `a698ba2` - 13https://git.io/Jg7JH [19:18:45] miraheze/CreateWiki - paladox the build passed. [19:21:01] !log [paladox@mw11] starting deploy of {'world': True, 'l10n': True, 'gitinfo': True} to all [19:21:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:23] paladox: don't forget test3 [19:22:30] It's a separate system [19:24:51] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:25:01] Okay puppet fail [19:25:03] That's fun [19:27:30] !log [paladox@test3] starting deploy of {'world': True, 'gitinfo': True} to skip [19:27:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:28:13] !log [paladox@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - SUCCESS in 42s [19:28:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:30:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.70, 7.53, 8.00 [19:32:04] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.01, 8.07, 8.15 [19:35:22] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jg7DP [19:35:24] [02miraheze/CreateWiki] 07Universal-Omega 039e683bd - Add support to regenerate extension list cache if a file is changed [19:35:25] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [19:35:27] [02CreateWiki] 07Universal-Omega opened pull request 03#243: Add support to regenerate extension list cache if a file is changed - 13https://git.io/Jg7DQ [19:36:59] paladox: how far is the deploy? [19:37:10] it's deploying the i18n [19:37:30] Cool [19:38:03] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:38:19] CosmicAlpha: id have expected impact if it's onto l10n [19:38:25] !log [paladox@mw11] finished deploy of {'world': True, 'l10n': True, 'gitinfo': True} to all - SUCCESS in 1043s [19:38:32] paladox: can you see the cache files? [19:38:33] it's finished now [19:38:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:38:42] I see the log [19:38:48] https://www.irccloud.com/pastebin/k98xL90V/ [19:39:36] Why hasn't load dropped then [19:40:47] have no idea [19:41:09] JohnLewis: could something else be involved [19:41:37] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 18.84, 20.70, 19.81 [19:41:52] miraheze/CreateWiki - Universal-Omega the build passed. [19:41:53] Oh [19:41:56] Potentially, as always - these things can be multi cause [19:42:00] i guess it's because it reads each extension [19:42:02] constantly [19:42:11] https://www.irccloud.com/pastebin/8rN2Xum5/ [19:42:58] I think I know how to fix that then. [19:43:37] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.51, 19.08, 19.34 [19:54:08] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/Jg580 [19:54:10] [02miraheze/CreateWiki] 07Universal-Omega 0386c9d6f - Don't load all extension data everytime [19:54:12] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vpJTL [19:54:13] [02CreateWiki] 07Universal-Omega opened pull request 03#244: Don't load all extension data everytime - 13https://git.io/Jg58r [19:54:25] paladox: ^ [19:55:55] that's much faster [19:56:25] Let's roll then [19:56:30] paladox: you merging? [19:56:40] yes will in a sec [19:57:19] [02CreateWiki] 07paladox closed pull request 03#244: Don't load all extension data everytime - 13https://git.io/Jg58r [19:57:20] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jg5zv [19:57:22] [02miraheze/CreateWiki] 07Universal-Omega 034563a8a - Don't load all extension data everytime (#244) [19:57:23] [02CreateWiki] 07paladox deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vpJTL [19:57:25] [02miraheze/CreateWiki] 07paladox deleted branch 03Universal-Omega-patch-3 [19:58:22] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_36 [+0/-0/±1] 13https://git.io/Jg523 [19:58:24] [02miraheze/mediawiki] 07paladox 036a81bac - Update CreateWiki [19:58:25] [02miraheze/mediawiki] 07paladox 0361f51be - Merge branch 'REL1_36' of github.com:miraheze/mediawiki into REL1_36 [19:58:49] !log [paladox@mw11] starting deploy of {'world': True, 'gitinfo': True} to all [19:58:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:22] hmm [19:59:30] What hmm [19:59:34] CosmicAlpha: visualeditor doesn't work [19:59:56] That was being looked at I think [20:00:01] Coco reported it [20:00:37] Related to this or something else, paladox? If this I'm unsure why only VE wouldn't. [20:01:10] miraheze/CreateWiki - Universal-Omega the build passed. [20:01:20] !log [paladox@mw11] finished deploy of {'world': True, 'gitinfo': True} to all - SUCCESS in 151s [20:01:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:45] loads gone down [20:01:57] but [20:02:07] i think we've improved performance from even before the deploy [20:02:16] (the change to switch to automatic loading) [20:02:38] paladox: don't forget to clear the old cache file if it wasn't done. Extensions aren't loaded at all now. (At least on meta) [20:02:44] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.29, 6.66, 7.85 [20:02:54] paladox: looks better [20:02:56] CosmicAlpha: what do you mean clear cache? [20:03:08] "we've improved performance" -> "extensions aren't loaded" I wonder why ;) [20:03:09] oh you mean extension-list? [20:03:22] paladox: rm on the extension-list.json. [20:03:25] ah [20:03:28] paladox: the new json file [20:03:40] JohnLewis: well turning them off does help ye [20:04:02] miraheze/CreateWiki - paladox the build passed. [20:04:29] !log all hosts: rm /srv/mediawiki/cache/extension-list.json [20:04:37] Works now! [20:04:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:31] I'm seeing alarms again [20:05:56] mw10 isn't happy [20:06:25] Nor 11 [20:06:40] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.67, 7.15, 7.74 [20:07:18] paladox: how we looking [20:07:34] it's all done though looks like the load is spiking again [20:07:43] Yeah [20:07:50] That still didn't hold up [20:07:50] but cpu seems down [20:08:37] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.81, 6.58, 7.45 [20:08:44] Let's give it 15 minutes [20:08:52] And see if it settles [20:09:16] paladox: can you sync test3 too [20:09:23] ok [20:09:23] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.09, 6.99, 7.96 [20:10:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.86, 6.66, 7.94 [20:10:36] !log [paladox@test3] starting deploy of {'world': True, 'gitinfo': True} to skip [20:10:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:44] .in 10mins check icinga [20:10:44] RhinosF1: Okay, will remind at 2021-10-01 - 21:20:44BST [20:10:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.06, 6.46, 7.96 [20:11:00] !log [paladox@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - SUCCESS in 24s [20:11:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:12:28] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 7.60, 7.25, 8.02 [20:12:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 7.69, 7.21, 8.08 [20:12:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.51, 6.77, 7.95 [20:13:15] Criticals are not good [20:14:25] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.11, 5.38, 6.68 [20:14:29] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.47, 7.08, 7.88 [20:14:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.85, 6.75, 7.81 [20:18:11] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.65, 6.70, 7.95 [20:18:56] paladox, CosmicAlpha: we seem ok now [20:19:17] ok [20:19:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.09, 5.64, 6.71 [20:20:45] RhinosF1: check icinga [20:22:22] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.31, 8.41, 8.28 [20:23:01] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.08, 7.12, 7.35 [20:23:16] Seriously [20:23:30] We're better [20:24:35] RhinosF1, paladox: Apologies for the issues the patch caused. I did test it pretty significantly, but I guess it still had some issues. [20:24:46] ok [20:24:50] Np [20:24:55] We found the issue [20:25:00] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.51, 6.60, 7.15 [20:25:08] We just need a way to prevent it in future [20:25:43] When making big/fundamental changes, I always conducted performance analyses to show impact [20:26:19] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.19, 6.63, 6.96 [20:26:23] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.12, 7.62, 7.98 [20:26:40] We've had many issues with ManageWiki + CreateWiki 'improvements' before [20:26:44] we need to put a focus on performance tbh. We need to come up with a plan to lower the load. That means hardware and also maybe rewriting our software. [20:27:02] From causing outages to private wikis public [20:27:13] paladox: definately [20:27:39] SSDs would be something but there's a cost [20:27:44] and seeing as our contracts with ovh for the 2 ovh servers is up we should definitely come up with a plan beforehand [20:27:50] for new hardware etc [20:28:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.91, 6.54, 6.91 [20:28:17] We're ok in normal issues [20:28:18] *in january [20:28:24] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.00, 7.85, 8.02 [20:28:31] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.97, 5.65, 6.58 [20:28:37] RhinosF1: that's not relevant to addressing how to monitor and prevent performance impact though? [20:29:07] JohnLewis: disk usage would have had an impact in this incident [20:29:27] I meant your first comment [20:29:45] re private/public wikis and the quotation around 'improvements' [20:29:49] My point is performance hasn't been our only issue [20:30:15] improvements made have caused major issues a few times [20:30:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.20, 7.55, 7.23 [20:30:30] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.15, 7.15, 7.74 [20:30:34] But we are addressing performance here, which is what my suggestion was around [20:31:40] And I'm making a more general comment than just performance [20:32:12] And it was timed in such a way it came across a rude to me. The general problem is seemingly being solved by CI [20:32:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.04, 6.88, 7.01 [20:32:39] CI will have a benefit [20:32:44] The use of quotation around 'improvements' and then directly commenting an incident caused by a change I made which was a performance improvement, immediately after I made a positive suggestion [20:33:46] Performance is something I admit I should've considered more with such a change. Especially considering I should've realised the impact on performance it could have if done incorrectly, which I did it'd seem. But yes, the whole point of the CI work I've been doing is to prevent something (non-performance related, as CI doesn't give performance stats) in the future. [20:34:30] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.96, 7.77, 7.30 [20:34:41] CosmicAlpha: they are the performance jobs upstream [20:34:51] But they're not great in all honesty [20:35:12] They're not voting for a reason [20:35:19] Either way, a reliance on CI to collect data shouldn't be the approach taken [20:35:22] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.59, 7.35, 7.15 [20:35:25] I've had them fail for no reason before [20:35:37] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.85, 6.92, 6.61 [20:35:49] Decisions need to be more data driven, so changes to performance etc. should be manually collected via data during development in both a loaded and unloaded system [20:35:52] JohnLewis: it shouldn't be our only tool but done properly it's a useful one [20:36:03] Yes [20:36:06] I know [20:36:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.84, 6.69, 6.72 [20:36:42] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.60, 7.58, 7.29 [20:36:52] I'd like for us to have a better environment as a beta but I'm not sure how we go about it yet [20:37:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.68, 7.78, 7.73 [20:37:05] Rather than just cherry picking on test3 [20:37:14] Which is semi-production [20:37:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.37, 7.26, 7.17 [20:37:40] test3 shouldn't be 'semi-production' [20:37:45] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.45, 6.58, 6.51 [20:37:45] test4 isn't for example [20:38:00] JohnLewis: it's still part of central auth etc [20:38:19] That can be detached easily [20:38:31] It's something I'd like to do [20:39:09] The idea behind test servers is they are only soft-assigned roles [20:39:18] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.61, 7.44, 7.62 [20:39:32] I'd like us to have a bit of a train for core + extensions [20:39:51] If we needed it, we could remove test3 as a mediawiki server if it was necessary for another purpose for which new provisions couldn't be made for [20:40:21] If we could replace test3wiki with a mini beta cluster [20:40:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.74, 6.73, 6.72 [20:41:20] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.30, 7.54, 7.62 [20:41:28] On limited resources we hold, it's unlikely to meet the threshold to be approved [20:41:46] JohnLewis: yeah, I have a very ideal plan in my head [20:41:54] But resources are an issue [20:42:09] I can guarntee we don't have enough [20:43:02] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.86, 6.31, 6.77 [20:43:20] So I still need to think about how we can make the best of limited resources [20:43:26] Indeed, as we'd be duplicating the whole infra in essence given at least 75% is dedicated to running MediaWiki as a cluster as a whole [20:43:29] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.61, 6.37, 7.17 [20:48:16] JohnLewis: a beta cluster would only really need 2 mw* servers [20:48:27] And in my head we'd drop test3 [20:48:27] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.91, 6.47, 6.79 [20:48:40] Because we're handling far less traffic [20:48:59] It would need a database server I assume [20:49:09] tbh we shouldn't be focusing on more test servers rather on fixing performance which should be the immediate focus. [20:49:29] we are big enough where performance matters now [20:49:36] paladox: we're talking about ways we can test changes too better [20:49:56] But yes production needs something to handle spikes [20:50:05] the longer we leave performance at a terrible state, we push users off/or don't grow [20:50:13] As JohnLewis has said before in normal conditions we're not that bad [20:50:23] and also i would love a page to load at the same time rather then fluctuate. [20:50:53] this isn't about what happened, this is about in general. [20:50:56] paladox: do you mean personal JS? [20:51:04] No i mean loading a wiki [20:51:20] compare it performance wise to wikimedia and you'll see stark differences [20:51:29] Of course [20:52:04] the solution is a long term one but i feel we should try and start resolving it. That be hardware & a overhaul of software. [20:54:13] They have a rack with as many mw servers as us [20:54:28] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.11, 7.37, 7.26 [20:54:46] And more staff working on performance than we have volunteers [20:55:34] We have 5 SRE [20:55:53] They have 4 staff members in just the perf team [20:56:23] Hardware overhaul will likely happen next server renewal [20:56:35] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.39, 6.83, 7.09 [20:56:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.34, 3.14 [20:57:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.07, 6.98, 6.84 [20:57:43] We might need some interim solutions [20:58:00] Not sure what yet but something [20:59:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.60, 2.69, 2.93 [20:59:13] interim would be software wise [20:59:21] It's a two-fold problem [20:59:25] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.42, 6.44, 6.65 [20:59:31] Yes that's a given [21:00:14] There's stuff like pool counter which might limit peaks [21:01:23] But in general, we need to dig far deeper into actual causes [21:02:18] Might be worth discussing a re-prioritisation of efforts then to focus more on debugging and investigating this specific problem [21:03:15] We need to start with ways to debug the issues [21:03:58] Which would be part of refocusing onto it, as if more time is spent on trying to figure that out, solutions/paths will come clear [21:05:07] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.12, 7.34, 6.77 [21:05:12] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.22, 6.27, 6.71 [21:07:14] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.11, 6.86, 6.66 [21:09:10] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.33, 6.38, 6.51 [21:10:25] PROBLEM - cp15 Current Load on cp15 is CRITICAL: CRITICAL - load average: 4.39, 2.20, 1.15 [21:12:22] RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.87, 1.61, 1.06 [21:30:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.04, 2.87 [21:32:18] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 2.82, 2.80 [21:59:20] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.07, 7.50, 6.60 [22:01:19] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.71, 6.99, 6.53 [22:02:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.31, 6.64, 5.98 [22:03:19] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.45, 6.49, 6.41 [22:05:54] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.41, 7.66, 6.74 [22:05:57] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.03, 7.72, 6.42 [22:06:59] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.09, 5.96, 5.93 [22:07:26] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.58, 7.31, 6.77 [22:07:53] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.26, 7.04, 6.62 [22:07:54] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.92, 7.01, 6.31 [22:08:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.25, 6.95, 6.46 [22:08:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.39, 7.14, 6.56 [22:09:54] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.28, 6.34, 6.15 [22:10:17] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.56, 7.78, 6.79 [22:12:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.88, 7.34, 6.73 [22:12:33] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.98, 6.53, 6.50 [22:14:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.35, 6.61, 6.54 [22:15:46] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.60, 6.41, 6.57 [22:18:11] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.62, 7.32, 6.84 [22:20:12] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.07, 7.25, 6.85 [22:21:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.77, 6.66, 6.08 [22:21:43] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.19, 7.85, 7.16 [22:23:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.37, 6.57, 6.12 [22:24:30] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.09, 6.94, 6.44 [22:25:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.00, 7.59, 7.22 [22:26:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.88, 7.52, 7.24 [22:26:29] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.09, 6.76, 6.42 [22:28:26] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.16, 6.14, 6.23 [22:29:44] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.46, 5.80, 6.56 [22:32:09] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.32, 6.10, 6.70 [22:36:08] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.73, 7.30, 7.01 [22:36:33] !log [paladox@test3] starting deploy of {'world': True, 'gitinfo': True} to skip [22:36:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:36:48] !log [paladox@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - SUCCESS in 14s [22:36:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:38:09] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.66, 6.47, 6.73 [22:39:37] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.07, 6.60, 6.14 [22:40:11] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JgA7N [22:40:13] [02miraheze/mediawiki] 07paladox 031208087 - Update CentralAuth [22:40:38] !log [paladox@mw11] starting deploy of {'world': True, 'gitinfo': True} to all [22:40:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:41:34] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.60, 6.24, 6.07 [22:42:05] !log [paladox@mw11] finished deploy of {'world': True, 'gitinfo': True} to all - SUCCESS in 87s [22:42:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:08:15] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.80, 7.12, 6.70 [23:09:24] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.64, 7.55, 6.62 [23:11:24] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.56, 6.71, 6.42 [23:12:09] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.78, 6.50, 6.56 [23:31:19] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.50, 7.58, 6.42 [23:33:18] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.10, 7.09, 6.39 [23:35:26] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.61, 6.80, 6.39 [23:35:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.65, 6.39, 5.82 [23:36:50] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.35, 7.21, 6.54 [23:37:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.46, 7.04, 6.47 [23:39:57] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.97, 6.47, 6.03 [23:42:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.22, 7.23, 6.76 [23:45:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.91, 7.76, 6.98 [23:45:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4121-Reception123-patch-1 [+0/-0/±1] 13https://git.io/JgpAy [23:45:24] [02miraheze/mw-config] 07Universal-Omega 03bd14483 - Revert "set $wgDplSettings['functionalRichness'] = 0; temporarily (#4121)" [23:45:26] [02mw-config] 07Universal-Omega created branch 03revert-4121-Reception123-patch-1 - 13https://git.io/vbvb3 [23:45:27] [02mw-config] 07Universal-Omega opened pull request 03#4123: Revert "set $wgDplSettings['functionalRichness'] = 0; temporarily" - 13https://git.io/JgpAN [23:46:24] miraheze/mw-config - Universal-Omega the build passed. [23:47:12] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.47, 6.93, 6.34 [23:47:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.24, 7.03, 6.80 [23:49:07] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.38, 7.62, 7.38 [23:49:17] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.99, 6.79, 6.75 [23:51:14] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.71, 8.55, 7.11 [23:52:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.60, 7.08, 6.27 [23:52:45] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.64, 6.39, 6.70 [23:53:12] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.53, 7.84, 7.03 [23:53:59] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.56, 6.73, 6.24 [23:57:06] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.46, 6.53, 6.68 [23:57:17] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.90, 5.90, 6.74