[00:01:31] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:06:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.03, 3.52, 3.19 [00:09:03] RECOVERY - mabi.myomi.net - reverse DNS on sslhost is OK: SSL OK - mabi.myomi.net reverse DNS resolves to cp21.miraheze.org - CNAME OK [00:09:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.47, 3.03 [00:09:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.53, 1.53, 1.99 [00:11:25] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.31, 2.98, 2.90 [00:11:42] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.41, 1.71, 1.27 [00:12:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 4.00, 3.57 [00:13:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.76, 9.55, 10.13 [00:13:41] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.81, 1.33, 1.18 [00:13:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.34, 4.32, 3.74 [00:17:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.91, 10.51, 10.35 [00:19:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.66, 11.45, 10.72 [00:19:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.59, 3.66, 3.63 [00:19:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.43, 1.90, 1.89 [00:21:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.91, 11.84, 10.97 [00:21:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.16, 4.03, 3.75 [00:21:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.90, 1.89 [00:23:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.95, 4.12, 3.30 [00:23:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.92, 3.39, 3.05 [00:27:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.47, 12.30, 11.40 [00:27:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.92, 3.63, 3.19 [00:29:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.54, 11.00, 11.05 [00:29:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.68, 3.26 [00:29:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.71, 1.76 [00:31:24] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.74, 3.79, 3.56 [00:31:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.47, 2.86, 3.01 [00:31:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.61, 1.64, 1.72 [00:32:32] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.03, 10.54, 9.27 [00:33:50] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.50, 1.56, 1.68 [00:35:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 15.03, 11.14, 9.49 [00:36:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.16, 10.65, 9.54 [00:37:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.71, 3.26, 3.38 [00:38:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.92, 11.28, 10.08 [00:38:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.79, 3.53, 3.93 [00:38:54] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.04, 9.06, 9.09 [00:39:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.76, 11.89, 11.30 [00:40:18] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.05, 10.05, 9.76 [00:44:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.75, 3.68, 3.80 [00:46:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.74, 3.81 [00:49:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.56, 2.12, 1.81 [00:51:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:53:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.49, 11.68, 11.82 [00:56:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 2.80, 3.28 [00:56:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.24, 1.71, 1.34 [00:57:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.87, 1.96, 1.88 [00:58:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.76, 3.11, 2.87 [00:58:50] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.79, 1.75, 1.40 [00:59:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.55, 11.63, 11.54 [00:59:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.76, 2.14, 1.95 [01:00:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.78, 2.68, 2.75 [01:00:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.20, 2.00, 1.54 [01:02:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.35, 1.78, 1.52 [01:04:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 8.22, 3.34, 2.07 [01:05:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:05:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.99, 1.97 [01:10:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.24, 1.89, 1.82 [01:13:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.21, 10.88, 11.85 [01:14:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.33, 1.45, 1.65 [01:15:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:17:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.21, 12.81, 9.99 [01:19:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.72, 11.38, 9.84 [01:19:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.52, 1.91, 1.83 [01:21:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.93, 9.88, 9.48 [01:21:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.83, 10.53, 11.00 [01:21:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.85, 1.89, 1.83 [01:23:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:25:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.92, 11.37, 11.27 [01:25:50] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.27, 1.47, 1.67 [01:27:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.60, 12.20, 11.59 [01:29:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.57, 11.74, 11.49 [01:33:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:41:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.79, 3.50, 2.86 [01:41:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.92, 4.49, 3.54 [01:43:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.33, 3.04, 2.77 [01:43:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.77, 3.37 [01:45:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.29, 9.06, 10.12 [01:45:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.04, 4.17, 3.56 [01:47:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.98, 3.56 [01:49:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.03, 3.32, 3.37 [01:49:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.59, 1.41 [01:51:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.84, 10.10, 10.23 [01:51:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.67, 1.46 [01:52:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.98, 1.58, 1.35 [01:53:50] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.46, 1.60, 1.46 [01:55:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 4.07, 3.62 [01:56:11] [02ImportDump] 07Universal-Omega edited pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:57:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.60, 3.47, 3.46 [02:00:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.70, 1.39, 1.41 [02:02:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.43, 3.12, 3.32 [02:03:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.89, 9.78, 10.18 [02:08:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.27, 2.95 [02:10:16] PROBLEM - housing.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - housing.wiki All nameservers failed to answer the query. [02:10:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.20, 3.03, 2.89 [02:15:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.84, 3.51 [02:17:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.42, 3.28, 3.35 [02:20:43] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.23, 1.93, 1.63 [02:21:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.57, 4.36, 3.75 [02:22:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.53, 1.77, 1.61 [02:24:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:24:35] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.48, 1.67, 1.59 [02:30:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 3.61, 3.76 [02:32:22] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.20, 1.82, 1.67 [02:34:18] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.80, 1.68 [02:34:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:36:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.93, 3.34, 3.01 [02:38:10] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.17, 1.86, 1.72 [02:38:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.30, 2.85, 2.87 [02:39:31] RECOVERY - housing.wiki - reverse DNS on sslhost is OK: SSL OK - housing.wiki reverse DNS resolves to cp31.miraheze.org - CNAME FLAT [02:40:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.26, 3.64, 3.62 [02:42:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.92, 1.87, 1.76 [02:42:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.16, 3.78, 3.18 [02:42:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.53, 3.84, 3.24 [02:44:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.46, 3.18 [02:44:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.81, 2.72, 1.89 [02:45:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:46:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.08, 3.56, 3.30 [02:46:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.94, 2.84, 2.98 [02:46:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.51, 3.70, 3.73 [02:47:51] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.28, 2.03, 1.86 [02:48:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.89, 2.97, 3.11 [02:48:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.80, 2.00, 1.76 [02:50:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.94, 2.91, 3.38 [02:52:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.31, 1.55, 1.63 [02:55:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.17, 1.75, 1.87 [03:00:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:01:50] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.17, 1.44, 1.69 [03:02:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.40, 1.78, 1.66 [03:04:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.06, 1.97, 1.75 [03:06:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.58, 1.63 [03:10:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.19, 2.80 [03:12:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.90, 2.68, 2.66 [03:18:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.58, 3.27 [03:19:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.63, 1.54 [03:19:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.52, 3.37, 3.22 [03:21:50] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.10, 1.42, 1.47 [03:21:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 9.83, 4.97, 3.78 [03:22:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.42, 2.93, 2.07 [03:24:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.10, 4.41, 3.33 [03:25:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.05, 4.28, 3.37 [03:26:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.11, 1.92, 1.85 [03:29:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:30:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.98, 2.12, 1.92 [03:32:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.23, 1.74, 1.80 [03:34:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:36:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.10, 3.67, 3.62 [03:36:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.10, 1.50, 1.70 [03:38:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.88, 3.82 [03:40:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.44, 3.10, 3.39 [03:44:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.64, 3.73, 3.72 [03:44:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.44, 3.48 [03:46:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.38, 3.51, 3.65 [03:48:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.53, 4.08, 3.82 [03:48:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.89, 3.96, 3.68 [03:49:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.23, 1.80, 1.50 [03:50:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.74, 3.68, 3.62 [03:51:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.62, 1.77, 1.53 [03:52:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.93, 3.98, 3.85 [03:53:50] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.35, 1.61, 1.50 [03:54:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.24, 3.92, 3.83 [03:54:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.62, 3.08, 3.38 [03:58:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.69, 3.75 [04:00:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 3.54, 3.50 [04:02:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 4.00, 3.87 [04:04:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.37, 4.32, 3.74 [04:12:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.19, 3.61, 3.87 [04:12:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.83, 3.93, 3.87 [04:14:40] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.75, 1.61 [04:14:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.56, 4.37, 4.02 [04:16:36] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.93, 1.78, 1.64 [04:18:32] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.44, 2.02, 1.74 [04:18:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.81, 3.77, 3.87 [04:24:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.68, 3.42, 3.54 [04:24:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.10, 2.91, 3.39 [04:24:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.19, 3.14, 3.95 [04:26:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.94, 1.84 [04:26:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.30, 3.48 [04:28:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.53, 4.29, 3.80 [04:28:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.44, 3.57, 3.53 [04:28:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.67, 3.95, 4.08 [04:30:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.76, 3.72, 3.66 [04:30:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.83, 3.21, 3.40 [04:32:05] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.90, 1.85 [04:33:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:34:01] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.78, 1.81 [04:34:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.54, 3.38, 3.84 [04:36:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.29, 9.73, 8.89 [04:36:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.77, 3.71 [04:38:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.67, 9.45, 8.90 [04:38:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.74, 3.33, 3.56 [04:42:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.64, 2.73, 3.29 [04:43:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:43:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.67, 9.69, 8.26 [04:44:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.66, 10.93, 9.62 [04:44:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.82, 2.91, 3.40 [04:45:42] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.19, 6.47, 4.83 [04:45:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.37, 8.34, 7.93 [04:45:50] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.46, 1.56, 1.69 [04:46:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.87, 10.87, 9.79 [04:47:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.11, 9.14, 9.27 [04:48:15] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.05, 11.91, 9.36 [04:48:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:48:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.93, 4.48, 3.97 [04:49:42] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.00, 6.91, 5.52 [04:49:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.55, 2.09, 1.88 [04:50:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.18, 10.86, 9.29 [04:51:42] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.58, 5.84, 5.30 [04:51:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.52, 1.96, 1.87 [04:52:15] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.12, 9.63, 9.03 [04:53:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 3.52, 3.34 [04:53:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:53:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.45, 10.21, 9.63 [04:55:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.29, 9.38, 9.40 [04:55:50] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.03, 1.44, 1.67 [04:57:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.90, 3.58 [04:57:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.79, 3.46 [04:58:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.88, 3.96 [05:00:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:01:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.62, 2.85, 3.18 [05:02:50] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.59, 2.93, 3.31 [05:05:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:06:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.37, 3.56 [05:08:04] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [05:08:04] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:08:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.48, 3.47 [05:08:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.17, 3.47 [05:10:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.70, 3.54 [05:12:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.59, 3.31, 3.42 [05:14:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.30, 2.61, 3.15 [05:16:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.95, 2.98, 3.30 [05:21:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.87, 1.77, 1.51 [05:23:50] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.40, 1.64, 1.50 [05:37:26] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [05:43:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 3.18, 2.93 [05:45:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.68, 2.72, 2.80 [05:46:20] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.79, 1.60, 1.51 [05:48:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.72, 1.56 [05:52:08] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.87, 1.67 [05:56:00] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.38, 1.61, 1.61 [06:01:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.69, 1.64 [06:05:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.39, 1.89, 1.72 [06:11:36] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:31:36] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:39:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.77, 3.87, 3.16 [06:41:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.52, 3.11 [06:43:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.35, 3.22, 3.03 [06:52:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.43, 4.75, 3.59 [06:52:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.73, 3.04 [06:52:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.84, 3.82, 3.31 [06:54:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.82, 3.62, 3.32 [06:54:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.32, 3.16, 2.91 [06:54:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.63, 3.08, 3.11 [06:56:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.08, 3.00, 3.12 [07:17:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.54, 3.15 [07:18:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 9.65, 5.06, 3.71 [07:23:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:25:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.92, 3.99, 3.33 [07:26:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.00, 3.58, 3.63 [07:27:13] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.47, 3.34, 3.16 [07:34:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.24, 2.46, 3.14 [07:34:51] Is Babel fully back? [07:38:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:38:41] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [07:38:41] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:47:15] PROBLEM - experimyco.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - experimyco.com All nameservers failed to answer the query. [08:08:02] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [08:15:59] PROBLEM - experimyco.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/py [08:16:00] st-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query experimyco.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:37:24] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [09:15:15] RECOVERY - experimyco.com - reverse DNS on sslhost is OK: SSL OK - experimyco.com reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [09:23:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.50, 1.70, 1.96 [09:25:17] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.14, 1.61, 1.29 [09:27:16] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.89, 1.37, 1.24 [09:29:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.79, 1.90 [09:31:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.87, 1.81, 1.90 [09:38:59] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [09:47:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.27, 1.97, 1.89 [09:53:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.27, 1.81, 1.89 [09:59:50] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.43, 1.48, 1.70 [10:08:21] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [10:08:21] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:24:47] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.33, 1.77, 1.25 [10:26:46] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.99, 1.53, 1.23 [10:27:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:32:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.87, 4.08, 3.04 [10:34:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.75, 3.51, 2.96 [10:36:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.53, 3.29, 2.96 [10:37:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:47:38] [02mw-config] 07ugochimobi opened pull request 03#4635: remove wgJsonConfigInterwikiPrefix "c" for gpcommonswiki.... - 13https://github.com/miraheze/mw-config/pull/4635 [10:48:51] miraheze/mw-config - ugochimobi the build passed. [10:49:52] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.31, 1.86, 1.56 [10:51:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.89, 1.61 [10:57:31] PROBLEM - tensegritywiki.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/li [10:57:31] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query tensegritywiki.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:57:50] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.52, 1.64, 1.59 [11:15:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.81, 1.69, 1.59 [11:17:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.34, 1.92, 1.69 [11:23:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.73, 1.95, 1.81 [11:29:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.93, 1.84 [11:31:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.86, 1.83 [11:36:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [11:39:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.91, 3.91, 3.12 [11:41:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:41:50] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.37, 1.54, 1.68 [11:43:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.43, 3.33, 3.07 [11:46:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:49:51] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.72, 2.07, 1.82 [11:50:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 3.68, 3.28 [11:50:42] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.76, 1.68, 0.98 [11:50:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.85, 9.67, 8.01 [11:51:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.45, 1.83, 1.76 [11:52:42] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.90, 1.36, 0.94 [11:52:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.36, 8.64, 7.83 [11:53:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.79, 3.77, 3.45 [11:55:31] RECOVERY - tensegritywiki.com - reverse DNS on sslhost is OK: SSL OK - tensegritywiki.com reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [11:55:50] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.11, 1.53, 1.66 [11:55:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.27, 5.04, 3.94 [11:56:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:58:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.06, 3.85, 3.30 [11:58:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.85, 3.25 [11:59:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.51, 10.45, 9.54 [12:00:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.62, 3.94, 3.39 [12:00:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.48, 4.19, 3.43 [12:02:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.11, 3.41, 3.27 [12:02:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.23, 3.53, 3.29 [12:04:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.16, 3.15, 3.20 [12:04:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.25, 3.20, 3.20 [12:05:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.24, 3.58, 3.82 [12:05:46] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [12:05:46] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:06:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:08:57] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.83, 9.95, 9.85 [12:09:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.71, 3.74, 3.80 [12:10:42] [02mw-config] 07ugochimobi opened pull request 03#4636: fixing Tabular.JsonConfig and Map.JsonConfig for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4636 [12:11:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.56, 3.73 [12:12:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.49, 4.23, 3.64 [12:12:44] miraheze/mw-config - ugochimobi the build passed. [12:13:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.07, 4.06, 3.89 [12:14:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.02, 3.64, 3.35 [12:15:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:18:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.86, 3.49, 3.38 [12:19:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.46, 1.86, 1.54 [12:20:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.63, 3.84, 3.51 [12:21:23] [02mw-config] 07ugochimobi opened pull request 03#4637: setting JsonConfig "isLocal" to true for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4637 [12:21:50] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.27, 1.63, 1.50 [12:22:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.28, 3.96, 3.86 [12:22:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.63, 3.48 [12:23:41] miraheze/mw-config - ugochimobi the build passed. [12:25:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:26:15] [02mw-config] 07ugochimobi commented on pull request 03#4636: fixing Tabular.JsonConfig and Map.JsonConfig for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4636#issuecomment-1109733052 [12:26:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.81, 4.52, 4.06 [12:26:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.35, 4.28, 3.75 [12:28:25] Reception123: I doubt if you guys saw my question above :) [12:28:32] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.31, 1.75, 1.30 [12:30:28] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.29, 1.60, 1.30 [12:31:47] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.81, 1.84, 1.49 [12:34:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.08, 3.76, 3.99 [12:34:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.40, 3.89, 3.92 [12:35:21] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [12:35:47] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.40, 1.76, 1.54 [12:36:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.26, 4.39, 4.09 [12:38:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.74, 3.90 [12:40:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.19, 10.98, 10.15 [12:41:47] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.00, 1.58, 1.56 [12:42:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.36, 10.63, 10.14 [12:42:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.21, 3.80, 3.85 [12:42:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.78, 4.35, 4.04 [12:44:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:44:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.07, 10.18, 10.04 [12:44:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.70, 3.76, 3.83 [12:45:47] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.24, 1.72, 1.65 [12:46:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.83, 3.90 [12:47:47] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.76, 1.43, 1.55 [12:48:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.22, 3.72, 3.77 [12:49:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:49:35] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.32, 1.87, 1.65 [12:50:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.03, 10.21, 9.11 [12:51:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.85, 1.67 [12:52:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.98, 3.63, 3.69 [12:52:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.83, 10.45, 9.33 [12:53:27] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.43, 1.66, 1.62 [12:56:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.58, 3.66 [12:57:21] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.71, 1.65 [12:58:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.30, 3.89, 3.76 [12:58:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.85, 9.92, 9.50 [13:00:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.79, 3.74 [13:02:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.58, 3.72, 3.99 [13:05:05] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.83, 1.71 [13:05:22] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [13:05:22] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:06:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.00, 3.96, 3.99 [13:07:01] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.83, 1.73 [13:10:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.29, 3.10, 3.38 [13:12:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.34, 3.39, 3.76 [13:12:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.32, 2.98, 3.85 [13:16:41] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.16, 1.95, 1.83 [13:18:25] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/792882f65580...3f8b9cb68c52 [13:18:26] [02miraheze/dns] 07Reception123 033f8b9cb - add wikicompliance.com zone [13:20:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.92, 3.97 [13:21:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.38, 9.44, 9.07 [13:22:29] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.60, 1.98, 1.92 [13:22:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.09, 3.71, 3.89 [13:23:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:23:42] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.05, 5.75, 4.51 [13:23:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.96, 9.62, 9.18 [13:24:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 4.05, 3.99 [13:25:42] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.73, 5.68, 4.63 [13:26:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.12, 3.74, 3.89 [13:28:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:28:31] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.96, 2.92, 3.29 [13:29:02] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/3f8b9cb68c52...1f14a810540f [13:29:04] [02miraheze/dns] 07Reception123 031f14a81 - add dragdown.wiki zone [13:30:14] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.30, 1.47, 1.69 [13:30:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.88, 3.39, 3.27 [13:30:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.23, 3.97, 3.91 [13:32:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.79, 3.61 [13:32:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.93, 3.18, 3.21 [13:32:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.60, 3.77 [13:34:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.83, 4.05, 3.91 [13:35:21] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [13:36:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.77, 4.14, 3.78 [13:36:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.78, 3.67, 3.41 [13:36:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.81, 3.84 [13:37:50] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/1f14a810540f...9ef69bd4f95f [13:37:51] [02miraheze/dns] 07Reception123 039ef69bd - add mariopedia.org zone [13:38:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.93, 3.75 [13:40:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.61, 11.47, 10.28 [13:40:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 4.04, 3.81 [13:41:23] [02miraheze/ssl] 07Reception123 pushed 036 commits to 03master [+6/-0/±7] 13https://github.com/miraheze/ssl/compare/7d7c24f8622a...3f9e8783bda9 [13:41:24] [02miraheze/ssl] 07Reception123 036d60d95 - add wiki.evscope.org cert [13:41:26] [02miraheze/ssl] 07Reception123 0397648fb - add wikicompliance.com cert [13:41:27] [02miraheze/ssl] 07Reception123 03f554ed7 - add www.lgbtqia.wiki cert [13:41:29] [02miraheze/ssl] ... and 3 more commits. [13:42:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.91, 11.54, 10.45 [13:42:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.84, 3.84, 3.75 [13:42:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.38, 4.38, 3.71 [13:42:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.31, 3.92, 3.81 [13:44:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.57, 12.04, 10.74 [13:44:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.75, 3.95, 3.63 [13:44:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.98, 3.90, 3.83 [13:46:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.37, 3.78, 3.71 [13:46:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.00, 4.71, 3.94 [13:46:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.61, 10.66, 9.24 [13:47:21] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.80, 1.62, 1.38 [13:48:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.35, 3.63, 3.67 [13:49:21] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.82, 1.30, 1.29 [13:49:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.92, 1.67, 1.53 [13:50:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 4.01, 3.80 [13:50:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.24, 11.59, 9.87 [13:50:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.42, 4.26, 3.98 [13:51:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:51:50] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.37, 1.55, 1.51 [13:52:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.76, 10.74, 9.79 [13:54:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.67, 11.53, 11.24 [13:54:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.83, 9.72, 9.53 [13:58:42] [02mw-config] 07ugochimobi opened pull request 03#4638: T9137: set wgDefaultUserOptions['thumbsize'] to "3" on the following wikis - 13https://github.com/miraheze/mw-config/pull/4638 [13:59:54] miraheze/mw-config - ugochimobi the build passed. [14:01:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:04:48] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.67, 2.66, 1.82 [14:05:22] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [14:06:13] ugochimobi: hm? [14:08:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:08:45] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.89, 1.72, 1.63 [14:08:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.24, 3.70, 3.99 [14:09:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.81, 3.72, 3.93 [14:10:08] Reception123: you saw it? [14:10:35] what exactly? [14:10:41] I said if Babel has been enabled back, like, if we have it back [14:10:44] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.89, 1.50, 1.57 [14:10:49] you know it cause issues back then [14:11:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.72, 4.05, 4.02 [14:11:02] s/cause/caused [14:11:02] ugochimobi meant to say: you know it caused issues back then [14:11:36] do you mean if it's currently enabled or when it will be? [14:12:28] if it's currently enabled? [14:12:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.95, 4.03 [14:12:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.08, 11.16, 10.83 [14:12:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.80, 3.75, 3.93 [14:13:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:14:47] ugochimobi: yes, it is currently enabled [14:14:48] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.53, 11.38, 10.95 [14:14:55] extensions currently disabled are https://github.com/miraheze/mw-config/blob/master/LocalSettings.php#L5068 [14:14:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.08, 4.11, 4.03 [14:14:56] [url] mw-config/LocalSettings.php at master · miraheze/mw-config · GitHub | github.com [14:15:23] oh.. [14:15:31] there's a problem then [14:15:54] The Translate Extension is broken on gratisdatawiki [14:16:10] maybe that's the best description I can give [14:16:18] hmm, what errors are there? [14:16:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.10, 3.64, 3.89 [14:16:53] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.47, 2.87, 2.28 [14:16:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.94, 3.59, 3.84 [14:17:24] try visiting https://gratisdata.miraheze.org/wiki/Special:NewItem [14:17:24] You'd see everywhere [14:17:26] [url] Create a new Item - Gratisdata | gratisdata.miraheze.org [14:17:52] especially when you have {{#babel:}} tag on your userpage [14:17:52] https://meta.miraheze.org/wiki/%23babel:Template: [14:17:54] that's very odd [14:17:57] I guess [14:18:24] but happens every time, whether you have it on your userpage or not [14:18:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.12, 10.72, 10.68 [14:18:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.12, 3.84, 3.93 [14:18:50] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.81, 2.79, 2.32 [14:20:04] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/3f9e8783bda9...557f8c5f25a8 [14:20:05] [02miraheze/ssl] 07Reception123 03557f8c5 - add pastport.org cert [14:20:19] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.60, 1.93, 1.62 [14:20:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.58, 3.82 [14:20:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.57, 9.79, 9.09 [14:23:49] Reception123: what do we do? is it reportable? [14:24:00] yes, please do open a phab task [14:24:19] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.60, 1.79, 1.64 [14:24:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.25, 3.05, 3.84 [14:24:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.15, 9.71, 9.26 [14:24:43] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.27, 2.59, 3.30 [14:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:26:19] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.28, 1.63, 1.60 [14:26:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.92, 2.47, 3.24 [14:26:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.63, 11.56, 11.26 [14:31:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:32:19] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=obeymewiki (START) [14:32:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:34:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.28, 3.92, 3.79 [14:34:17] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=gratisdatawiki (START) [14:34:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:36:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.62, 3.35, 3.60 [14:36:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.81, 3.32, 3.25 [14:36:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:38:24] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.81, 3.20, 3.22 [14:38:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.39, 10.59, 10.58 [14:40:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.60, 3.77, 3.65 [14:41:33] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.05, 10.08, 9.40 [14:42:03] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 7.47, 4.32, 2.50 [14:42:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.63, 3.62 [14:43:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.62, 10.14, 9.53 [14:44:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.88, 4.08, 3.79 [14:44:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.52, 11.75, 11.12 [14:46:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.99, 3.79 [14:48:19] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.54, 2.02, 1.75 [14:50:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.09, 10.86, 9.53 [14:50:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 4.08, 3.85 [14:50:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.14, 11.35, 11.07 [14:52:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.14, 10.96, 9.76 [14:52:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.76, 3.76 [14:54:03] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.50, 1.73, 1.17 [14:54:09] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.57, 9.36, 9.32 [14:54:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.99, 3.47, 3.26 [14:54:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.30, 11.29, 11.12 [14:56:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.33, 1.69, 1.23 [14:56:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.23, 3.84, 3.78 [14:58:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.21, 4.72, 3.79 [14:58:20] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.94, 2.00, 1.93 [14:58:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.72, 4.24, 3.59 [15:01:15] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 2.30, 3.33, 3.78 [15:02:27] PROBLEM - pastport.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - pastport.org All nameservers failed to answer the query. [15:03:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.71, 10.26, 9.45 [15:03:13] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.68, 3.96, 3.96 [15:05:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.97, 11.88, 11.20 [15:07:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.57, 11.53, 11.14 [15:09:08] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.00, 9.90, 9.63 [15:09:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.29, 12.34, 11.47 [15:11:40] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.30, 11.76, 11.47 [15:13:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.45, 10.23, 8.86 [15:13:23] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=gratisdatawiki (END - exit=0) [15:13:27] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.23, 1.43, 1.68 [15:13:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:13:38] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 1.862 second response time [15:14:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.00, 9.76, 9.25 [15:14:57] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.25, 3.44, 3.98 [15:15:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.96, 11.69, 10.41 [15:15:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 10.20, 10.13, 8.98 [15:16:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.62, 9.65, 9.27 [15:17:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.68, 11.16, 10.37 [15:17:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.58, 10.20, 9.74 [15:18:52] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.61, 2.35, 3.40 [15:19:19] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.55, 1.94, 1.80 [15:20:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.11, 11.17, 9.97 [15:20:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.24, 10.51, 9.72 [15:22:39] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.60, 9.64, 9.49 [15:23:01] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [15:23:12] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.33, 1.76, 1.77 [15:24:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.38, 11.52, 10.41 [15:24:43] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 5.01, 2.68, 1.72 [15:24:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.54, 11.68, 11.49 [15:24:59] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.40, 9.93, 9.93 [15:25:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.67, 9.39, 9.96 [15:25:09] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.07, 1.53, 1.68 [15:26:03] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.05, 10.05, 10.02 [15:28:02] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:07] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:09] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:26] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:28] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:03] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 4.742 second response time [15:30:08] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22342 bytes in 5.392 second response time [15:30:12] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 4.452 second response time [15:30:28] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 4.398 second response time [15:30:30] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 4.992 second response time [15:30:41] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.12, 1.89, 1.68 [15:32:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.07, 1.68, 1.63 [15:38:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.31, 11.85, 11.80 [15:41:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.71, 9.95, 9.63 [15:42:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.91, 11.90, 11.78 [15:43:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.93, 10.47, 10.02 [15:45:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.27, 9.88, 9.68 [15:46:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.55, 11.69, 11.75 [15:46:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.13, 10.21, 9.81 [15:47:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.40, 11.37, 10.47 [15:48:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.17, 12.56, 12.06 [15:49:34] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.24, 1.78, 1.51 [15:51:01] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:51:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.45, 11.46, 10.75 [15:52:49] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.85, 10.11, 9.92 [15:53:28] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.50, 1.80, 1.60 [15:53:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.05, 11.86, 10.99 [15:55:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.67, 11.36, 10.91 [15:57:23] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.62, 1.66, 1.58 [16:03:10] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [16:03:10] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:07:53] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.33, 8.92, 9.98 [16:08:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.45, 3.36, 3.94 [16:09:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.71, 10.79, 11.66 [16:10:36] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.85, 4.06 [16:11:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:18:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:20:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.25, 1.88, 1.68 [16:22:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.49, 3.01, 3.71 [16:22:41] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.52, 1.74, 1.65 [16:22:45] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=obeymewiki (END - exit=0) [16:22:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.53, 10.74, 10.94 [16:24:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.23, 2.93, 3.76 [16:25:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.86, 11.12, 11.06 [16:25:57] [02mw-config] 07Universal-Omega closed pull request 03#4635: remove wgJsonConfigInterwikiPrefix "c" for gpcommonswiki.... - 13https://github.com/miraheze/mw-config/pull/4635 [16:25:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/fd10953b45f4...b7e784a3960a [16:26:00] [02miraheze/mw-config] 07ugochimobi 03b7e784a - remove wgJsonConfigInterwikiPrefix "c" for gpcommonswiki.... (#4635) [16:27:13] miraheze/mw-config - Universal-Omega the build passed. [16:27:24] [02mw-config] 07Universal-Omega closed pull request 03#4637: setting JsonConfig "isLocal" to true for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4637 [16:27:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b7e784a3960a...0fe0b4dbe2dd [16:27:27] [02miraheze/mw-config] 07ugochimobi 030fe0b4d - setting JsonConfig "isLocal" to true for gpcommonswiki (#4637) [16:27:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.22, 10.98, 10.99 [16:28:31] [02mw-config] 07Universal-Omega closed pull request 03#4638: T9137: set wgDefaultUserOptions['thumbsize'] to "3" on the following wikis - 13https://github.com/miraheze/mw-config/pull/4638 [16:28:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0fe0b4dbe2dd...90e34e64de1a [16:28:34] [02miraheze/mw-config] 07ugochimobi 0390e34e6 - T9137: set wgDefaultUserOptions['thumbsize'] to "3" on the following wikis (#4638) [16:28:34] miraheze/mw-config - Universal-Omega the build passed. [16:29:36] miraheze/mw-config - Universal-Omega the build passed. [16:30:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.66, 3.90, 3.87 [16:31:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.30, 11.24, 11.11 [16:32:05] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [16:32:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.46, 3.70 [16:34:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.01, 2.75, 3.31 [16:36:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.85, 3.80 [16:37:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.17, 11.37, 11.21 [16:40:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.81, 3.81 [16:42:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.32, 11.03, 9.84 [16:44:06] [02ImportDump] 07Universal-Omega edited pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:44:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.31, 3.47, 3.64 [16:44:39] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.24, 1.98, 1.85 [16:44:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.96, 10.51, 9.78 [16:46:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.75, 10.08, 9.73 [16:46:42] [02ImportDump] 07Universal-Omega edited pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:47:06] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/32e5514e5e7e...25d51c238a1f [16:47:07] [02miraheze/ImportDump] 07Universal-Omega 0325d51c2 - Update Main.php [16:47:09] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:47:15] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/25d51c238a1f...8c19355d6d6f [16:47:17] [02miraheze/ImportDump] 07Universal-Omega 038c19355 - Update extension.json [16:47:18] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:47:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.37, 11.91, 11.85 [16:47:49] [02ImportDump] 07Universal-Omega edited pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:47:52] [02ImportDump] 07Universal-Omega edited pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:49:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.37, 12.10, 11.95 [16:50:02] [02ImportDump] 07Universal-Omega edited pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:50:14] !log [@test101] starting deploy of {'config': True} to all [16:50:15] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [16:50:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:50:21] miraheze/ImportDump - Universal-Omega the build has errored. [16:50:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:51:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.28, 11.38, 11.70 [16:52:17] !log [@mwtask111] starting deploy of {'config': True} to all [16:52:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:52:35] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 18s [16:52:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.38, 3.97, 3.90 [16:52:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:52:52] [02miraheze/ImportDump] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/8c19355d6d6f...704b033d0333 [16:52:53] [02miraheze/ImportDump] 07github-actions 03704b033 - CI: lint code to MediaWiki standards [16:52:55] [02ImportDump] 07github-actions[bot] synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:53:33] [02ImportDump] 07Universal-Omega edited pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:54:34] miraheze/ImportDump - Universal-Omega the build has errored. [16:54:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.53, 1.86, 1.91 [16:56:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.62, 3.97 [16:57:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.74, 11.16, 11.40 [16:58:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:59:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.42, 9.58, 9.06 [16:59:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.37, 11.24, 11.41 [17:00:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 3.62, 3.66 [17:01:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.82, 10.49, 9.43 [17:02:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 4.06, 3.96 [17:02:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.47, 3.68, 3.69 [17:02:39] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.28, 1.38, 1.64 [17:03:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.72, 11.26, 9.87 [17:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:03:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.00, 11.74, 11.54 [17:04:14] miraheze/ImportDump - Universal-Omega the build has errored. [17:04:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.22, 3.77, 3.86 [17:05:11] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.36, 10.03, 9.61 [17:06:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.55, 3.70, 3.66 [17:07:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.85, 11.37, 11.50 [17:08:05] miraheze/ImportDump - Universal-Omega the build has errored. [17:08:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.18, 3.77, 3.82 [17:08:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:08:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 3.60, 3.63 [17:10:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.78, 3.82 [17:12:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 3.90, 3.86 [17:13:16] miraheze/ImportDump - Universal-Omega the build has errored. [17:14:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.96, 3.91, 3.87 [17:16:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.95, 4.20, 3.97 [17:16:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.90, 3.61, 3.55 [17:17:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.92, 3.57 [17:18:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.85, 3.24, 3.43 [17:19:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.39, 11.24, 11.05 [17:20:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.29, 3.52, 3.77 [17:20:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.79, 2.93, 3.28 [17:20:39] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.23, 1.72, 1.53 [17:21:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.25, 10.07, 9.53 [17:21:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.79, 10.51, 10.81 [17:21:57] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 4.03, 3.67 [17:22:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 4.43, 4.11 [17:22:39] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.38, 1.64, 1.53 [17:23:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.05, 9.32, 9.33 [17:23:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.20, 11.58, 11.16 [17:23:46] miraheze/ImportDump - Universal-Omega the build has errored. [17:23:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.32, 3.40, 3.48 [17:24:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.97, 3.99 [17:24:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:25:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.94, 3.23, 3.40 [17:26:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.57, 4.21, 4.07 [17:26:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.96, 4.68, 3.88 [17:28:04] miraheze/ImportDump - Universal-Omega the build has errored. [17:28:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.77, 3.91 [17:28:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.39, 3.77, 3.65 [17:29:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.79, 11.70, 11.39 [17:30:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.22, 4.11, 4.00 [17:30:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 4.05, 3.77 [17:31:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.13, 10.35, 9.83 [17:31:51] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [17:31:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 3.56, 3.45 [17:32:28] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/704b033d0333...8130e341223b [17:32:30] [02miraheze/ImportDump] 07Universal-Omega 038130e34 - Update SpecialRequestImportDump.php [17:32:31] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [17:33:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.62, 10.35, 9.66 [17:33:14] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.70, 9.83, 9.70 [17:33:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.10, 12.02, 11.59 [17:33:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.87, 3.47, 3.44 [17:34:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.70, 3.79, 3.76 [17:35:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.99, 10.13, 9.67 [17:35:35] miraheze/ImportDump - Universal-Omega the build has errored. [17:35:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.58, 2.81, 3.21 [17:36:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.19, 3.51, 3.88 [17:39:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.33, 10.81, 9.37 [17:40:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.95, 2.99, 3.37 [17:43:05] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.14, 1.73, 1.46 [17:43:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.73, 10.94, 9.76 [17:45:05] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.56, 1.63, 1.45 [17:45:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.45, 9.77, 9.48 [17:46:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 3.71, 3.69 [17:47:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.68, 11.04, 10.18 [17:47:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.46, 11.66, 11.84 [17:49:34] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.69, 2.00, 1.67 [17:49:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.73, 10.89, 10.23 [17:49:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.17, 11.92, 11.92 [17:49:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.23, 10.62, 9.87 [17:51:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.97, 1.71 [17:51:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.71, 11.59, 11.79 [17:51:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.97, 3.55 [17:52:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.90, 4.68, 3.82 [17:53:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.53, 12.04, 11.94 [17:53:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.91, 3.58 [17:55:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.15, 11.86, 11.90 [17:55:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 4.10, 3.70 [17:57:21] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.32, 1.59, 1.62 [17:57:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.64, 12.12, 11.99 [17:57:59] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.60, 1.95, 1.66 [17:59:41] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.05, 9.72, 9.90 [17:59:59] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.07, 1.61, 1.57 [18:00:46] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [18:01:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.22, 11.48, 11.80 [18:01:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.85, 3.97, 3.79 [18:03:38] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.77, 11.71, 10.63 [18:03:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.93, 12.37, 12.09 [18:03:57] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 19.23, 12.98, 10.86 [18:04:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.56, 12.04, 9.98 [18:04:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.97, 3.60, 3.96 [18:04:52] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 16.71, 13.08, 11.38 [18:05:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 19.14, 13.69, 10.80 [18:05:28] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:36] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.37, 10.28, 10.26 [18:05:40] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.245 second response time [18:05:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.52, 11.49, 11.82 [18:05:45] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:50] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:55] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:56] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:06:18] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:06:25] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:07:06] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.38, 1.85, 1.68 [18:07:28] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22342 bytes in 2.228 second response time [18:07:36] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.64, 8.62, 9.65 [18:07:49] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.017 second response time [18:07:52] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.305 second response time [18:07:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.67, 2.95, 3.34 [18:08:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.52, 11.72, 10.48 [18:08:16] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.339 second response time [18:08:21] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.017 second response time [18:08:33] [02dns] 07MacFan4000 closed pull request 03#269: add wikicompliance.com zone - 13https://github.com/miraheze/dns/pull/269 [18:08:56] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 2.576 second response time [18:08:57] Reception123: also you seem not to have seen ^ [18:09:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.10, 11.43, 11.23 [18:09:33] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.026 second response time [18:09:43] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.333 second response time [18:09:45] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22342 bytes in 0.430 second response time [18:09:49] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.021 second response time [18:11:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.77, 11.83, 11.16 [18:11:26] MacFan4000: oh my bad, I didn't realise you already did the DNS :( [18:11:30] will check PRs next time before starting [18:12:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.73, 10.06, 10.07 [18:12:18] [02dns] 07Reception123 closed pull request 03#268: Re-add pastport.org zone - 13https://github.com/miraheze/dns/pull/268 [18:12:19] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/9ef69bd4f95f...f285967eb5e1 [18:12:21] [02miraheze/dns] 07MacFan4000 03f285967 - Re-add pastport.org zone (#268) [18:13:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.08, 11.34, 11.77 [18:14:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:14:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.59, 3.39, 3.50 [18:15:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.52, 3.43 [18:16:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.52, 3.55 [18:17:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.21, 3.10, 3.28 [18:18:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.24, 3.66, 3.58 [18:21:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.43, 9.00, 10.01 [18:21:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.91, 11.22, 10.96 [18:21:52] !log reception@mwtask111:~$ sudo -u www-data php /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/assignImportedEdits.php --wiki=waysofdarknesswiki --import-prefix="waysofdarkness" [18:21:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:22:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.81, 3.65, 3.63 [18:23:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.45, 9.05, 9.97 [18:24:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.84, 4.17, 3.82 [18:25:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.08, 11.45, 11.12 [18:25:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.38, 3.42, 3.38 [18:26:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.58, 3.63 [18:26:42] RECOVERY - pastport.org - reverse DNS on sslhost is OK: SSL OK - pastport.org reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [18:26:49] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.69, 2.62, 1.76 [18:27:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:28:06] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.66, 4.28, 2.30 [18:30:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.59, 9.94, 9.22 [18:30:48] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.24, 1.84, 1.64 [18:31:38] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [18:31:38] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:31:53] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.67, 8.95, 10.09 [18:31:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.92, 3.19, 3.33 [18:34:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 4.10, 3.80 [18:34:47] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.80, 1.40, 1.51 [18:35:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.87, 11.40, 11.07 [18:35:52] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.84, 1.62, 1.76 [18:35:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.79, 3.50, 3.47 [18:36:20] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.30, 10.08, 9.60 [18:36:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.91, 3.76 [18:37:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.39, 11.49, 11.16 [18:37:48] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.79, 1.35, 1.64 [18:39:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.40, 10.21, 9.53 [18:41:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.23, 10.20, 9.57 [18:42:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.27, 3.28, 3.91 [18:43:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.89, 10.37, 9.80 [18:43:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.16, 10.31, 9.70 [18:43:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.79, 3.63 [18:45:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.51, 9.35, 9.50 [18:45:15] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.32, 9.95, 9.64 [18:45:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.88, 12.20, 11.61 [18:45:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.36, 3.49, 3.53 [18:47:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.51, 1.92, 1.74 [18:47:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.01, 11.57, 11.45 [18:47:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.07, 2.94, 3.32 [18:48:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.63, 2.84, 3.28 [18:48:45] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/8f5e6700fd71 [18:48:46] [02miraheze/MirahezeMagic] 07Universal-Omega 038f5e670 - AssignImportedEdits: use ActorNormalization [18:48:48] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/MirahezeMagic [18:48:49] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.23, 11.51, 10.42 [18:48:49] [02MirahezeMagic] 07Universal-Omega opened pull request 03#336: AssignImportedEdits: use ActorNormalization - 13https://github.com/miraheze/MirahezeMagic/pull/336 [18:50:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.48, 11.55, 10.58 [18:51:43] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.96, 1.55, 1.62 [18:52:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:53:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.17, 11.90, 11.58 [18:54:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.75, 3.76, 3.65 [18:54:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.80, 9.71, 10.07 [18:55:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.42, 4.52, 3.75 [18:55:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.66, 11.28, 11.39 [18:55:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.66, 4.73, 3.87 [18:57:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.07, 1.90, 1.76 [18:58:05] miraheze/MirahezeMagic - Universal-Omega the build passed. [18:58:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.19, 10.53, 10.34 [18:59:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.10, 1.61, 1.67 [19:00:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:00:33] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [19:03:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.56, 11.46, 11.36 [19:07:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.43, 11.61, 11.45 [19:10:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 10.03, 9.89, 10.17 [19:11:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.97, 3.34, 3.86 [19:15:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:20:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.76, 10.45, 10.17 [19:20:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.46, 3.47, 3.98 [19:21:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:22:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.81, 3.73, 4.02 [19:23:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 1.32, 1.01 [19:23:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.63, 12.11, 11.33 [19:25:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.26, 1.19, 1.00 [19:25:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.10, 3.47, 3.57 [19:27:28] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.02, 1.58, 1.31 [19:28:28] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.71, 9.90, 10.08 [19:29:27] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.18, 1.46, 1.31 [19:29:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.97, 10.38, 9.68 [19:29:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.52, 11.63, 11.42 [19:31:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [19:31:25] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:31:38] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.80, 10.10, 9.65 [19:31:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 11.70, 11.46 [19:32:59] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.72, 10.49, 9.64 [19:33:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.95, 11.57, 11.43 [19:36:59] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.04, 9.56, 9.48 [19:37:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.82, 10.48, 9.92 [19:37:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.62, 3.90, 3.91 [19:38:23] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.48, 1.86, 1.58 [19:39:16] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.21, 10.86, 10.11 [19:40:22] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.49, 2.01, 1.66 [19:41:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.30, 11.16, 10.33 [19:41:47] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [19:41:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.71, 4.13, 3.98 [19:41:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.10, 10.55, 9.64 [19:42:22] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.73, 1.89, 1.66 [19:43:04] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.11, 10.20, 10.08 [19:43:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.95, 3.95 [19:44:21] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.87, 1.46, 1.52 [19:45:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.49, 10.55, 9.95 [19:47:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.95, 9.52, 9.65 [19:47:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.93, 4.06, 3.97 [19:48:59] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.45, 10.61, 10.08 [19:49:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.84, 9.66, 9.62 [19:49:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.94, 3.94 [19:50:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.14, 3.71, 3.97 [19:50:59] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.91, 9.60, 9.77 [19:53:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.87, 3.90 [19:55:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.30, 9.56, 10.16 [19:55:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.20, 3.43, 3.73 [19:59:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.47, 4.15, 3.95 [20:00:20] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [20:00:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 4.04, 3.92 [20:03:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.42, 3.49, 3.77 [20:06:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:06:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.69, 3.64, 3.85 [20:07:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.93, 9.94, 9.61 [20:09:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.68, 9.80, 9.59 [20:09:47] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:11:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:11:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.47, 3.62 [20:12:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.12, 4.09, 3.88 [20:14:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.18, 3.23, 3.59 [20:15:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.74, 3.42, 3.60 [20:16:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.09, 2.82, 3.39 [20:16:48] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.95, 10.85, 9.97 [20:17:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.64, 10.30, 10.01 [20:18:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.12, 10.54, 9.99 [20:19:38] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.41, 9.19, 9.64 [20:20:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.52, 3.54 [20:21:13] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.84, 12.11, 10.92 [20:21:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.33, 2.97, 3.35 [20:22:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.53, 9.83, 9.85 [20:28:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.99, 3.72 [20:28:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.25, 3.97, 3.62 [20:29:15] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [20:29:15] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:30:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.74, 3.75, 3.99 [20:30:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.98, 3.53, 3.57 [20:30:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 3.62, 3.53 [20:30:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.91, 11.88, 11.39 [20:32:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.58, 11.05, 10.24 [20:32:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.55, 12.28, 11.60 [20:34:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 4.17, 4.10 [20:34:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.36, 10.14, 10.02 [20:34:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.16, 3.79, 3.60 [20:36:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.52, 3.53 [20:38:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.54, 3.50, 3.87 [20:38:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.79, 3.10, 3.35 [20:38:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.22, 3.02, 3.34 [20:40:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.11, 11.01, 11.45 [20:42:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 3.55, 3.46 [20:43:22] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#336: AssignImportedEdits: use ActorNormalization - 13https://github.com/miraheze/MirahezeMagic/pull/336 [20:43:23] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/8f5e6700fd71...e6f89b032381 [20:43:25] [02miraheze/MirahezeMagic] 07Universal-Omega 03e6f89b0 - rewrite to support --from and --to; drop --user [20:44:20] [02MirahezeMagic] 07Universal-Omega edited pull request 03#336: Rewrite AssignImportedEdits to support `--from` and `--to`; drop `--user` and use ActorNormalization - 13https://github.com/miraheze/MirahezeMagic/pull/336 [20:44:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.77, 3.82, 3.85 [20:44:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.94, 3.55, 3.46 [20:46:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:46:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.43, 3.04, 3.28 [20:50:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.64, 3.33, 3.41 [20:52:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.26, 3.83, 3.34 [20:53:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:55:02] miraheze/MirahezeMagic - Universal-Omega the build passed. [20:56:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.60, 11.89, 11.45 [20:56:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.64, 3.92, 3.65 [20:56:35] !log sudo -u www-data php /home/universalomega/assignImportedEdits.php --wiki=waysofdarknesswiki --import-prefix="waysofdarkness" --from=Metalhead33 --to=Metalhead339 [20:56:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:58:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.87, 11.73, 11.43 [20:58:10] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [20:58:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.85, 3.96, 3.70 [21:00:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.94, 3.52, 3.95 [21:00:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.76, 3.66, 3.58 [21:01:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.66, 10.19, 9.64 [21:01:44] [02MirahezeMagic] 07Universal-Omega closed pull request 03#336: Rewrite AssignImportedEdits to support `--from` and `--to`; drop `--user` and use ActorNormalization - 13https://github.com/miraheze/MirahezeMagic/pull/336 [21:01:46] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/6a608f2c6f95...050420fcaf8c [21:01:47] [02miraheze/MirahezeMagic] 07Universal-Omega 03050420f - Rewrite AssignImportedEdits to support `--from` and `--to`; drop `--user` and use ActorNormalization (#336) [21:01:49] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/MirahezeMagic [21:01:50] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:02:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.75, 3.98 [21:03:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:03:37] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.06, 9.36, 9.40 [21:03:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.25, 11.46, 11.39 [21:04:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.55, 3.88 [21:04:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.67, 3.14, 3.38 [21:05:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.45, 11.52, 11.45 [21:06:30] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.77, 3.18, 3.40 [21:06:59] miraheze/MirahezeMagic - Universal-Omega the build has errored. [21:09:20] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/8130e341223b...e76ac5f7c49d [21:09:22] [02miraheze/ImportDump] 07Universal-Omega 03e76ac5f - Update ServiceWiring.php [21:09:23] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [21:09:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:10:00] [02mw-config] 07ugochimobi commented on pull request 03#4636: fixing Tabular.JsonConfig and Map.JsonConfig for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4636#issuecomment-1110251853 [21:10:01] [02mw-config] 07ugochimobi closed pull request 03#4636: fixing Tabular.JsonConfig and Map.JsonConfig for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4636 [21:12:23] miraheze/ImportDump - Universal-Omega the build has errored. [21:14:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 3.87, 3.91 [21:16:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.75, 3.46, 3.75 [21:20:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 3.80, 3.79 [21:21:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.19, 10.73, 10.67 [21:22:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.48, 3.18 [21:24:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.12, 3.91, 3.89 [21:24:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.39, 3.04, 3.05 [21:26:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.39, 3.91, 3.87 [21:27:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.90, 11.95, 11.35 [21:28:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.73, 3.82 [21:39:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:44:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.94, 3.77, 3.65 [21:46:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.61, 3.77, 3.35 [21:47:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.80, 9.04, 10.09 [21:48:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:49:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.64, 3.74, 3.28 [21:49:25] [02mw-config] 07ugochimobi synchronize pull request 03#4612: override page HTML title with the wikibase label, if available, or else the id - 13https://github.com/miraheze/mw-config/pull/4612 [21:50:30] miraheze/mw-config - ugochimobi the build passed. [21:50:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.79, 3.49 [21:50:52] CosmicAlpha: I am confident with this now^^ [21:51:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.78, 3.82, 3.37 [21:52:18] [02mw-config] 07ugochimobi commented on pull request 03#4612: override page HTML title with the wikibase label, if available, or else the id - 13https://github.com/miraheze/mw-config/pull/4612#issuecomment-1110285739 [21:52:21] !log sudo -u www-data php /home/universalomega/assignImportedEdits.php --wiki=obeymewiki --import-prefix="wikia:obey-me" --from=Shironecko [21:52:21] !log sudo -u www-data php /home/universalomega/assignImportedEdits.php --wiki=obeymewiki --import-prefix="wikia:obey-me" --from=Windchimey [21:52:21] !log sudo -u www-data php /home/universalomega/assignImportedEdits.php --wiki=obeymewiki --import-prefix="wikia:obey-me" --from=Devigarbage69 --to=Devi [21:52:22] !log sudo -u www-data php /home/universalomega/assignImportedEdits.php --wiki=obeymewiki --import-prefix="wikia:obey-me" --from=Breeze94 --to=LOEY [21:52:22] !log sudo -u www-data php /home/universalomega/assignImportedEdits.php --wiki=obeymewiki --import-prefix="wikia:obey-me" --from=Joye0627 [21:52:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:52:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:52:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:52:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:53:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.74, 3.77, 3.42 [21:55:28] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/e76ac5f7c49d...72772ff16d85 [21:55:30] [02miraheze/ImportDump] 07Universal-Omega 0372772ff - Update Main.php [21:55:31] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [21:57:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.46, 3.85, 3.51 [21:58:13] miraheze/ImportDump - Universal-Omega the build has errored. [21:58:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.39, 3.68, 3.93 [21:58:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:58:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.85, 1.85, 2.00 [21:59:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.84, 3.36, 3.37 [22:00:50] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/72772ff16d85...5afba079c5f7 [22:00:52] [02miraheze/ImportDump] 07Universal-Omega 035afba07 - Update Main.php [22:00:53] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [22:01:59] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/5afba079c5f7...f6765d31cc2a [22:02:01] [02miraheze/ImportDump] 07Universal-Omega 03f6765d3 - Update Main.php [22:02:02] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [22:02:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.14, 2.56, 3.39 [22:02:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.31, 2.94, 3.29 [22:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:04:50] miraheze/ImportDump - Universal-Omega the build has errored. [22:05:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.41, 10.55, 9.31 [22:06:14] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/f6765d31cc2a...96a8268d79ea [22:06:16] [02miraheze/ImportDump] 07Universal-Omega 0396a8268 - Update en.json [22:06:17] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [22:06:19] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/96a8268d79ea...2b189c6afd73 [22:06:20] [02miraheze/ImportDump] 07Universal-Omega 032b189c6 - Update qqq.json [22:06:22] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [22:06:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.18, 11.57, 10.64 [22:07:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.32, 10.37, 9.40 [22:08:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.08, 10.74, 10.46 [22:09:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.58, 9.51, 9.19 [22:09:31] miraheze/ImportDump - Universal-Omega the build has errored. [22:10:39] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.29, 1.43, 1.70 [22:12:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.24, 3.23 [22:13:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.01, 3.69, 3.32 [22:14:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.50, 3.86, 3.47 [22:14:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.12, 9.58, 10.05 [22:15:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.76, 3.45, 3.26 [22:16:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.53, 3.54, 3.41 [22:17:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.12, 3.32, 3.24 [22:18:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.36, 4.07, 3.61 [22:18:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:18:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.61, 1.65 [22:20:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.32, 3.74, 3.55 [22:21:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.77, 3.48 [22:22:41] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.35, 1.55, 1.63 [22:23:01] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [22:23:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 3.77, 3.54 [22:24:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.68, 4.39, 3.81 [22:25:22] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.87, 1.45, 0.93 [22:25:32] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.37, 9.69, 8.68 [22:25:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.31, 3.65, 3.53 [22:25:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.63, 4.15, 3.68 [22:27:21] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.20, 1.50, 1.02 [22:27:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.30, 9.47, 8.72 [22:27:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.53, 3.86, 3.61 [22:27:40] [02mw-config] 07ugochimobi opened pull request 03#4639: set wgExtraLanguageNames to "qqq" for gpcommonswiki and gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/4639 [22:28:52] miraheze/mw-config - ugochimobi the build passed. [22:29:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.86, 3.68 [22:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:31:08] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/2b189c6afd73...9b70c5b3ae9a [22:31:10] [02miraheze/ImportDump] 07Universal-Omega 039b70c5b - Check UploadFromUrl permissions and upload enabled stuff in submission. [22:31:11] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [22:31:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.83, 3.68 [22:31:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.33, 10.79, 9.43 [22:32:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.93, 3.84 [22:33:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.08, 4.32, 3.87 [22:33:27] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.02, 9.92, 9.27 [22:33:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.41, 4.34, 3.87 [22:34:00] miraheze/ImportDump - Universal-Omega the build has errored. [22:34:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.96, 4.82, 4.20 [22:35:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.51, 3.79, 3.74 [22:35:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:35:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.88, 11.03, 10.07 [22:35:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.08, 3.52, 3.63 [22:37:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.59, 10.66, 10.04 [22:38:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.88, 3.95 [22:38:58] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.05, 2.93, 3.40 [22:39:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.40, 3.53 [22:41:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.06, 2.95, 3.36 [22:42:14] [02mw-config] 07ugochimobi opened pull request 03#4640: Wikibase configurations for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4640 [22:43:23] miraheze/mw-config - ugochimobi the build passed. [22:44:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 3.87, 3.87 [22:45:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.05, 9.83, 10.01 [22:47:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:48:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.95, 3.91, 3.89 [22:49:34] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.20, 1.83, 1.53 [22:50:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.67, 3.31, 3.30 [22:51:01] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [22:51:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.59, 1.76, 1.54 [22:52:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.64, 4.19, 4.03 [22:52:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.52, 10.87, 10.40 [22:53:27] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.56, 1.69, 1.54 [22:54:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.06, 3.56, 3.83 [22:54:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.01, 3.11, 3.27 [22:59:55] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/9b70c5b3ae9a...cb04338d3665 [22:59:56] [02miraheze/ImportDump] 07Universal-Omega 03cb04338 - Add additional checks to stop request approval if user has been blocked since request was opened. [22:59:59] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [23:00:03] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:04] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:06] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/cb04338d3665...52183265a650 [23:00:07] [02miraheze/ImportDump] 07Universal-Omega 035218326 - Update en.json [23:00:09] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [23:00:11] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/52183265a650...6c18adab091d [23:00:12] [02miraheze/ImportDump] 07Universal-Omega 036c18ada - Update qqq.json [23:00:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:14] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [23:00:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:30] [02mw-config] 07ugochimobi opened pull request 03#4641: setting up entitysource for gratispaideiawiki, gpcommonswiki, gratisd… - 13https://github.com/miraheze/mw-config/pull/4641 [23:01:43] miraheze/mw-config - ugochimobi the build passed. [23:02:22] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.39, 9.53, 10.04 [23:03:13] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.71, 2.44, 1.91 [23:03:52] miraheze/ImportDump - Universal-Omega the build has errored. [23:06:18] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/6c18adab091d...4d0e6a78f1c4 [23:06:20] [02miraheze/ImportDump] 07Universal-Omega 034d0e6a7 - Update extension.json [23:06:21] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [23:06:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 3.86, 3.78 [23:07:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:08:36] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:10:11] PROBLEM - cp31 Stunnel HTTP for test101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:10:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.11, 3.33, 3.61 [23:12:12] RECOVERY - cp31 Stunnel HTTP for test101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14636 bytes in 1.351 second response time [23:13:58] !log reboot test101 [23:14:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:14:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:15:18] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.85, 0.51, 0.17 [23:15:59] miraheze/ImportDump - Universal-Omega the build passed. [23:16:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.60, 3.83, 3.70 [23:16:25] PROBLEM - test101 Check Gluster Clients on test101 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [23:17:10] PROBLEM - test101 JobChron Service on test101 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [23:17:15] PROBLEM - test101 JobRunner Service on test101 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [23:17:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.10, 0.66, 0.27 [23:18:21] RECOVERY - test101 Check Gluster Clients on test101 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [23:19:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:19:08] RECOVERY - test101 JobChron Service on test101 is OK: PROCS OK: 1 process with args 'redisJobChronService' [23:19:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.58, 3.73, 3.34 [23:19:14] RECOVERY - test101 JobRunner Service on test101 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [23:19:40] what... Memcached error for key "betawiki:messages:en:status" on server "127.0.0.1:11213": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY [23:19:51] paladox: ^ do you have any idea what that is? [23:20:07] Nope [23:21:23] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1279s [23:21:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:21:37] paladox: Memcached connection failure errors in logs, and logic exceptions on-wiki (on test101 only, thankfully) "Memcached error for key "global:CreateWiki:betawiki" on server "127.0.0.1:11212": CONNECTION FAILURE" [23:22:36] nutcracker hasn't started or it's failed [23:23:22] yes I just checked that [23:23:30] https://www.irccloud.com/pastebin/ANJ0Jubk/ [23:23:50] I see why [23:24:11] fixed [23:24:14] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 51878 bytes in 1.203 second response time [23:24:23] !log test101: created /var/log/nutcracker [23:24:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:24:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.24, 10.59, 10.10 [23:24:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.92, 3.59 [23:25:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.52, 3.25 [23:26:10] Thanks paladox! And yeah I realised that was the error once I posted the above status and saw it. Thanks for fixing though. [23:26:24] yw [23:26:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.87, 3.60 [23:26:47] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.42, 9.52, 9.78 [23:28:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.60, 3.54 [23:29:30] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.72, 3.33, 3.26 [23:31:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.18, 1.86, 1.47 [23:32:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.59, 3.66, 3.55 [23:34:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.72, 3.59 [23:35:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.95, 1.58, 1.46 [23:36:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.80, 3.84, 4.00 [23:38:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:38:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.22, 2.94, 3.30 [23:42:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.84, 2.63, 3.40 [23:43:28] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/4d0e6a78f1c4...0eddd34a590e [23:43:30] [02miraheze/ImportDump] 07Universal-Omega 030eddd34 - Add getInvolvedUsers() function [23:43:32] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [23:46:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.91, 4.17, 3.80 [23:48:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:48:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.47, 3.27 [23:50:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.85, 3.80, 3.43 [23:52:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.81, 3.77 [23:54:25] miraheze/ImportDump - Universal-Omega the build passed. [23:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:59:46] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/0eddd34a590e...184934edde67 [23:59:47] [02miraheze/ImportDump] 07Universal-Omega 03184934e - Update ImportDumpRequestManager.php [23:59:50] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1