[00:01:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.18, 10.39, 11.18 [00:01:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 3.26, 3.69 [00:03:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.99, 10.68, 11.20 [00:04:56] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 3933 MB (10% inode=96%); [00:05:31] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 3943 MB (10% inode=96%); [00:06:56] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 11218 MB (29% inode=96%); [00:07:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 19.71, 13.02, 11.92 [00:08:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.17, 11.73, 10.33 [00:09:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.36, 3.42 [00:09:31] RECOVERY - cp30 Disk Space on cp30 is OK: DISK OK - free space: / 11162 MB (29% inode=96%); [00:11:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.06, 3.89, 3.59 [00:11:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.57, 11.74, 11.77 [00:13:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.87, 3.46, 3.47 [00:13:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.14, 12.09, 11.87 [00:15:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.89, 11.34, 11.63 [00:17:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.81, 12.42, 11.99 [00:19:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.28, 3.84, 3.98 [00:22:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.89, 11.86, 11.58 [00:23:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.58, 4.28, 3.70 [00:23:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.34, 4.09, 4.02 [00:25:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.86, 3.62 [00:25:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.70, 3.52, 3.82 [00:29:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.37, 2.90, 3.28 [00:30:49] hello [00:33:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.16, 4.92, 4.21 [00:33:43] Hello Hispano76 [00:37:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.22, 10.11, 11.49 [00:38:51] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.21, 9.23, 10.19 [00:39:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.77, 3.27, 3.73 [00:42:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.17, 10.40, 10.50 [00:43:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.65, 2.73, 3.38 [00:46:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.97, 10.91, 10.63 [00:48:51] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.34, 9.55, 10.17 [00:49:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.58, 8.96, 10.19 [00:54:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.79, 10.25, 10.23 [00:55:18] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.84, 1.56, 1.98 [00:56:51] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.62, 9.81, 10.05 [01:01:18] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.60, 1.76, 1.89 [01:06:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.71, 10.28, 10.21 [01:08:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.84, 11.66, 10.71 [01:11:08] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.28, 1.66, 1.98 [01:11:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.34, 11.82, 10.59 [01:13:08] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.98, 1.98, 2.07 [01:13:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.80, 11.07, 10.47 [01:14:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.59, 11.92, 11.21 [01:15:18] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.46, 1.83, 1.96 [01:15:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.57, 10.04, 10.16 [01:17:08] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.44, 1.83, 2.00 [01:19:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.12, 10.75, 10.40 [01:21:08] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.56, 2.00, 2.02 [01:21:18] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.70, 2.54, 2.18 [01:24:56] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.51, 3.62, 2.69 [01:26:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.57, 11.33, 11.03 [01:26:59] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 2.99, 7.98, 4.31 [01:27:29] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.12, 1.95, 1.96 [01:28:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.89, 10.80, 10.88 [01:28:58] PROBLEM - mon141 Current Load on mon141 is WARNING: WARNING - load average: 0.69, 5.47, 3.83 [01:30:58] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 0.32, 3.83, 3.43 [01:31:27] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.50, 1.85, 1.93 [01:33:26] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.50, 2.11, 2.01 [01:34:12] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.57, 9.11, 10.11 [01:40:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.03, 11.10, 11.94 [01:50:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.62, 12.05, 11.68 [01:53:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.32, 4.25, 3.24 [01:57:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.57, 3.59, 3.21 [01:59:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.54, 3.15, 3.09 [01:59:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.14, 10.68, 9.92 [02:01:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.33, 9.74, 9.66 [02:05:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.81, 3.90, 3.37 [02:07:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.59, 3.32 [02:10:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.96, 1.42, 1.95 [02:11:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.36, 3.05, 3.16 [02:12:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.82, 10.57, 11.87 [02:15:57] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.06, 1.79, 1.95 [02:17:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.86, 1.96 [02:19:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.30, 1.93, 1.97 [02:24:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.75, 10.11, 10.59 [02:26:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.66, 10.24, 10.58 [02:27:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.76, 3.55, 2.94 [02:29:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.22, 3.31, 2.92 [02:32:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.56, 8.52, 9.89 [02:53:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.00, 1.60, 1.91 [03:05:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.47, 10.16, 8.93 [03:07:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.67, 1.54, 1.69 [03:09:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.14, 3.70, 2.93 [03:11:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.84, 3.85, 3.09 [03:11:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.21, 11.72, 9.99 [03:11:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.80, 3.81, 3.04 [03:12:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.45, 11.96, 9.88 [03:13:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 3.29, 2.95 [03:14:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.60, 11.86, 10.10 [03:17:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.96, 3.38, 3.14 [03:18:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.29, 9.67, 9.61 [03:20:18] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.26, 2.31, 1.88 [03:21:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.78, 3.92, 3.45 [03:21:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.43, 11.72, 11.17 [03:25:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.13, 4.50, 3.72 [03:25:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.38, 12.61, 11.62 [03:28:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.13, 11.82, 11.52 [03:30:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.35, 3.70, 3.66 [03:32:56] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.06, 11.12, 11.24 [03:34:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.57, 2.91, 3.34 [03:36:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.17, 11.23, 11.29 [03:39:17] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.34, 10.81, 7.52 [03:40:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.37, 1.89, 1.99 [03:40:50] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.06, 11.51, 11.41 [03:41:17] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.45, 9.07, 7.29 [03:42:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.99, 10.48, 9.25 [03:42:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.07, 11.06, 11.27 [03:44:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.23, 9.69, 9.11 [03:44:07] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.17, 1.96, 1.99 [03:47:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.03, 1.61, 1.96 [03:48:05] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.63, 1.85, 1.94 [03:48:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.00, 11.56, 10.02 [03:49:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.65, 2.08, 2.09 [03:50:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.26, 10.16, 9.70 [03:50:05] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.91, 2.31, 2.10 [03:54:03] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.27, 1.87, 1.98 [03:54:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.05, 8.80, 10.12 [04:00:00] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.82, 2.04, 1.97 [04:01:59] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.92, 2.00, 1.97 [04:02:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [04:06:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:09:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.47, 1.83, 1.98 [04:13:54] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.99, 1.38, 1.64 [04:21:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.01, 1.75, 1.82 [04:25:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.40, 1.78, 1.83 [04:29:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.16, 1.45, 1.69 [04:33:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 7.92, 5.79, 3.38 [04:51:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [04:52:32] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.06, 2.01, 1.65 [04:54:31] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.06, 1.67, 1.57 [05:04:37] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.49, 3.44, 2.71 [05:06:37] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.50, 3.05, 2.65 [05:10:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.58, 2.10, 1.69 [05:12:38] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:13:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.39, 1.63, 1.97 [05:15:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.88, 1.87, 2.02 [05:16:20] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.57, 1.94, 1.77 [05:17:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.08, 1.52, 1.87 [05:19:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.37, 1.74, 1.90 [05:21:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.50, 1.59, 1.82 [05:25:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.59, 1.77, 1.82 [05:27:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 1.80, 1.82 [05:28:15] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.29, 2.17, 1.89 [05:28:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [05:29:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.79, 2.07, 1.91 [05:31:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.72, 1.93, 1.88 [05:34:12] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.96, 1.77, 1.84 [05:36:11] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.59, 1.39, 1.70 [05:41:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.22, 1.53, 1.68 [05:46:59] [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/e9a90c7d9714...5afdda732657 [05:46:59] [url] Comparing e9a90c7d9714...5afdda732657 · miraheze/MirahezeMagic · GitHub | github.com [05:47:00] [02miraheze/MirahezeMagic] 07Reception123 035afdda7 - fix typo [05:50:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.33, 2.39, 1.97 [05:51:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.83, 2.08, 1.88 [05:53:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.85, 1.75, 1.79 [05:54:59] miraheze/MirahezeMagic - Reception123 the build passed. [05:56:59] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.04, 1.47, 1.68 [05:58:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.79, 1.98, 1.98 [06:04:23] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.42, 1.70 [06:04:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:14:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [06:21:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.50, 2.43, 1.90 [06:27:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.27, 4.15, 2.80 [06:29:13] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.73, 1.49, 1.07 [06:30:50] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 4.21, 2.95 [06:31:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.86, 2.99 [06:31:13] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.22, 1.40, 1.09 [06:32:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.53, 3.67, 2.91 [06:33:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.36, 1.90, 1.98 [06:34:40] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.73, 1.79, 1.59 [06:35:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.47, 3.22, 2.94 [06:35:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.62, 2.14, 2.05 [06:36:39] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.16, 1.57, 1.53 [06:36:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.06, 3.19, 2.93 [06:37:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.82, 1.98, 2.00 [06:39:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.83, 2.23, 2.09 [06:48:32] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.07, 1.88, 1.66 [06:50:31] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.98, 1.98, 1.73 [06:52:31] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.20, 2.17, 1.84 [06:54:30] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.56, 1.89, 1.77 [06:54:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:54:38] PROBLEM - mon141 Puppet on mon141 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[python-is-python3],Package[python] [06:56:29] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.39, 2.17, 1.89 [06:59:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.35, 1.68, 1.97 [07:01:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.53, 1.96, 2.04 [07:02:26] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.28, 1.88, 1.87 [07:03:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.98, 1.56, 1.88 [07:04:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.56, 2.05, 1.93 [07:04:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [07:07:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.19, 2.07, 2.00 [07:11:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.70, 1.90, 1.95 [07:12:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.07, 1.75, 1.90 [07:16:20] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.35, 1.86, 1.89 [07:19:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.39, 2.04, 1.97 [07:21:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.64, 1.90, 1.93 [07:22:37] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:24:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.04, 1.74, 1.91 [07:27:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.92, 1.38, 1.69 [07:28:14] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.77, 2.18, 2.04 [07:30:13] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.22, 1.80, 1.92 [07:40:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.01, 1.68, 1.75 [07:42:08] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.89, 1.83, 1.80 [07:50:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:52:03] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.07, 1.99, 1.83 [07:54:12] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/m.miraheze.org.key] [07:55:19] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/m.miraheze.org.key] [07:56:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.25, 1.65, 1.73 [07:56:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [07:58:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.08, 1.53, 1.68 [08:11:07] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e66c70b28fd5...7c146ffde81e [08:11:08] [url] Comparing e66c70b28fd5...7c146ffde81e · miraheze/ssl · GitHub | github.com [08:11:09] [02miraheze/ssl] 07Reception123 037c146ff - Update m.miraheze.org.crt [08:13:51] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.04, 1.84, 1.72 [08:15:10] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=rosettacodewiki --no-updates --username-prefix=rosettacode /home/reception/rosettacode.xml --report 1 (START) [08:15:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:15:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.59, 1.83, 1.73 [08:17:53] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 19.33, 13.79, 8.41 [08:19:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.84, 2.04, 1.82 [08:19:50] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 7.07, 11.03, 8.03 [08:20:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.85, 14.21, 9.14 [08:21:46] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 4.97, 8.86, 7.60 [08:22:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.08, 11.41, 8.73 [08:22:49] !log [salt-user@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=rosettacodewiki (END - exit=65280) [08:22:51] !log [salt-user@mw131] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=rosettacodewiki (END - exit=0) [08:22:52] !log [salt-user@mw142] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=rosettacodewiki (END - exit=0) [08:22:53] !log [salt-user@mw132] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=rosettacodewiki (END - exit=0) [08:22:54] !log [salt-user@mw121] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=rosettacodewiki (END - exit=0) [08:22:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:22:55] !log [salt-user@mw141] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=rosettacodewiki (END - exit=0) [08:22:56] !log [salt-user@mw122] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=rosettacodewiki (END - exit=0) [08:23:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:23:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:23:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:23:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:23:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:23:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:24:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.69, 9.46, 8.34 [08:26:32] Reception123: you do not need need to run that on all servers anymore... [08:27:27] CosmicAlpha: ah, the message needs to be updated then [08:27:32] where does it have to be run then? just mwtask*? [08:27:43] also I fixed that typo with 'accordance' on Discord that I saw [08:28:43] Reception123: Just need to run `mwscript extensions/SemanticMediaWiki/setupStore.php wiki && mwscript extensions/SemanticMediaWiki/setupStore.php wiki [08:28:43] deploy-mediawiki --files=../mediawiki/w/extensions/SemanticMediaWiki/.smw.json --servers=all` on mwtask141 (sometimes twice on the one server) yeah... [08:29:15] I did not update the message because it is a much longer command that does not fit quite right there. [08:29:23] And yeah saw that. Thanks! [08:29:58] ah I see [08:31:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.30, 1.95, 1.95 [08:31:58] Reception123: I might make a shell script or something to deploy it, so that it is easily ran, and the command can fit better on-wiki. [08:32:47] yeah that might be easier than having that long command [08:33:46] Reception123: yeah... I might try to add one to https://github.com/miraheze/puppet/tree/master/modules/mediawiki/files/cookbooks when I have time. [08:33:47] [url] puppet/modules/mediawiki/files/cookbooks at master · miraheze/puppet · GitHub | github.com [08:35:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.22, 1.88, 1.90 [08:36:29] sounds good :) [08:37:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.25, 1.55, 1.77 [08:38:40] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.19, 1.72, 1.52 [08:39:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.17, 1.40, 1.69 [08:40:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:40:39] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.70, 1.89, 1.62 [08:42:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.44, 2.01, 1.69 [08:46:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.21, 1.74, 1.67 [08:48:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [08:50:34] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.33, 1.63, 1.64 [08:53:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.35, 2.01, 1.80 [08:55:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.78, 1.94, 1.80 [08:57:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.75, 1.53, 1.67 [09:05:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.10, 1.83, 1.71 [09:09:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.59, 1.71, 1.69 [09:11:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.96, 1.44, 1.59 [09:20:20] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.50, 1.72, 1.54 [09:23:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.93, 2.37, 1.89 [09:26:17] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.47, 2.06, 1.75 [09:28:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.83, 1.95, 1.74 [09:29:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.51, 1.96, 1.88 [09:30:15] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.05, 2.00, 1.79 [09:33:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.80, 1.37, 1.66 [09:34:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:40:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [09:42:10] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.99, 1.91, 1.96 [09:50:06] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.13, 1.82, 1.84 [09:51:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.19, 1.68, 1.48 [09:54:04] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.26, 1.62, 1.76 [09:56:03] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.38, 1.81, 1.80 [09:58:03] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.72, 1.66, 1.74 [09:59:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.52, 2.00, 1.82 [10:01:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.08, 2.11, 1.88 [10:05:59] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.61, 1.89, 1.80 [10:07:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.41, 1.95, 1.90 [10:09:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.10, 2.16, 1.99 [10:09:57] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.68, 1.78, 1.78 [10:15:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.10, 1.46, 1.64 [10:22:50] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.28, 2.02, 1.82 [10:26:38] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:29:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.56, 1.80, 1.98 [10:30:46] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.60, 1.98, 1.94 [10:31:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.00, 1.98, 2.04 [10:32:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [10:35:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.46, 1.90, 2.00 [10:36:43] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.94, 1.31, 1.65 [10:43:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.00, 1.35, 1.68 [10:47:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.63, 1.94, 1.86 [10:50:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.49, 1.88, 1.67 [10:51:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.89, 1.93, 1.87 [10:52:35] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.85, 1.48, 1.55 [10:55:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.16, 1.93, 1.88 [10:57:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.25, 1.66, 1.78 [11:05:28] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.44, 1.78, 1.64 [11:06:04] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [11:07:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.07, 1.95, 1.87 [11:09:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.56, 1.77, 1.80 [11:11:26] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.07, 1.48, 1.57 [11:17:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.76, 1.84, 1.79 [11:18:38] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:21:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.61, 1.72, 1.65 [11:23:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.13, 1.53, 1.59 [11:24:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [11:27:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.44, 1.88, 1.71 [11:27:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.89, 1.95, 1.90 [11:29:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.77, 1.79, 1.70 [11:31:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.35, 2.18, 1.86 [11:31:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.22, 1.95, 1.91 [11:33:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.20, 1.81, 1.76 [11:35:24] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp31.miraheze.org - NS RECORDS OK [11:35:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.99, 1.83, 1.86 [11:37:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.23, 1.58, 1.68 [11:39:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 9.07, 3.38, 2.35 [12:03:26] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=rosettacodewiki --no-updates --username-prefix=rosettacode /home/reception/rosettacode.xml --report 1 (END - exit=0) [12:03:27] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=rosettacodewiki (START) [12:03:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:03:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:10:10] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.23, 1.91, 1.66 [12:12:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:14:08] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.54, 1.87, 1.70 [12:16:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [12:17:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.85, 10.71, 9.77 [12:18:06] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.36, 2.22, 1.84 [12:19:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.20, 11.00, 9.97 [12:24:03] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.29, 1.74, 1.78 [12:24:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.86, 10.17, 9.02 [12:25:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.55, 11.83, 10.73 [12:28:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.80, 10.08, 9.28 [12:31:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.95, 12.05, 11.10 [12:33:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.39, 1.73, 1.93 [12:33:59] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.91, 1.39, 1.61 [12:36:01] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:42:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.01, 10.49, 9.63 [12:43:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.38, 11.25, 11.89 [12:44:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.06, 10.06, 9.58 [12:47:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.56, 12.74, 12.18 [12:49:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.76, 2.12, 1.90 [12:49:50] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 6.31, 3.31, 2.19 [12:51:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.80, 1.90, 1.84 [12:51:48] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 6.26, 5.37, 4.47 [12:52:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.02, 11.89, 10.47 [12:54:23] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.98, 3.34, 2.59 [12:56:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.73, 3.72, 2.82 [12:56:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.29, 2.47 [12:57:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.12, 1.49, 1.70 [13:00:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.89, 4.13, 2.97 [13:02:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.63, 2.93 [13:02:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:03:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.95, 3.42 [13:04:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.12, 11.31, 11.29 [13:05:21] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp31.miraheze.org - NS RECORDS OK [13:06:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.90, 3.19, 2.91 [13:07:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.01, 3.39, 3.31 [13:10:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.95, 11.23, 11.17 [13:10:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [13:14:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.35, 11.47, 11.39 [13:16:09] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=rosettacodewiki (END - exit=0) [13:16:10] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=rosettacodewiki --active --update (END - exit=0) [13:16:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:16:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:18:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.40, 11.32, 11.29 [13:20:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.55, 10.80, 11.09 [13:22:26] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.78, 2.58, 1.97 [13:23:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.94, 3.57, 3.02 [13:24:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.59, 3.15 [13:24:57] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.57, 4.57, 5.81 [13:25:26] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.56, 2.77, 2.79 [13:25:33] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.04, 1.62, 1.92 [13:26:06] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.41, 3.20, 3.07 [13:26:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.48, 9.63, 11.53 [13:26:47] Reception123: I see you updated that cert again, though puppet is still failing [13:26:53] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (1 critical updates). [13:28:57] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.94, 3.37, 5.05 [13:30:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.41, 8.63, 9.99 [13:30:17] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.63, 1.99, 1.95 [13:31:31] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.16, 1.36, 1.70 [13:32:14] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.91, 2.06, 1.99 [13:35:28] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.99, 1.96, 1.87 [13:37:27] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.46, 2.33, 2.03 [13:39:26] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.04, 1.95, 1.93 [13:41:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.14, 2.13, 2.01 [13:42:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.55, 9.10, 10.17 [13:43:24] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.41, 1.84, 1.91 [13:44:01] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.91, 1.98 [13:45:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.06, 1.93, 1.93 [13:45:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.09, 2.27, 2.10 [13:47:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.75, 1.85, 1.90 [13:49:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.42, 2.06, 1.97 [13:55:52] MacFan4000: yeah, not sure what more I can do [13:56:01] I revoked the old one and the private key now is completely different [13:56:12] It didn't ask for challenges so if you want you could try as well [13:56:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:59:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.70, 9.58, 9.57 [14:02:38] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [14:03:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.48, 1.87, 1.98 [14:05:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.53, 2.02, 2.01 [14:05:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.30, 11.07, 10.25 [14:07:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.28, 10.98, 10.31 [14:07:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.33, 1.80, 1.94 [14:09:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.23, 11.94, 10.75 [14:11:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.76, 10.79, 10.48 [14:11:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.63, 1.99, 1.97 [14:16:24] CosmicAlpha: ^ ping again, any ideas for the issue with ^ [14:17:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.85, 1.97, 1.97 [14:19:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.09, 1.92, 1.94 [14:23:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.56, 10.74, 10.53 [14:23:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.54, 1.88, 1.94 [14:25:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.94, 10.75, 10.58 [14:25:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.25, 2.04, 1.99 [14:25:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.09, 2.68 [14:26:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.27, 10.36, 9.04 [14:27:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.15, 11.81, 10.95 [14:27:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.85, 3.92, 3.03 [14:28:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.23, 10.94, 9.41 [14:29:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.26, 1.64, 1.84 [14:30:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.75, 11.05, 9.61 [14:30:58] Reception123: I'm looking now, and it is asking me to deploy a txt reccord [14:31:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.70, 11.31, 11.03 [14:31:21] for reference the command im using is as follows [14:31:22] sudo /root/ssl-certificate -d m.miraheze.org -s *.m.miraheze.org -r -w -o [14:32:11] hmm, I only did -d as CosmicAlpha suggested that [14:32:18] MacFan4000: you can give me the TXT record and I can deploy now [14:32:26] you have to use -w for wildcards [14:32:41] -o just outputs the new public key [14:33:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.32, 12.07, 11.29 [14:33:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.87, 3.91, 3.35 [14:33:39] what I did afaik was sudo /root/ssl-certificate -d *.m.miraheze.org -g -w -o -p [14:34:24] -g generates a completly new cert, -r just renews [14:34:28] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/53605a58655f...0336e97e8e11 [14:34:28] [url] Comparing 53605a58655f...0336e97e8e11 · miraheze/dns · GitHub | github.com [14:34:29] [02miraheze/dns] 07Reception123 030336e97 - new acme challenge for *.m.miraheze.org [14:35:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 4.27, 3.55 [14:35:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.15, 4.01, 3.03 [14:35:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.19, 1.47, 1.70 [14:36:01] MacFan4000: yeah, though after -r didn't work the first time I used --revoke and then tried to generate a new one with -g [14:36:04] TXT deployed to ns* [14:37:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.36, 3.54, 2.99 [14:37:55] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7c146ffde81e...ad4bc271ab2d [14:37:56] [url] Comparing 7c146ffde81e...ad4bc271ab2d · miraheze/ssl · GitHub | github.com [14:37:57] [02miraheze/ssl] 07MacFan4000 03ad4bc27 - Update m.miraheze.org.crt [14:38:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.12, 9.22, 9.49 [14:38:45] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:38:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.68, 10.38, 11.01 [14:39:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.61, 3.18, 2.92 [14:39:11] Reception123: If I was able to i would verify that the private key in /home/ssl-admins/ssl-keys (that is commited and pushed) matches the one is /etc/letsencrypt/live [14:39:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.21, 1.71, 1.77 [14:39:32] in* [14:40:06] I cant though as /etc/letsencrypt/live is not readable for me [14:40:45] I'll look [14:40:49] but for the first cert I generated it did [14:41:52] actually the proper private key doesn't even exist in /home [14:42:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.06, 12.54, 10.82 [14:42:27] MacFan4000: the two keys do match [14:42:30] for *.m.miraheze.org [14:42:30] *.m.miraheze.org.key exists but for a public key named m.miraheze.org.crt that is incorrect [14:42:46] Ah, I guess that's where things went wrong [14:42:51] the key should be called m.miraheze.org.key [14:43:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 3.82, 3.66 [14:43:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.37, 1.88, 1.82 [14:43:41] anyway, the public key name is correct (with root subdomain as primary and wildcard as seccondary) [14:44:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.91, 11.10, 10.49 [14:44:21] Reception123: you'll need to rename the private key, commit, and push [14:45:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.89, 4.09, 3.76 [14:45:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.15, 1.66, 1.75 [14:45:46] MacFan4000: ah so you mean the current key that's *.m.miraheze.org.key is actually the one for an invalid m.miraheze.org cert? [14:46:00] but if that's the case then why does /etc/letsencrypt/live/*.m.miraheze.org match? [14:47:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.76, 3.68 [14:47:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.47, 1.98, 1.86 [14:47:37] the deployed private key needs to be called m.miraheze.org.key to match the naming of the public key, and it needs to be the private key that is in /etc/letsencrypt/live/m.miraheze.org [14:48:01] the certificate name is also the primary domain for the certificate [14:48:26] the wildcard portion was set as the secondary using -s [14:49:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.92, 3.89, 3.72 [14:49:25] in this case the cert is valid both for m.miraheze.org and *.m.miraheze.org [14:50:48] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.17, 9.10, 10.09 [14:51:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.57, 3.61 [14:52:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.63, 12.05, 10.98 [14:53:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.06, 5.36, 4.26 [14:54:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 24.55, 16.54, 12.81 [14:56:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.85, 3.51 [14:56:41] Reception123: I would guess you originally specified the domains incorrectly when first attempting a renewal [14:58:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.39, 3.37, 3.37 [14:59:04] MacFan4000: would it be easier if I just completely removed both private keys (*.m and m.) and you tried to generate again? [14:59:09] that way there can be no more confusion [14:59:52] yeah lets do that [15:00:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.83, 11.68, 11.64 [15:02:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.05, 11.84, 11.70 [15:02:32] Reception123: ^ [15:02:34] MacFan4000: there was no m.miraheze.org but I've now removed *.m.miraheze.org and pushed the changs [15:02:36] *changes [15:02:40] (the key) [15:02:41] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:51] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 4.725 second response time [15:03:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.62, 3.88 [15:03:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.17, 1.59, 1.94 [15:03:28] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:01] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ad4bc271ab2d...2b4a0bb3bfec [15:04:02] [url] Comparing ad4bc271ab2d...2b4a0bb3bfec · miraheze/ssl · GitHub | github.com [15:04:03] [02miraheze/ssl] 07MacFan4000 032b4a0bb - Update m.miraheze.org.crt [15:04:26] Reception123: "Reusing existing private key from /etc/letsencrypt/live/m.miraheze.org/privkey.pem." [15:04:36] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.318 second response time [15:04:45] you will need to commit the one from that location [15:04:50] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.340 second response time [15:05:06] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw122 [15:05:21] MacFan4000: so the m.miraheze.org not the *.m.miraheze.org? [15:05:24] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.411 second response time [15:05:33] Reception123: correct [15:06:45] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:07] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [15:08:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:08:47] done [15:09:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.95, 3.55, 3.72 [15:09:13] Reception123: great, now we wait for puppet to run and deploy everything [15:11:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.81, 2.18, 2.00 [15:12:33] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 22 Nov 2022 14:02:45 GMT +0000. [15:12:37] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [15:13:22] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:27] so yeah if -d is ever specified incorrectly then unless you update the private key and rename that and the public key, youll have issues such as that mismatch [15:14:06] because if -d is incorrect then a new cert would be generated [15:14:25] instead of renewing the existing one [15:14:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [15:14:47] i would always double check the cert name, and see what domains the cert is valid for [15:15:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.82, 4.05, 3.46 [15:17:11] Yeah let's hope it works [15:17:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.42, 3.46, 3.31 [15:18:37] the icinga alerts are resolved now, so it worked [15:19:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.54, 3.35 [15:20:44] PROBLEM - cp31 APT on cp31 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (1 critical updates). [15:21:26] Reception123: I only suggested only *.miraheze.org because I thought you said the other way did not work. I must have misunderstood. Sorry. [15:22:36] PROBLEM - cp30 APT on cp30 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (1 critical updates). [15:23:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.48, 3.37 [15:24:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.60, 10.99, 11.92 [15:25:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.68, 4.46, 3.72 [15:28:40] PROBLEM - es131 APT on es131 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (1 critical updates). [15:31:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.78, 3.89, 3.78 [15:33:34] ^ yes apt makes up the bulk of our critical alerts currently [15:34:14] 35 alerts for apt [15:36:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.06, 10.05, 10.63 [15:38:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.04, 9.44, 10.35 [15:39:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.95, 3.47, 4.00 [15:40:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 18.04, 11.75, 11.02 [15:41:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.59, 3.74, 4.05 [15:41:52] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.48, 2.87, 3.35 [15:45:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.51, 3.48, 3.91 [15:46:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.27, 11.41, 11.45 [15:49:24] MacFan4000: I plan to deal with some of those later today also. [15:52:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.72, 10.97, 12.00 [15:54:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.68, 11.76, 12.14 [15:56:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.04, 8.42, 9.83 [15:56:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.70, 10.58, 11.64 [15:57:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.02, 3.01, 3.37 [15:59:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 3.51, 3.14 [16:01:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.13, 2.97, 2.99 [16:02:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:04:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.46, 10.84, 11.26 [16:06:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.59, 10.32, 11.02 [16:08:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [16:14:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.50, 8.85, 10.08 [16:33:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.64, 1.77, 1.99 [16:33:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.18, 1.70, 2.00 [16:37:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.50, 2.03, 2.05 [16:39:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.19, 1.79, 1.97 [16:41:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.12, 1.92, 1.99 [16:45:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.78, 1.82, 1.93 [16:47:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.27, 1.97, 1.90 [16:49:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.61, 1.85, 1.87 [16:53:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.21, 1.82, 1.83 [16:53:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.23, 1.99, 1.96 [16:57:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.92, 1.83, 1.82 [16:57:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.85, 1.93, 1.94 [16:59:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.24, 2.25, 2.04 [17:07:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.37, 1.48, 1.67 [17:22:17] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.81, 2.47, 1.94 [17:24:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.70, 4.28, 2.97 [17:25:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.73, 4.43, 2.85 [17:28:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.33, 12.32, 10.30 [17:30:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.30, 11.86, 10.37 [17:35:56] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.04, 9.93, 10.05 [17:36:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.40, 1.80, 1.97 [17:36:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.83, 3.38, 3.36 [17:37:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.75, 3.30, 3.32 [17:38:10] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.24, 1.89, 1.98 [17:40:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.49, 1.71, 1.90 [17:44:07] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.41, 1.79, 1.88 [17:45:53] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.91, 1.75, 0.98 [17:46:06] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.89, 1.51, 1.77 [17:46:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:47:53] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.98, 1.80, 1.10 [17:48:05] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.98, 1.34, 1.68 [17:48:08] RECOVERY - phab121 APT on phab121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [17:49:53] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.30, 1.91, 1.22 [17:51:49] !log apt upgrade and reboot phab121 (includes kernel upgrade) [17:51:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:52:02] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.92, 2.29, 1.95 [17:52:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [17:53:20] RECOVERY - matomo131 APT on matomo131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [17:53:53] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 0.89, 0.85, 0.37 [17:55:01] !log apt upgrade and reboot matomo131 (includes kernel upgrade) [17:55:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:56:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.46, 1.94, 1.88 [17:59:47] PROBLEM - reports121 Puppet on reports121 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[logrotate],Package[nano] [17:59:59] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.37, 1.94, 1.89 [18:00:04] PROBLEM - reports121 Current Load on reports121 is WARNING: WARNING - load average: 1.94, 1.26, 0.60 [18:00:08] RECOVERY - reports121 APT on reports121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:01:58] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.28, 1.71, 1.81 [18:03:23] !log apt upgrade and reboot reports121 (includes kernel upgrade) [18:03:45] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [18:03:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:04:04] RECOVERY - reports121 Current Load on reports121 is OK: OK - load average: 0.39, 0.38, 0.16 [18:06:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.42, 3.21 [18:06:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.85, 3.46, 2.90 [18:07:55] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.33, 1.97, 1.86 [18:10:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 10.23, 5.31, 3.90 [18:12:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.97, 3.64, 3.25 [18:12:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.41, 10.27, 9.60 [18:13:53] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.11, 1.64, 1.79 [18:14:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.62, 3.02, 3.07 [18:15:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.11, 3.35, 3.51 [18:16:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.14, 9.32, 9.37 [18:17:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.51, 2.76, 3.28 [18:19:50] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.12, 1.84, 1.83 [18:21:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.37, 1.68, 1.78 [18:26:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.60, 10.11, 9.64 [18:27:58] [02miraheze/dns] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+1/-0/±0] 13https://github.com/miraheze/dns/commit/9cb6a180fe67 [18:27:59] [02miraheze/dns] 07Reception123 039cb6a18 - add rosettacode.org zone [18:28:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.91, 12.08, 10.45 [18:28:01] [02dns] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/dns [18:28:01] [url] Page not found · GitHub · GitHub | github.com [18:28:02] [02dns] 07Reception123 opened pull request 03#335: add rosettacode.org zone - 13https://github.com/miraheze/dns/pull/335 [18:28:03] [url] Page not found · GitHub · GitHub | github.com [18:28:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.91, 10.61, 9.71 [18:29:46] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.97, 2.04, 1.89 [18:30:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.90, 11.61, 10.49 [18:30:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.76, 9.27, 9.33 [18:31:30] [02dns] 07Reception123 closed pull request 03#335: add rosettacode.org zone - 13https://github.com/miraheze/dns/pull/335 [18:31:31] [url] Page not found · GitHub · GitHub | github.com [18:31:32] [02miraheze/dns] 07Reception123 deleted branch 03Reception123-patch-1 [18:31:33] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/0336e97e8e11...d612c12c112d [18:31:34] [url] Comparing 0336e97e8e11...d612c12c112d · miraheze/dns · GitHub | github.com [18:31:35] [02miraheze/dns] 07Reception123 03d612c12 - add rosettacode.org zone (#335) [18:31:36] [02dns] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/dns [18:31:37] [url] Page not found · GitHub · GitHub | github.com [18:31:45] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.88, 2.00, 1.90 [18:32:14] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/d612c12c112d...d48ea130e71c [18:32:15] [url] Comparing d612c12c112d...d48ea130e71c · miraheze/dns · GitHub | github.com [18:32:16] [02miraheze/dns] 07Reception123 03d48ea13 - fix alignment [18:34:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.25, 9.95, 10.07 [18:35:36] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [18:35:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [18:37:42] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.75, 1.87, 1.83 [18:38:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 20.99, 14.96, 12.01 [18:38:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.68, 6.03, 3.92 [18:38:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:39:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.66, 7.02, 4.60 [18:39:06] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/d48ea130e71c...e7b40ab3268e [18:39:07] [url] Comparing d48ea130e71c...e7b40ab3268e · miraheze/dns · GitHub | github.com [18:39:08] [02miraheze/dns] 07Reception123 03e7b40ab - fix [18:39:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.79, 12.06, 10.47 [18:44:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [18:45:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.59, 11.17, 10.68 [18:47:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.16, 11.53, 10.86 [18:49:19] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/e7b40ab3268e...4e5daf1b4125 [18:49:20] [url] Comparing e7b40ab3268e...4e5daf1b4125 · miraheze/dns · GitHub | github.com [18:49:21] [02miraheze/dns] 07Reception123 034e5daf1 - attempt fix for rosettacode.org cname [18:49:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.25, 10.45, 10.53 [18:49:36] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:50:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.62, 3.67 [18:51:27] paladox: around? [18:51:29] (nothing urgent) [18:52:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.70, 10.59, 11.63 [18:53:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.14, 3.37, 3.94 [18:53:36] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [18:54:05] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.40, 2.79, 3.30 [18:55:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:59:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.97, 4.51, 4.19 [19:01:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.34, 10.88, 10.69 [19:03:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.84, 3.98 [19:03:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.05, 10.70, 10.66 [19:04:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.92, 11.53, 11.01 [19:04:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.60, 4.46, 3.66 [19:04:13] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [19:04:51] Reception123: semi [19:05:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.41, 4.14, 4.11 [19:05:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.26, 13.65, 11.79 [19:06:21] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/4e5daf1b4125...2bb2353f4c9e [19:06:21] [url] Comparing 4e5daf1b4125...2bb2353f4c9e · miraheze/dns · GitHub | github.com [19:06:22] [02miraheze/dns] 07Universal-Omega 032bb2353 - Attempt fix for rosettacode.org MX record [19:07:35] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:18:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:19:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.80, 3.43, 3.91 [19:21:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.50, 4.39, 4.21 [19:21:54] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/86155b721bad...c6856e9317de [19:21:55] [url] Comparing 86155b721bad...c6856e9317de · miraheze/puppet · GitHub | github.com [19:21:56] [02miraheze/puppet] 07Universal-Omega 03c6856e9 - graylog: upgrade to 4.3.5 [19:22:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [19:23:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.80, 3.81, 4.00 [19:25:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.18, 4.13, 4.11 [19:33:48] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 7.59, 4.44, 1.82 [19:36:15] RECOVERY - graylog121 APT on graylog121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:37:42] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.19, 4.62, 2.56 [19:39:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.59, 2.78, 3.96 [19:40:35] !log apt dist-upgrade and reboot graylog121 (includes kernel upgrade) [19:40:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:41:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.14, 2.91, 3.84 [19:41:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.12, 3.26, 4.02 [19:43:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.62, 2.82, 3.78 [19:45:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.71, 4.06, 4.15 [19:47:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.12, 3.53, 3.94 [19:49:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.39, 2.54, 3.30 [19:49:49] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/ssl/compare/2b4a0bb3bfec...c955b034c7b9 [19:49:50] [url] Comparing 2b4a0bb3bfec...c955b034c7b9 · miraheze/ssl · GitHub | github.com [19:49:51] [02miraheze/ssl] 07Reception123 03c955b03 - add rosettacode.org cert [19:50:59] miraheze/ssl - Reception123 the build has errored. [19:52:09] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/2bb2353f4c9e...e2d10abe148c [19:52:10] [url] Comparing 2bb2353f4c9e...e2d10abe148c · miraheze/dns · GitHub | github.com [19:52:11] [02miraheze/dns] 07Reception123 03e2d10ab - add apuntes.tk zone [19:53:51] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.87, 2.50, 3.35 [19:54:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:54:46] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://github.com/miraheze/ssl/compare/c955b034c7b9...982863b7c151 [19:54:47] [url] Comparing c955b034c7b9...982863b7c151 · miraheze/ssl · GitHub | github.com [19:54:48] [02miraheze/ssl] 07Reception123 03603d823 - add wiki.ultravanilla.world cert [19:54:49] [02miraheze/ssl] 07Reception123 03982863b - add apuntes.tk cert [19:55:53] miraheze/ssl - Reception123 the build has errored. [19:57:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.18, 4.65, 3.84 [19:57:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.83, 3.69, 3.63 [19:58:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [20:01:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.80, 3.69 [20:02:53] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp31.miraheze.org - NS RECORDS OK [20:03:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.45, 3.96, 3.87 [20:03:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 4.26, 3.89 [20:07:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.98, 4.13, 3.93 [20:07:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.73, 1.74, 1.99 [20:07:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.80, 3.79 [20:09:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.89, 3.63, 3.77 [20:11:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.36, 1.90, 1.98 [20:13:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.40, 3.47, 3.56 [20:15:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.30, 3.48 [20:17:03] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/982863b7c151...b35c4c333117 [20:17:03] [url] Comparing 982863b7c151...b35c4c333117 · miraheze/ssl · GitHub | github.com [20:17:04] [02miraheze/ssl] 07Reception123 03b35c4c3 - EOL [20:17:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.57, 2.65, 3.22 [20:18:16] miraheze/ssl - Reception123 the build has errored. [20:21:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.77, 4.01, 3.79 [20:21:56] ^ CosmicAlpha [20:23:13] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b35c4c333117...aa4fe55621de [20:23:13] [url] Comparing b35c4c333117...aa4fe55621de · miraheze/ssl · GitHub | github.com [20:23:14] [02miraheze/ssl] 07Universal-Omega 03aa4fe55 - fix [20:23:19] Reception123: ^ [20:24:14] CosmicAlpha: I only added the extra line after the first one failed [20:24:41] Reception123: because you had the first one with whitespaces, not an empty line. [20:25:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.58, 3.61, 3.70 [20:25:10] But that does not fail syntax. [20:27:07] Ah [20:29:01] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:34:03] Is there a reason we use crons for pulling ssl and puppet itself, why can the crons not be removed, and https://github.com/miraheze/puppet/blob/9e9b530b340c32c188fa38ff4021846f7f960c47/modules/puppetserver/manifests/init.pp#L69 and https://github.com/miraheze/puppet/blob/9e9b530b340c32c188fa38ff4021846f7f960c47/modules/puppetserver/manifests/init.pp#L76 made to be `ensure => latest`? [20:34:04] [url] puppet/init.pp at 9e9b530b340c32c188fa38ff4021846f7f960c47 · miraheze/puppet · GitHub | github.com [20:34:04] [url] puppet/init.pp at 9e9b530b340c32c188fa38ff4021846f7f960c47 · miraheze/puppet · GitHub | github.com [20:36:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.80, 2.73, 3.29 [20:38:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [20:42:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:46:43] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [20:57:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.84, 10.65, 11.96 [20:59:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.82, 12.09, 12.35 [21:02:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.86, 3.65, 3.27 [21:03:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.55, 3.26, 3.17 [21:05:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.86, 11.23, 11.90 [21:07:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.27, 12.05, 12.14 [21:21:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.38, 3.88, 3.25 [21:22:22] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:22:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.98, 10.81, 11.84 [21:23:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.94, 3.07, 3.03 [21:32:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.84, 9.42, 11.83 [21:32:22] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [21:36:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.37, 8.50, 9.98 [21:38:22] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:40:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.42, 11.65, 10.85 [21:41:59] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.65, 1.71, 1.92 [21:42:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.27, 10.85, 10.67 [21:43:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.81, 2.41, 2.14 [21:44:22] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.00, 8.75, 10.19 [21:46:22] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [21:46:51] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.18, 9.27, 10.06 [21:48:50] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.91, 3.75, 2.28 [21:50:50] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.59, 3.31, 2.30 [21:57:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.57, 1.74, 1.94 [22:01:44] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.96, 2.12, 2.01 [22:03:12] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:04:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.05, 11.08, 10.44 [22:10:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.05, 11.39, 10.76 [22:12:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.79, 10.07, 10.35 [22:14:22] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:14:51] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.05, 9.45, 10.09 [22:18:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.37, 12.99, 10.45 [22:20:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.27, 11.43, 10.19 [22:22:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.54, 12.25, 10.63 [22:24:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.14, 11.48, 10.55 [22:26:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.74, 11.74, 10.73 [22:28:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.13, 11.68, 10.85 [22:30:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.65, 12.77, 11.33 [22:31:55] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp31.miraheze.org - NS RECORDS OK [22:38:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.54, 11.79, 11.65 [22:40:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.59, 12.01, 11.74 [22:41:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.74, 10.41, 9.48 [22:42:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.03, 11.85, 11.73 [22:43:20] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.69, 9.88, 9.38 [22:50:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.37, 12.31, 11.81 [22:54:57] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.39, 3.50, 2.56 [22:59:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.98, 12.06, 10.42 [23:00:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.54, 2.86 [23:01:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.61, 3.15 [23:02:33] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [23:02:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.09, 3.23, 2.82 [23:03:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.04, 11.54, 10.62 [23:03:49] RECOVERY - prometheus131 APT on prometheus131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:05:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.15, 2.89, 2.97 [23:06:59] !log apt dist-upgrade and reboot prometheus131 (includes kernel upgrade) [23:07:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:07:58] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.17, 1.55, 1.96 [23:09:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.07, 12.00, 11.05 [23:09:38] RECOVERY - mw122 APT on mw122 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:10:03] RECOVERY - mw141 APT on mw141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:10:38] RECOVERY - mw142 APT on mw142 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:10:47] !log apt dist-upgrade and reboot test131 (includes kernel upgrade) [23:10:50] RECOVERY - mwtask141 APT on mwtask141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:10:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:11:08] RECOVERY - mw132 APT on mw132 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:11:13] RECOVERY - mw121 APT on mw121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:11:25] RECOVERY - test131 APT on test131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:11:26] !log apt dist-upgrade and reboot mwtask141 (includes kernel upgrade) [23:11:29] RECOVERY - mw131 APT on mw131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:11:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.75, 11.45, 10.99 [23:11:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:11:58] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.37, 1.64, 1.89 [23:12:19] !log apt dist-upgrade on mw* (includes kernel upgrades on mw131, mw132, mw141, and mw142) [23:12:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:13:58] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.66, 1.79, 1.92 [23:14:34] PROBLEM - test131 Check Gluster Clients on test131 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [23:15:06] PROBLEM - mwtask141 Check Gluster Clients on mwtask141 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [23:15:21] !log depool, reboot, and repool mw131 [23:15:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:17:08] !log depool, reboot, and repool mw132 [23:17:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:17:58] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.47, 1.88, 1.91 [23:18:28] !log depool, reboot, and repool mw141 [23:18:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.80, 12.09, 11.24 [23:19:45] !log depool, reboot, and repool mw142 [23:19:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:23:15] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/6400484cae87 [23:23:17] [02miraheze/dns] 07Universal-Omega 036400484 - Depool cp30 [23:23:18] [02dns] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/dns [23:23:19] [url] Page not found · GitHub · GitHub | github.com [23:23:20] [02dns] 07Universal-Omega opened pull request 03#336: Depool cp30 - 13https://github.com/miraheze/dns/pull/336 [23:23:20] [url] Page not found · GitHub · GitHub | github.com [23:25:05] !log apt dist-upgrade on cp* [23:25:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:25:32] RECOVERY - cp31 APT on cp31 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:25:34] !log apt dist-upgrade on cp* (includes kernel upgrades) [23:25:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:05] RECOVERY - cp30 APT on cp30 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:26:37] !log reboot cp20 and cp21 [23:26:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.39, 11.66, 11.49 [23:27:38] [02dns] 07Universal-Omega closed pull request 03#336: Depool cp30 - 13https://github.com/miraheze/dns/pull/336 [23:27:38] [url] Page not found · GitHub · GitHub | github.com [23:27:39] [02miraheze/dns] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [23:27:41] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/e2d10abe148c...56133589ac6f [23:27:41] [url] Comparing e2d10abe148c...56133589ac6f · miraheze/dns · GitHub | github.com [23:27:42] [02miraheze/dns] 07Universal-Omega 035613358 - Depool cp30 [23:28:09] [02dns] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/dns [23:28:09] [url] Page not found · GitHub · GitHub | github.com [23:29:00] !log reboot cp30 [23:29:07] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03revert-336-Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/4e7800595abb [23:29:09] [02miraheze/dns] 07Universal-Omega 034e78005 - Revert "Depool cp30" [23:29:10] [02dns] 07Universal-Omega created branch 03revert-336-Universal-Omega-patch-1 - 13https://github.com/miraheze/dns [23:29:11] [url] Page not found · GitHub · GitHub | github.com [23:29:12] [02dns] 07Universal-Omega opened pull request 03#337: Revert "Depool cp30" - 13https://github.com/miraheze/dns/pull/337 [23:29:12] ... [23:29:17] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.58, 0.14, 0.05 [23:29:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:29:41] [02dns] 07Universal-Omega closed pull request 03#337: Revert "Depool cp30" - 13https://github.com/miraheze/dns/pull/337 [23:29:41] ... [23:29:42] [02miraheze/dns] 07Universal-Omega deleted branch 03revert-336-Universal-Omega-patch-1 [23:29:44] [02dns] 07Universal-Omega deleted branch 03revert-336-Universal-Omega-patch-1 - 13https://github.com/miraheze/dns [23:29:44] ... [23:29:45] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/56133589ac6f...b07a192806de [23:29:46] [url] Comparing 56133589ac6f...b07a192806de · miraheze/dns · GitHub | github.com [23:29:47] [02miraheze/dns] 07Universal-Omega 03b07a192 - Revert "Depool cp30" [23:30:52] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/11e0a128ec81 [23:30:53] [02miraheze/dns] 07Universal-Omega 0311e0a12 - Depool cp31 [23:30:55] [02dns] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/dns [23:30:55] [url] Page not found · GitHub · GitHub | github.com [23:30:56] [02dns] 07Universal-Omega opened pull request 03#338: Depool cp31 - 13https://github.com/miraheze/dns/pull/338 [23:30:57] [url] Page not found · GitHub · GitHub | github.com [23:31:06] [02dns] 07Universal-Omega closed pull request 03#338: Depool cp31 - 13https://github.com/miraheze/dns/pull/338 [23:31:06] [url] Page not found · GitHub · GitHub | github.com [23:31:07] [02miraheze/dns] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [23:31:09] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/b07a192806de...f9333d4defa7 [23:31:09] [url] Comparing b07a192806de...f9333d4defa7 · miraheze/dns · GitHub | github.com [23:31:10] [02miraheze/dns] 07Universal-Omega 03f9333d4 - Depool cp31 [23:31:12] [02dns] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/dns [23:31:12] [url] Page not found · GitHub · GitHub | github.com [23:31:16] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp30.miraheze.org - NS RECORDS OK [23:31:16] PROBLEM - cp30 HTTPS on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 6061 bytes in 0.331 second response time [23:31:17] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [23:31:49] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 93% [23:32:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.07, 7.92, 10.02 [23:34:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 4.92, 8.03, 11.91 [23:35:17] RECOVERY - cp30 HTTPS on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3331 bytes in 0.859 second response time [23:35:17] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [23:35:50] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 2% [23:36:29] !log reboot cp31 [23:36:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:36:59] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03revert-338-Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/db7c4ce226ce [23:37:00] [02miraheze/dns] 07Universal-Omega 03db7c4ce - Revert "Depool cp31" [23:37:02] [02dns] 07Universal-Omega created branch 03revert-338-Universal-Omega-patch-1 - 13https://github.com/miraheze/dns [23:37:02] [url] Page not found · GitHub · GitHub | github.com [23:37:03] [02dns] 07Universal-Omega opened pull request 03#339: Revert "Depool cp31" - 13https://github.com/miraheze/dns/pull/339 [23:37:04] ... [23:37:11] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/f9333d4defa7...494a476f46f7 [23:37:12] [url] Comparing f9333d4defa7...494a476f46f7 · miraheze/dns · GitHub | github.com [23:37:13] [02miraheze/dns] 07Universal-Omega 03494a476 - Revert "Depool cp31" [23:37:14] [02dns] 07Universal-Omega closed pull request 03#339: Revert "Depool cp31" - 13https://github.com/miraheze/dns/pull/339 [23:37:15] ... [23:37:16] [02miraheze/dns] 07Universal-Omega deleted branch 03revert-338-Universal-Omega-patch-1 [23:37:17] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 7.79, 4.02, 1.68 [23:37:17] [02dns] 07Universal-Omega deleted branch 03revert-338-Universal-Omega-patch-1 - 13https://github.com/miraheze/dns [23:37:18] [url] Page not found · GitHub · GitHub | github.com [23:37:58] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.78, 0.31, 0.12 [23:40:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.44, 9.95, 11.43 [23:40:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.55, 9.88, 9.80 [23:41:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 3.73, 3.23 [23:42:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.52, 3.97, 3.18 [23:42:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.95, 9.80, 9.79 [23:43:58] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.19, 1.70, 0.81 [23:44:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.82, 4.52, 3.46 [23:46:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.51, 11.00, 11.47 [23:46:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.71, 10.21, 9.97 [23:48:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.87, 11.80, 11.68 [23:49:02] RECOVERY - bast101 APT on bast101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:49:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.24, 3.90, 3.54 [23:50:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 10.01, 9.99, 9.93 [23:52:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.69, 3.32, 3.40 [23:52:10] !log apt dist-upgrade and reboot bast101 (includes kernel upgrade) [23:52:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:52:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.70, 11.50, 11.62 [23:53:17] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.95, 1.93, 1.99 [23:54:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.62, 12.69, 11.99 [23:55:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.75, 3.06, 3.35 [23:55:47] RECOVERY - bast121 APT on bast121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:56:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.51, 3.48 [23:56:48] !log apt dist-upgrade and reboot bast121 (includes kernel upgrade) [23:56:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:57:17] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.34, 2.05, 2.00 [23:58:13] !log sudo mount /mnt/mediawiki-static on test131 and mwtask141 [23:58:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:58:24] RECOVERY - test131 Check Gluster Clients on test131 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [23:58:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.20, 10.77, 10.26 [23:59:06] RECOVERY - mwtask141 Check Gluster Clients on mwtask141 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [23:59:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.92, 4.97, 3.98