[00:00:19] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.94, 1.71, 1.97 [00:01:10] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.78, 3.52, 3.31 [00:02:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.71, 3.43, 3.92 [00:02:18] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.23, 1.89, 2.01 [00:02:40] RECOVERY - cp30 Disk Space on cp30 is OK: DISK OK - free space: / 7815 MB (19% inode=96%); [00:02:47] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.47, 1.61, 1.31 [00:03:05] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.26, 3.52, 3.34 [00:03:13] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 7825 MB (20% inode=96%); [00:03:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.91, 4.79, 4.11 [00:04:16] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 8542 MB (21% inode=96%); [00:04:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.03, 4.80, 4.34 [00:04:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.85, 1.84, 1.97 [00:04:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.39, 4.47, 3.74 [00:04:38] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8263 MB (21% inode=96%); [00:04:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.16, 1.40, 1.26 [00:08:47] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.61, 2.18, 1.57 [00:10:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.55, 1.78, 1.49 [00:10:47] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.67, 3.03, 3.20 [00:12:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.92, 1.53, 1.44 [00:16:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.13, 1.87, 1.88 [00:18:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.85, 1.84, 1.60 [00:20:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.10, 3.51, 4.00 [00:20:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.11, 1.53, 1.51 [00:21:24] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.43, 3.42, 3.27 [00:22:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.71, 4.01 [00:24:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.50, 3.89 [00:25:15] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.35, 3.81, 3.44 [00:26:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.36, 3.96, 4.02 [00:28:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.89, 3.64, 3.90 [00:31:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.40, 3.19, 3.88 [00:34:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.46, 10.28, 9.08 [00:36:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.56, 9.18, 8.82 [00:36:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.31, 2.86, 3.98 [00:37:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.69, 2.84, 3.36 [00:40:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.13, 3.80, 4.09 [00:40:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.65, 1.83, 2.00 [00:41:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.60, 2.82, 3.30 [00:42:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.56, 3.96 [00:44:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.69, 3.95 [00:44:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.92, 2.00 [00:50:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.92, 6.80, 6.05 [00:52:38] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.26, 6.66, 6.10 [01:00:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.66, 3.95 [01:01:25] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14603 MB (10% inode=99%); [01:04:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.64, 3.88, 3.95 [01:05:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.71, 3.41 [01:06:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.58, 1.80, 1.97 [01:09:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.58, 5.04, 3.91 [01:11:13] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.18, 6.77, 6.35 [01:11:45] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:47] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:13:11] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.89, 6.97, 6.47 [01:13:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.09, 9.95, 8.79 [01:13:40] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.324 second response time [01:13:41] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.457 second response time [01:15:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.59, 9.38, 8.73 [01:15:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.63, 3.66, 3.66 [01:16:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.48, 3.78, 3.88 [01:18:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.53, 2.03, 1.93 [01:24:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.52, 1.95, 1.96 [01:25:25] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 16585 MB (12% inode=99%); [01:26:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.93, 3.95, 3.76 [01:26:55] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.15, 6.37, 6.63 [01:27:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.65, 3.95, 3.67 [01:28:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 4.00, 3.82 [01:29:09] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.19, 3.59, 3.57 [01:30:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 3.54, 3.98 [01:32:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.12, 1.80, 1.86 [01:33:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.15, 3.58, 3.54 [01:33:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.38, 3.98, 3.83 [01:34:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.66, 4.50, 4.23 [01:34:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.70, 1.71, 1.81 [01:38:41] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:39:18] huh? That is not even active.... [01:40:52] PROBLEM - prometheus101 Current Load on prometheus101 is CRITICAL: CRITICAL - load average: 5.31, 2.99, 1.46 [01:42:17] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.15, 1.43, 1.65 [01:42:34] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.98, 6.37, 6.23 [01:42:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.96, 3.91 [01:44:32] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.84, 3.56, 3.99 [01:44:32] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.10, 6.92, 6.44 [01:45:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.76, 3.98, 4.00 [01:48:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.77, 1.97, 1.76 [01:48:27] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.76, 7.09, 6.64 [01:48:41] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:50:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.20, 3.41, 3.72 [01:50:54] PROBLEM - prometheus101 Current Load on prometheus101 is WARNING: WARNING - load average: 0.82, 1.73, 1.85 [01:51:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.82, 1.68, 1.31 [01:51:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.55, 4.08, 4.02 [01:52:23] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.69, 7.98, 7.05 [01:52:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.79, 2.92, 3.35 [01:52:53] RECOVERY - prometheus101 Current Load on prometheus101 is OK: OK - load average: 0.32, 1.27, 1.67 [01:53:45] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.35, 1.63, 1.34 [01:53:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.28, 3.76, 3.91 [01:54:20] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.61, 7.64, 7.05 [01:54:31] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.77, 3.22, 3.58 [01:56:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.33, 1.83, 1.84 [01:58:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.08, 3.58, 3.64 [01:59:11] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.60, 1.99, 1.42 [02:00:14] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.21, 7.73, 7.24 [02:00:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.05, 3.43, 3.89 [02:00:31] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.19, 2.99, 3.41 [02:01:11] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.06, 1.61, 1.35 [02:02:11] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.75, 7.42, 7.18 [02:02:32] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.05, 2.97, 3.35 [02:07:33] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.94, 1.55 [02:09:32] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.06, 1.62, 1.48 [02:09:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.11, 2.70, 3.26 [02:10:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.31, 2.77, 3.33 [02:10:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.24, 1.89, 1.85 [02:17:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.96, 3.68 [02:17:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.44, 3.22 [02:19:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.12, 3.82, 3.68 [02:19:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.59, 3.19, 3.16 [02:22:52] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.68, 3.58, 3.34 [02:24:47] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.95, 3.40, 3.31 [02:25:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.57, 3.03, 3.37 [02:28:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.31, 3.74, 3.46 [02:29:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 3.99, 3.69 [02:30:04] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.58, 5.78, 6.57 [02:30:14] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.68, 3.96, 3.56 [02:30:35] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.71, 3.52, 3.37 [02:32:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.96, 3.40, 3.39 [02:33:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.54, 3.56 [02:34:31] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.56, 3.81, 3.50 [02:35:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.97, 4.11, 3.80 [02:35:59] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.40, 3.41 [02:36:32] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.73, 3.70, 3.49 [02:39:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.76, 3.40 [02:40:32] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 1.74, 2.96, 3.26 [02:41:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.64, 3.27, 3.26 [02:41:24] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [02:41:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.00, 3.24, 3.36 [02:42:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.83, 2.00 [02:42:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.88, 3.88 [02:44:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 4.05, 3.94 [02:46:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.19, 1.93, 2.00 [02:46:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.96, 3.93 [02:48:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 4.06, 3.96 [02:54:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.89, 3.79, 3.93 [02:57:24] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.20, 3.72, 3.40 [02:58:20] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.29, 1.74, 1.99 [02:59:20] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.00, 3.43, 3.33 [03:01:16] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.16, 3.29, 3.28 [03:06:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.02, 2.95, 3.35 [03:09:24] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [03:09:57] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.83, 3.32, 3.25 [03:11:52] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.83, 3.17, 3.20 [03:14:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.99, 3.94, 3.61 [03:18:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.49, 1.99, 1.88 [03:18:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.78, 3.88, 3.66 [03:20:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.84, 4.19, 3.80 [03:22:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.79, 3.68, 3.65 [03:24:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.49, 1.86, 1.90 [03:24:32] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.62, 3.55, 3.39 [03:26:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.15, 3.61, 3.43 [03:28:32] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.44, 3.57, 3.44 [03:30:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.59, 3.93, 3.67 [03:30:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.72, 4.00, 3.61 [03:32:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.16, 1.85, 1.85 [03:32:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.57, 3.57 [03:32:32] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.77, 3.96, 3.65 [03:34:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.85, 1.81, 1.83 [03:34:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 3.83, 3.66 [03:34:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.20, 4.00, 3.70 [03:36:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.64, 3.61 [03:36:32] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.37, 3.77, 3.65 [03:38:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.81, 4.12, 3.79 [03:40:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.48, 11.69, 9.79 [03:40:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 16.94, 11.27, 8.27 [03:40:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 19.48, 12.29, 8.46 [03:42:17] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.44, 1.58, 1.70 [03:42:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.54, 3.53 [03:42:31] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.56, 3.76, 3.71 [03:44:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.30, 3.43 [03:44:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.59, 10.79, 10.00 [03:44:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.05, 10.13, 8.64 [03:46:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.23, 2.85, 3.25 [03:46:30] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.74, 9.76, 9.71 [03:46:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.93, 9.87, 8.96 [03:48:18] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.52, 2.05, 1.84 [03:50:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.89, 3.64 [03:50:31] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 5.78, 4.29, 3.86 [03:52:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 4.36, 3.85 [03:54:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.22, 3.91, 3.75 [03:54:32] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.44, 3.96, 3.83 [04:02:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.12, 3.00, 3.31 [04:04:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.22, 3.00 [04:06:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.59, 2.59, 2.79 [04:06:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.73, 1.95, 2.00 [04:08:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.04, 3.68, 3.66 [04:09:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.14, 2.96 [04:10:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.55, 3.47 [04:10:32] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.55, 3.54, 3.61 [04:11:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.30, 3.28, 3.04 [04:12:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.02, 1.91, 1.96 [04:12:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.02, 3.64, 3.63 [04:14:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.71, 3.53 [04:16:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 3.15, 3.35 [04:16:31] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.44, 3.71, 3.68 [04:24:31] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.10, 2.89, 3.33 [04:26:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.85, 1.99 [04:27:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.41, 3.28 [04:28:18] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.47, 3.10 [04:30:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.01, 3.41, 3.13 [04:32:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.94, 3.24, 3.09 [04:34:14] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.47, 1.73, 1.31 [04:35:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 3.17, 3.28 [04:36:13] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.40, 1.25, 1.18 [04:44:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.82, 3.44 [04:46:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.02, 1.77, 1.75 [04:48:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.81, 3.55 [04:49:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.84, 3.59 [04:51:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.01, 3.49, 3.48 [04:55:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.43, 3.13, 3.34 [04:56:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.61, 1.88, 1.90 [05:02:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.92, 2.15, 1.98 [05:10:37] [02puppet] 07Universal-Omega opened pull request 03#2575: Remove `install_cloud_class` - 13https://github.com/miraheze/puppet/pull/2575 [05:10:37] [url] Page not found · GitHub · GitHub | github.com [05:10:44] [02puppet] 07Universal-Omega edited pull request 03#2575: Remove `install_cloud_class` - 13https://github.com/miraheze/puppet/pull/2575 [05:10:45] [url] Page not found · GitHub · GitHub | github.com [05:11:12] [02puppet] 07Universal-Omega synchronize pull request 03#2575: Remove `install_cloud_class` - 13https://github.com/miraheze/puppet/pull/2575 [05:11:12] [url] Page not found · GitHub · GitHub | github.com [05:11:27] [02puppet] 07Universal-Omega synchronize pull request 03#2575: Remove `install_cloud_class` - 13https://github.com/miraheze/puppet/pull/2575 [05:11:27] ... [05:24:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.14, 1.72, 1.29 [05:26:57] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.24, 1.21, 1.15 [05:38:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.62, 4.04, 3.46 [05:40:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.44, 3.51, 3.34 [05:42:26] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.68, 3.34, 3.30 [05:46:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 4.07, 3.62 [05:46:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.23, 4.10, 3.43 [05:48:31] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.21, 3.75, 3.39 [05:49:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.04, 3.64, 3.20 [05:50:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.00, 3.02, 3.02 [05:52:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.80, 3.69 [05:52:32] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.69, 3.23, 3.27 [05:58:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.69, 3.52, 3.54 [05:58:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.66, 3.35, 3.13 [05:59:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.74, 4.28, 2.45 [06:00:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.32, 3.46 [06:00:33] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.13, 3.32, 3.15 [06:02:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.67, 2.77, 3.24 [06:05:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.55, 1.72, 1.86 [06:09:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.67, 1.19, 1.61 [06:15:23] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 221.84 ms [06:19:22] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 115.63 ms [06:26:56] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[postgresql-contrib-13] [06:31:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.33, 3.07 [06:32:16] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.43, 2.01, 1.75 [06:33:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.37, 3.07, 3.01 [06:34:10] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.94, 1.68, 1.66 [06:35:54] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.94, 3.92, 2.82 [06:40:31] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [06:40:31] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:44:31] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.84, 3.55, 3.09 [06:46:31] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.34, 3.36, 3.07 [06:47:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.75, 3.41, 3.08 [06:50:16] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.15, 1.78, 1.58 [06:50:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.25, 4.05, 3.43 [06:51:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 4.01, 3.40 [06:52:31] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.97, 3.73, 3.40 [06:52:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.71, 3.88, 3.43 [06:54:55] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:55:57] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.60, 1.80, 1.65 [06:56:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.62, 3.39 [06:57:50] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.24, 1.60, 1.59 [06:58:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.50, 3.38 [06:59:54] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 2.91, 3.61, 3.95 [07:00:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 5.66, 4.14, 3.62 [07:00:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.68, 3.17, 3.27 [07:07:54] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.03, 3.43, 3.66 [07:09:11] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [07:09:54] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.93, 3.51, 3.66 [07:10:51] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.54, 3.32 [07:14:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.45, 3.34 [07:15:54] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.23, 2.76, 3.29 [07:16:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.06, 3.26, 3.28 [07:18:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.48, 1.84, 1.50 [07:18:31] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.93, 3.70, 3.97 [07:19:58] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.74, 1.76, 1.51 [07:20:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.81, 4.11, 4.09 [07:21:51] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.22, 1.59, 1.48 [07:34:40] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 4287 MB (10% inode=96%); [07:36:32] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.68, 3.69, 3.97 [07:37:51] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [07:38:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.05, 3.83, 3.99 [07:40:31] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.51, 3.47, 3.85 [07:44:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 5.13, 3.99, 3.95 [07:45:13] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 4281 MB (10% inode=96%); [07:56:39] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.31, 1.66, 1.29 [07:57:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.06, 1.92, 1.60 [07:58:36] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.54, 1.54, 1.29 [07:59:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.52, 1.94, 1.66 [08:00:22] [02puppet] 07Universal-Omega opened pull request 03#2576: check_reverse_dns: add `eu.org` to `extra_known_tlds` - 13https://github.com/miraheze/puppet/pull/2576 [08:00:22] [url] Page not found · GitHub · GitHub | github.com [08:01:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.15, 1.70, 1.61 [08:01:49] Reception|away: ^ are you able to merge that? [08:04:30] Or paladox ^ [08:08:18] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.60, 1.74, 1.51 [08:08:27] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [08:08:27] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:10:15] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.35, 1.59, 1.48 [08:11:13] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.60, 1.78, 1.69 [08:13:07] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.35, 1.64, 1.65 [08:14:16] [02puppet] 07Reception123 closed pull request 03#2576: check_reverse_dns: add `eu.org` to `extra_known_tlds` - 13https://github.com/miraheze/puppet/pull/2576 [08:14:16] [url] Page not found · GitHub · GitHub | github.com [08:14:17] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e43a6feff0b4...7501c39f9fde [08:14:18] [url] Comparing e43a6feff0b4...7501c39f9fde · miraheze/puppet · GitHub | github.com [08:14:19] [02miraheze/puppet] 07Universal-Omega 037501c39 - check_reverse_dns: add `eu.org` to `extra_known_tlds` (#2576) [08:14:26] Ty Reception|away [08:14:29] no problem [08:15:59] [02ssl] 07Universal-Omega opened pull request 03#523: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/523 [08:16:00] ... [08:16:16] [02ssl] 07Universal-Omega synchronize pull request 03#523: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/523 [08:16:16] ... [08:16:28] [02ssl] 07Universal-Omega synchronize pull request 03#523: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/523 [08:16:29] [url] Page not found · GitHub · GitHub | github.com [08:16:41] [02ssl] 07Universal-Omega synchronize pull request 03#523: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/523 [08:16:41] [url] Page not found · GitHub · GitHub | github.com [08:16:58] [02ssl] 07Universal-Omega synchronize pull request 03#523: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/523 [08:16:58] [url] Page not found · GitHub · GitHub | github.com [08:17:24] [02ssl] 07Universal-Omega synchronize pull request 03#523: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/523 [08:17:25] ... [08:18:05] [02ssl] 07Universal-Omega synchronize pull request 03#523: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/523 [08:18:06] [url] Page not found · GitHub · GitHub | github.com [08:18:52] [02dns] 07Universal-Omega opened pull request 03#274: Remove petrawiki.org zone - 13https://github.com/miraheze/dns/pull/274 [08:18:52] [url] Page not found · GitHub · GitHub | github.com [08:26:32] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 1.06, 2.68, 3.71 [08:28:16] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.02, 1.84, 1.66 [08:28:32] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 1.25, 2.14, 3.38 [08:29:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 3.18, 2.59 [08:31:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.76, 2.92, 2.57 [08:31:41] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.19, 1.79, 1.53 [08:33:38] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.13, 1.65, 1.52 [08:35:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.69, 2.97 [08:35:51] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.71, 2.00, 1.89 [08:39:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.86, 3.22 [08:40:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.34, 3.08, 2.91 [08:41:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.10, 1.38, 1.65 [08:42:31] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.03, 2.56, 2.74 [08:43:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.18, 3.21, 3.12 [08:46:28] RECOVERY - largedu.eu.org - reverse DNS on sslhost is OK: SSL OK - largedu.eu.org reverse DNS resolves to cp21.miraheze.org - CNAME FLAT [08:55:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.03, 1.69, 1.65 [08:57:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.97, 1.82, 1.71 [09:01:04] PROBLEM - 3.141592653589793238462643383279502884197169399375105820974944592.ml - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for 3.141592653589793238462643383279502884197169399375105820974944592.ml could not be found [09:01:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.31, 1.58, 1.64 [09:02:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.93, 3.34, 3.07 [09:04:52] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.74, 3.10, 3.02 [09:06:10] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [09:26:25] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.36, 1.62, 1.40 [09:26:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.90, 2.26, 1.80 [09:28:22] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.39, 1.47, 1.37 [09:30:23] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.66, 1.93, 1.77 [09:36:10] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [09:45:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.53, 1.62, 1.69 [09:49:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.37, 1.68, 1.72 [09:50:45] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.07, 1.72, 1.47 [09:52:42] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.17, 1.58, 1.46 [09:53:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.37, 1.97, 1.81 [10:01:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.38, 1.89, 1.90 [10:11:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.25, 1.74, 1.79 [10:13:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.59, 1.70, 1.77 [10:21:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.08, 1.54, 1.68 [10:24:38] PROBLEM - cp21 Disk Space on cp21 is WARNING: DISK WARNING - free space: / 4273 MB (10% inode=96%); [10:27:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.45, 1.88, 1.74 [10:31:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.86, 1.54, 1.65 [10:36:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.93, 2.90, 2.50 [10:38:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.57, 2.41, 2.37 [10:52:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.84, 1.64, 1.61 [10:54:30] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.64, 1.87, 1.69 [10:56:23] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.42, 1.75, 1.67 [11:00:53] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.98, 6.54, 5.53 [11:02:50] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.18, 6.41, 5.60 [11:09:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.23, 2.80 [11:10:08] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is WARNING: WARNING - NGINX Error Rate is 52% [11:11:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.91, 2.80, 2.70 [11:13:57] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 17% [11:14:17] PROBLEM - cp20 Disk Space on cp20 is WARNING: DISK WARNING - free space: / 4270 MB (10% inode=96%); [11:15:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.16, 1.95, 1.84 [11:19:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.96, 1.96, 1.88 [11:21:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.60, 2.45, 2.06 [11:27:03] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.64, 1.81, 1.41 [11:31:03] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.15, 1.72, 1.50 [11:33:03] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.11, 1.82, 1.56 [11:33:41] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.29, 3.75, 3.07 [11:35:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.50, 1.79, 2.00 [11:37:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.52, 2.07, 2.08 [11:39:03] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.42, 1.88, 1.72 [11:43:03] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.13, 1.87, 1.74 [11:43:52] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.19, 7.55, 6.78 [11:45:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.13, 1.58, 1.65 [11:45:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.38, 3.65, 2.97 [11:47:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 3.24, 2.60 [11:49:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 3.92, 3.29 [11:49:45] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.16, 7.32, 7.02 [11:49:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.76, 2.97, 2.57 [11:53:13] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.75, 3.39, 3.21 [11:54:51] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.29, 3.72, 3.82 [11:55:38] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.82, 6.09, 6.63 [11:57:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.13, 2.75 [11:59:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.33, 3.21, 2.83 [12:03:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.65, 9.70, 8.93 [12:03:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.82, 3.63, 3.09 [12:04:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.38, 3.53, 2.91 [12:04:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.84, 3.77, 3.71 [12:05:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.88, 3.52 [12:05:50] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.28, 3.14, 2.98 [12:05:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.26, 3.46, 2.97 [12:07:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.66, 3.26, 2.96 [12:08:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.63, 9.11, 9.05 [12:11:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.77, 3.21, 3.34 [12:13:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.96, 3.83, 3.26 [12:13:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.35, 3.07 [12:14:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.51, 3.77, 3.57 [12:15:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.13, 3.26, 3.07 [12:16:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.56, 3.28, 3.40 [12:17:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.81, 2.95, 3.04 [12:21:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.30, 1.66, 2.00 [12:25:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.83, 1.98, 2.03 [12:27:32] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 2.00, 1.69, 1.51 [12:29:29] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.07, 1.50, 1.46 [12:37:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.35, 1.77, 1.98 [12:38:10] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [12:38:10] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:41:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 10.73, 7.19, 4.03 [12:43:06] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.76, 1.67, 1.48 [12:43:13] PROBLEM - cp31 Disk Space on cp31 is CRITICAL: DISK CRITICAL - free space: / 2339 MB (5% inode=96%); [12:45:03] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.59, 1.89, 1.57 [12:47:03] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 2.00, 1.83, 1.58 [12:49:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.52, 1.68, 1.56 [12:51:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.64, 3.24 [12:53:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.35, 3.33, 3.16 [12:53:54] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.67, 3.22, 2.93 [12:55:54] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.92, 3.09, 2.92 [12:59:03] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.01, 1.76, 1.57 [13:01:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.59, 1.62, 1.54 [13:06:40] PROBLEM - cp30 Disk Space on cp30 is CRITICAL: DISK CRITICAL - free space: / 2327 MB (5% inode=96%); [13:06:49] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [13:09:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.84, 3.37, 3.12 [13:13:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.60, 3.34, 3.19 [13:16:38] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: DISK CRITICAL - free space: / 2331 MB (5% inode=96%); [13:17:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.81, 3.46 [13:19:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.44, 3.34, 3.33 [13:20:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.65, 9.06, 8.71 [13:22:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.70, 8.78, 8.67 [13:23:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.42, 3.36 [13:27:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 3.11, 3.26 [13:27:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.49, 1.78, 1.55 [13:29:52] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.50, 1.61, 1.51 [13:34:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.94, 3.56 [13:36:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.88, 3.83, 3.56 [13:36:10] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [13:36:10] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:37:38] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.15, 2.20, 1.74 [13:38:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.77, 4.27, 3.73 [13:39:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.38, 4.01, 3.44 [13:41:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.05, 3.30, 3.25 [13:43:29] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.27, 1.91, 1.80 [13:43:43] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.18, 6.49, 6.25 [13:45:41] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.89, 6.27, 6.23 [13:45:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.95, 3.81 [13:47:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 3.97, 3.83 [13:49:21] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.10, 1.57, 1.70 [13:49:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.86, 3.73, 3.75 [13:53:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.83, 3.07, 2.97 [13:55:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.78, 2.92, 2.93 [13:57:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 3.69, 3.64 [13:59:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.38, 3.16, 3.45 [14:01:03] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.74, 1.80, 1.70 [14:03:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.76, 1.45, 1.59 [14:03:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.62, 3.11, 3.39 [14:04:17] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: DISK CRITICAL - free space: / 2326 MB (5% inode=96%); [14:06:10] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [14:10:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.47, 9.20, 7.86 [14:11:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.10, 4.37, 3.68 [14:12:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.32, 4.70, 3.55 [14:12:14] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.21, 4.16, 3.33 [14:14:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.19, 9.20, 8.21 [14:15:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.68, 3.39 [14:16:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.37, 9.56, 8.39 [14:16:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.81, 3.78, 3.39 [14:16:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.21, 9.59, 8.33 [14:18:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.44, 9.41, 8.42 [14:19:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.58, 1.74, 1.96 [14:19:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.76, 3.17, 3.25 [14:20:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 10.15, 9.81, 8.75 [14:21:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.99, 3.92 [14:21:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.04, 3.39, 3.35 [14:23:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.40, 1.74, 1.88 [14:31:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.27, 10.66, 9.28 [14:31:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.23, 3.34, 3.52 [14:31:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.72, 1.99, 2.00 [14:32:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.65, 9.69, 9.03 [14:33:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.49, 9.64, 9.06 [14:33:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.68, 3.64 [14:33:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.77, 3.51, 3.25 [14:33:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.51, 3.35 [14:34:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.93, 9.22, 8.95 [14:35:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.63, 4.01, 3.77 [14:35:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.62, 3.80, 3.37 [14:35:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.91, 3.53 [14:37:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.53, 3.62 [14:37:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.45, 3.27, 3.23 [14:37:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.62, 3.41, 3.39 [14:39:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.97, 3.78 [14:41:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.96, 3.71, 3.41 [14:43:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.16, 1.36, 1.70 [14:45:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.33, 3.29, 3.33 [14:45:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.75, 3.30, 3.38 [14:46:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.94, 1.77, 1.40 [14:48:11] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.58, 1.35, 1.29 [14:49:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.16, 2.10, 1.89 [14:49:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.85, 4.15, 3.67 [14:51:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.63, 3.53 [14:53:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.80, 3.93 [14:53:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.33, 2.00, 1.94 [14:53:43] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:45] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:50] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:56] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.248 second response time [14:54:18] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [14:54:18] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.027 second response time [14:54:43] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:51] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 1.468 second response time [14:54:54] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.027 second response time [14:55:10] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.89, 2.04, 1.96 [14:55:48] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 6.880 second response time [14:55:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.81, 3.81, 3.57 [14:55:59] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:19] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:20] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:21] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:24] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:25] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 4 backends are down. mw101 mw112 mw121 mw122 [14:56:25] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.015 second response time [14:56:34] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:48] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:49] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:53] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [14:56:55] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 8.378 second response time [14:57:09] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:57:11] RhinosF1: getting 502s and 503s [14:57:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.48, 5.07, 4.36 [14:57:20] ugochimobi: aware [14:57:22] Agent|Away: hi [14:57:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.68, 1.89, 1.92 [14:57:40] RhinosF1: alright, nice [14:58:01] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.56, 10.84, 9.89 [14:58:06] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 9.31, 5.30, 3.95 [14:58:23] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:34] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.91, 2.75, 2.04 [14:58:54] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 4.051 second response time [14:59:03] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.017 second response time [14:59:53] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 3.291 second response time [14:59:53] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 2.077 second response time [14:59:56] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.088 second response time [15:00:01] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.321 second response time [15:00:08] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.504 second response time [15:00:13] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:17] PROBLEM - cp20 Disk Space on cp20 is WARNING: DISK WARNING - free space: / 2593 MB (6% inode=96%); [15:00:19] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.301 second response time [15:00:25] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [15:00:25] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.512 second response time [15:00:25] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.313 second response time [15:00:26] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 6.430 second response time [15:00:32] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 5.435 second response time [15:00:36] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 4.534 second response time [15:00:40] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.038 second response time [15:00:48] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.981 second response time [15:00:58] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.317 second response time [15:01:02] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.021 second response time [15:01:10] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.314 second response time [15:02:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.00, 9.11, 9.43 [15:02:55] !log delete a couple of access logs on cp20/21 to clean up disk [15:03:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.08, 1.92, 1.91 [15:04:36] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:37] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:37] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.815 second response time [15:04:37] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.666 second response time [15:04:42] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:45] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:52] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:02] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:16] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:17] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:29] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:31] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:23] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.00, 1.93, 2.00 [15:06:27] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.035 second response time [15:06:30] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.351 second response time [15:06:31] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.018 second response time [15:06:35] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.325 second response time [15:06:36] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 1.286 second response time [15:06:37] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.310 second response time [15:06:40] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 2.373 second response time [15:06:58] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.718 second response time [15:07:19] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 7.186 second response time [15:07:20] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 4.108 second response time [15:07:26] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw101 [15:07:32] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.322 second response time [15:07:55] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [15:08:16] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw111 [15:08:20] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.73, 2.53, 2.23 [15:08:48] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.582 second response time [15:09:21] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [15:09:37] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 5.268 second response time [15:09:41] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 2.838 second response time [15:09:52] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [15:10:16] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [15:10:42] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.444 second response time [15:10:53] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.036 second response time [15:11:08] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 1.014 second response time [15:12:04] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.35, 6.67, 5.88 [15:13:41] [02mw-config] 07ugochimobi opened pull request 03#4700: T9250: revoke "read" right from "user" usergroup on feignwiki - 13https://github.com/miraheze/mw-config/pull/4700 [15:13:41] [url] Page not found · GitHub · GitHub | github.com [15:14:49] miraheze/mw-config - ugochimobi the build passed. [15:16:04] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.91, 7.07, 6.24 [15:16:29] ugochimobi: no [15:17:00] RhinosF1: no wot? [15:18:04] ugochimobi: that pr [15:18:04] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.49, 6.76, 6.22 [15:18:30] RhinosF1: the user seems a bit confused [15:18:34] with everything [15:18:58] ugochimobi: yes [15:19:07] I'm not deploying that though [15:19:44] nah, that's why I left it draft [15:19:55] or isn't it draft? [15:20:19] ugochimobi: it is [15:20:27] [02mw-config] 07ugochimobi commented on pull request 03#4700: T9250: revoke "read" right from "user" usergroup on feignwiki - 13https://github.com/miraheze/mw-config/pull/4700#issuecomment-1133030951 [15:20:27] [url] T9250: revoke "read" right from "user" usergroup on feignwiki by ugochimobi · Pull Request #4700 · miraheze/mw-config · GitHub | github.com [15:20:28] okay [15:21:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.77, 9.73, 8.62 [15:22:17] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: DISK CRITICAL - free space: / 2332 MB (5% inode=96%); [15:23:16] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 10.07, 9.54, 8.66 [15:24:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.46, 10.28, 8.89 [15:26:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.12, 10.96, 9.29 [15:27:52] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.80, 1.86, 1.99 [15:28:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.51, 10.97, 9.49 [15:28:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.53, 6.90, 6.49 [15:32:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.91, 9.95, 9.43 [15:32:50] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.80, 6.68, 6.51 [15:33:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:33:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/542131ce79bd [15:33:41] [url] Page not found · GitHub · GitHub | github.com [15:33:42] [02miraheze/puppet] 07paladox 03542131c - matomo: fix enabling redis in php [15:33:43] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.26, 1.64, 1.82 [15:33:43] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [15:33:44] ... [15:33:45] [02puppet] 07paladox opened pull request 03#2577: matomo: fix enabling redis in php - 13https://github.com/miraheze/puppet/pull/2577 [15:33:45] ... [15:34:25] [02puppet] 07paladox closed pull request 03#2577: matomo: fix enabling redis in php - 13https://github.com/miraheze/puppet/pull/2577 [15:34:26] [url] Page not found · GitHub · GitHub | github.com [15:34:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7501c39f9fde...d47c667faa29 [15:34:27] [url] Comparing 7501c39f9fde...d47c667faa29 · miraheze/puppet · GitHub | github.com [15:34:28] [02miraheze/puppet] 07paladox 03d47c667 - matomo: fix enabling redis in php (#2577) [15:34:30] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [15:34:31] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [15:34:32] [url] Page not found · GitHub · GitHub | github.com [15:36:10] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [15:36:13] [02puppet] 07paladox closed pull request 03#2575: Remove `install_cloud_class` - 13https://github.com/miraheze/puppet/pull/2575 [15:36:13] [url] Page not found · GitHub · GitHub | github.com [15:36:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/d47c667faa29...19d421be330b [15:36:15] [url] Comparing d47c667faa29...19d421be330b · miraheze/puppet · GitHub | github.com [15:36:16] [02miraheze/puppet] 07Universal-Omega 0319d421b - Remove `install_cloud_class` (#2575) [15:36:59] [02puppet] 07paladox closed pull request 03#2572: Delete bacula2.yaml - 13https://github.com/miraheze/puppet/pull/2572 [15:37:00] [url] Page not found · GitHub · GitHub | github.com [15:37:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/19d421be330b...e83b00f8e8bb [15:37:02] [url] Comparing 19d421be330b...e83b00f8e8bb · miraheze/puppet · GitHub | github.com [15:37:02] [02miraheze/puppet] 07Universal-Omega 03e83b00f - Delete bacula2.yaml (#2572) [15:37:37] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.87, 1.85, 1.87 [15:37:58] [02puppet] 07paladox closed pull request 03#2569: Remove old PHP versions from `VMlib::Php_version` - 13https://github.com/miraheze/puppet/pull/2569 [15:37:59] [url] Page not found · GitHub · GitHub | github.com [15:38:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e83b00f8e8bb...9189ab9adf4c [15:38:00] [url] Comparing e83b00f8e8bb...9189ab9adf4c · miraheze/puppet · GitHub | github.com [15:38:01] [02miraheze/puppet] 07Universal-Omega 039189ab9 - Remove old PHP versions from `VMlib::Php_version` (#2569) [15:38:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.80, 10.61, 9.87 [15:38:49] [02puppet] 07paladox closed pull request 03#2568: Remove unused PHP-FPM symlinks - 13https://github.com/miraheze/puppet/pull/2568 [15:38:49] [url] Page not found · GitHub · GitHub | github.com [15:38:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-6/±0] 13https://github.com/miraheze/puppet/compare/9189ab9adf4c...2a882d6f9033 [15:38:51] [url] Comparing 9189ab9adf4c...2a882d6f9033 · miraheze/puppet · GitHub | github.com [15:38:52] [02miraheze/puppet] 07Universal-Omega 032a882d6 - Remove unused PHP-FPM symlinks (#2568) [15:39:00] [02puppet] 07paladox closed pull request 03#2570: grafana: update default values - 13https://github.com/miraheze/puppet/pull/2570 [15:39:01] [url] Page not found · GitHub · GitHub | github.com [15:39:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2a882d6f9033...60c7db720097 [15:39:03] [url] Comparing 2a882d6f9033...60c7db720097 · miraheze/puppet · GitHub | github.com [15:39:03] [02miraheze/puppet] 07Universal-Omega 0360c7db7 - grafana: update default values (#2570) [15:39:12] [02puppet] 07paladox closed pull request 03#2571: icinga: update default values - 13https://github.com/miraheze/puppet/pull/2571 [15:39:13] [url] Page not found · GitHub · GitHub | github.com [15:39:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/60c7db720097...5c454b9730f7 [15:39:14] [url] Comparing 60c7db720097...5c454b9730f7 · miraheze/puppet · GitHub | github.com [15:39:15] [02miraheze/puppet] 07Universal-Omega 035c454b9 - icinga: update default values (#2571) [15:42:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.96, 10.06, 9.81 [15:43:29] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.54, 1.50, 1.68 [15:47:10] !log delete a couple of access logs on cp30/31 to clean up disk [15:47:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:52:31] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 1.70, 2.89, 3.86 [15:55:06] paladox: thanks for merging my PRs! [15:55:15] yw [15:58:32] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.35, 2.40, 3.30 [15:58:37] paladox: do you have objections if I go through and completely removed the default lookup values for hosts, so they don't need changed everywhere if things get decommissioned, etc.., and then move the defaults to common.yaml, so they only need changed there for default? Just wondering, as doing that would make it easier in the future. [15:58:51] i have no objections [15:59:19] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:22] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:29] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:35] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:35] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:48] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:21] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 4.692 second response time [16:01:25] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 5.957 second response time [16:01:31] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 3.527 second response time [16:01:31] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.628 second response time [16:01:33] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.020 second response time [16:02:48] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 5.121 second response time [16:03:03] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.54, 2.47, 1.81 [16:03:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.45, 3.99 [16:04:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.74, 10.27, 9.33 [16:04:48] [02mw-config] 07Universal-Omega closed pull request 03#4700: T9250: revoke "read" right from "user" usergroup on feignwiki - 13https://github.com/miraheze/mw-config/pull/4700 [16:04:49] [url] Page not found · GitHub · GitHub | github.com [16:05:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.99, 4.35, 4.25 [16:05:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.06, 1.47, 1.92 [16:06:35] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:03] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.46, 1.86, 1.70 [16:08:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.74, 9.20, 9.08 [16:08:34] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.569 second response time [16:09:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.35, 1.59, 1.62 [16:17:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.15, 2.06, 1.92 [16:18:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.79, 9.97, 9.00 [16:19:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.37, 1.81, 1.85 [16:20:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.26, 9.31, 8.87 [16:21:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.65, 10.41, 9.48 [16:21:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.20, 10.40, 8.84 [16:21:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.39, 1.92, 1.88 [16:23:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.26, 10.57, 9.65 [16:23:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.41, 10.28, 9.00 [16:24:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.78, 10.10, 9.60 [16:25:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.08, 9.93, 9.03 [16:26:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.80, 3.80, 3.32 [16:27:52] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.92, 1.84, 1.64 [16:28:32] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.82, 3.70, 3.34 [16:28:48] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.82, 10.26, 9.79 [16:29:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.72, 2.00, 1.98 [16:29:49] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.46, 1.66, 1.60 [16:30:30] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 10.00, 8.11, 6.86 [16:30:32] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 1.78, 2.97, 3.11 [16:30:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.73, 9.65, 9.62 [16:31:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.45, 10.38, 9.72 [16:31:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.33, 2.02, 1.98 [16:32:27] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.72, 7.53, 6.80 [16:34:25] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.81, 8.21, 7.14 [16:35:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.53, 1.84, 1.92 [16:36:23] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.76, 7.99, 7.19 [16:36:40] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.78, 2.28, 1.49 [16:37:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.36, 10.32, 9.91 [16:38:39] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.74, 1.72, 1.38 [16:39:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.19, 3.64, 2.62 [16:39:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.50, 9.83, 9.44 [16:40:18] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.54, 8.14, 7.37 [16:40:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.29, 1.22, 1.23 [16:41:12] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.98, 9.89, 9.87 [16:41:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.81, 9.84, 9.88 [16:41:45] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.57, 9.78, 9.47 [16:42:16] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.87, 7.99, 7.41 [16:43:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:44:14] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.28, 8.36, 7.61 [16:49:10] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.09, 3.48, 3.22 [16:49:17] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.72, 1.68, 1.42 [16:50:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.87, 10.59, 10.19 [16:51:06] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.53, 3.46, 3.24 [16:53:01] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.98, 3.33, 3.22 [16:53:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.14, 1.49, 1.41 [16:54:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.49, 9.93, 10.02 [16:55:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.85, 10.21, 10.02 [16:58:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.31, 3.95 [17:01:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.38, 9.44, 9.77 [17:06:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.77, 10.40, 10.08 [17:08:04] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.27, 7.33, 7.86 [17:10:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.95, 9.50, 8.80 [17:10:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.81, 10.08, 10.02 [17:10:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.39, 3.89, 3.83 [17:11:56] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 2116 MB (10% inode=90%); [17:12:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.20, 3.73, 3.78 [17:14:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.25, 9.74, 9.36 [17:14:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.20, 9.20, 8.88 [17:15:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.14, 3.65, 4.00 [17:16:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.80, 8.50, 8.95 [17:17:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.30, 1.76, 1.99 [17:17:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.33, 3.78, 4.02 [17:18:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.10, 3.78, 3.77 [17:19:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.30, 2.04, 2.06 [17:19:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.80, 3.47, 3.87 [17:20:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.87, 3.43, 3.64 [17:22:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.24, 10.49, 10.06 [17:23:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.60, 1.81, 1.96 [17:24:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.88, 3.58, 3.61 [17:25:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.16, 1.85, 1.94 [17:25:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.06, 5.02, 4.33 [17:26:04] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.53, 5.82, 6.65 [17:26:30] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.32, 10.00, 9.99 [17:30:43] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.41, 9.90, 9.10 [17:32:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.38, 1.90, 1.50 [17:32:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.41, 9.53, 9.08 [17:33:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.52, 1.91, 1.98 [17:34:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.89, 1.53, 1.41 [17:35:28] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.94, 3.27, 2.98 [17:35:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.29, 2.30, 2.10 [17:36:03] RECOVERY - mw111 APT on mw111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [17:36:31] RECOVERY - mw102 APT on mw102 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [17:37:03] RECOVERY - mw112 APT on mw112 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [17:37:24] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.46, 3.08, 2.95 [17:37:33] RECOVERY - mw122 APT on mw122 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [17:37:34] RECOVERY - mw121 APT on mw121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [17:37:50] RECOVERY - jobchron121 APT on jobchron121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [17:37:50] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [17:38:02] RECOVERY - mw101 APT on mw101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [17:39:05] RECOVERY - test101 APT on test101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [17:44:04] RECOVERY - puppet111 APT on puppet111 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [17:48:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.16, 10.79, 9.77 [17:50:04] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.45, 9.87, 9.56 [17:52:24] RECOVERY - graylog121 APT on graylog121 is OK: APT OK: 40 packages available for upgrade (0 critical updates). [17:52:29] RECOVERY - cloud10 APT on cloud10 is OK: APT OK: 71 packages available for upgrade (0 critical updates). [17:52:34] RECOVERY - cloud12 APT on cloud12 is OK: APT OK: 71 packages available for upgrade (0 critical updates). [17:52:45] RECOVERY - mem121 APT on mem121 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [17:52:45] RECOVERY - matomo101 APT on matomo101 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [17:52:46] RECOVERY - ldap111 APT on ldap111 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [17:52:56] RECOVERY - mail121 APT on mail121 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [17:53:11] RECOVERY - ns2 APT on ns2 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [17:53:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.61, 1.76, 1.47 [17:53:28] RECOVERY - phab121 APT on phab121 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [17:53:33] RECOVERY - gluster111 APT on gluster111 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [17:53:33] RECOVERY - es111 APT on es111 is OK: APT OK: 40 packages available for upgrade (0 critical updates). [17:53:35] RECOVERY - cloud11 APT on cloud11 is OK: APT OK: 71 packages available for upgrade (0 critical updates). [17:53:39] RECOVERY - db101 APT on db101 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [17:53:42] RECOVERY - es101 APT on es101 is OK: APT OK: 40 packages available for upgrade (0 critical updates). [17:53:51] RECOVERY - mem101 APT on mem101 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [17:53:54] RECOVERY - db112 APT on db112 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [17:54:06] RECOVERY - db111 APT on db111 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [17:54:09] RECOVERY - bast101 APT on bast101 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [17:54:11] RECOVERY - mon111 APT on mon111 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [17:54:12] RECOVERY - gluster101 APT on gluster101 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [17:54:13] RECOVERY - bast121 APT on bast121 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [17:54:22] RECOVERY - ns1 APT on ns1 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [17:54:23] RECOVERY - db121 APT on db121 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [17:54:46] RECOVERY - prometheus101 APT on prometheus101 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [17:54:46] RECOVERY - gluster121 APT on gluster121 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [17:54:50] RECOVERY - es121 APT on es121 is OK: APT OK: 40 packages available for upgrade (0 critical updates). [17:55:20] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.61, 1.76, 1.51 [17:55:55] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:57] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.13, 1.57, 1.47 [17:58:16] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:58:22] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:58:33] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:00:13] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 8.708 second response time [18:00:17] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 4375 MB (11% inode=96%); [18:00:25] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 9.601 second response time [18:00:25] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 8.171 second response time [18:00:26] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:00:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:00:37] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 8.585 second response time [18:00:37] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:00:47] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:00:52] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:02:38] PROBLEM - cp21 Disk Space on cp21 is WARNING: DISK WARNING - free space: / 4157 MB (10% inode=96%); [18:02:40] RECOVERY - cp30 Disk Space on cp30 is OK: DISK OK - free space: / 6129 MB (15% inode=96%); [18:02:45] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 8.806 second response time [18:02:51] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 6.213 second response time [18:02:55] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 4.944 second response time [18:03:13] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 6531 MB (16% inode=96%); [18:06:20] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 8.639 second response time [18:06:49] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 8.427 second response time [18:08:10] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dnd.bellinrattin.it could not be found [18:10:45] RECOVERY - cp30 APT on cp30 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [18:10:46] RECOVERY - cp31 APT on cp31 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [18:11:09] RECOVERY - cp20 APT on cp20 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [18:11:09] RECOVERY - cp21 APT on cp21 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [18:12:25] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 10.27, 7.23, 6.07 [18:14:23] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.89, 6.50, 5.94 [18:16:17] PROBLEM - cp20 Disk Space on cp20 is WARNING: DISK WARNING - free space: / 4287 MB (10% inode=96%); [18:19:10] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.99, 3.38, 3.01 [18:20:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.09, 10.72, 9.11 [18:20:46] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.19, 1.38, 1.94 [18:22:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.56, 9.04, 8.69 [18:22:46] PROBLEM - prometheus101 Current Load on prometheus101 is CRITICAL: CRITICAL - load average: 5.45, 2.54, 1.29 [18:22:47] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 3.724 second response time [18:23:15] PROBLEM - prometheus101 APT on prometheus101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [18:24:56] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.69, 3.18, 3.06 [18:25:36] alerting : [FIRING:1, RESOLVED:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:25:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.72, 1.86, 1.99 [18:26:44] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.018 second response time [18:28:46] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 2.992 second response time [18:29:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.31, 1.64, 1.87 [18:29:42] RECOVERY - prometheus101 APT on prometheus101 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [18:30:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.54, 10.01, 9.16 [18:30:44] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.68, 3.51, 3.23 [18:32:03] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.67, 10.52, 9.26 [18:32:39] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.76, 3.29, 3.19 [18:32:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.67, 9.37, 9.02 [18:33:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.47, 10.35, 9.56 [18:33:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.07, 1.82, 1.89 [18:35:09] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.88, 10.06, 9.54 [18:36:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.74, 10.94, 9.80 [18:36:49] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [18:40:57] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.17, 6.90, 6.24 [18:41:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.59, 1.92, 1.96 [18:43:06] PROBLEM - prometheus101 Current Load on prometheus101 is WARNING: WARNING - load average: 0.17, 0.79, 1.95 [18:43:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.31, 10.72, 10.03 [18:44:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.18, 9.99, 9.88 [18:46:50] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.32, 7.64, 6.75 [18:47:04] RECOVERY - prometheus101 Current Load on prometheus101 is OK: OK - load average: 0.19, 0.50, 1.56 [18:47:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.15, 10.11, 9.94 [18:47:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.23, 1.89, 1.90 [18:49:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.73, 10.71, 9.70 [18:49:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.28, 1.69, 1.83 [18:50:46] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.42, 7.59, 6.92 [18:51:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.01, 10.14, 9.62 [18:51:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.08, 1.80, 1.85 [18:52:43] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.12, 6.75, 6.69 [18:53:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.68, 1.72, 1.81 [18:55:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.63, 2.16, 1.97 [18:56:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.72, 10.74, 10.16 [18:56:36] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.96, 7.39, 6.95 [18:57:58] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:58:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 11.43, 10.49 [18:58:16] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:58:21] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:58:22] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:58:34] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.75, 6.78, 6.79 [18:58:52] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:59:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.96, 11.08, 10.46 [19:00:06] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 9.755 second response time [19:02:03] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.01, 10.41, 9.68 [19:02:26] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 3.896 second response time [19:02:27] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 4.275 second response time [19:02:29] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 3.574 second response time [19:03:01] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 2.937 second response time [19:05:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.03, 11.14, 10.56 [19:06:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.94, 10.07, 9.70 [19:06:10] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [19:07:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.96, 10.77, 10.51 [19:07:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.54, 1.60, 1.87 [19:10:18] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.46, 6.87, 6.72 [19:10:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.65, 1.87, 1.42 [19:11:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.44, 2.04, 1.93 [19:12:16] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.47, 6.44, 6.56 [19:12:20] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.07, 2.31, 1.61 [19:14:19] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.68, 1.96, 1.57 [19:15:12] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.86, 9.72, 10.20 [19:15:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.13, 1.67, 1.82 [19:16:17] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.58, 1.47 [19:18:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.73, 9.94, 9.59 [19:18:06] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.21, 6.87, 6.62 [19:19:23] !log erase varnishlog-5xx.log on cp* to cleanup disk space [19:19:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.63, 1.85, 1.83 [19:19:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:20:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.85, 8.96, 9.27 [19:20:04] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.21, 6.53, 6.53 [19:20:34] CosmicAlpha: ^ [19:20:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.12, 10.32, 9.54 [19:21:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.52, 1.76, 1.81 [19:21:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.99, 10.12, 10.21 [19:22:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.37, 9.59, 9.37 [19:23:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.86, 1.43, 1.68 [19:29:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.48, 10.87, 10.41 [19:29:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.42, 1.76, 1.77 [19:31:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.49, 10.50, 10.34 [19:32:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.05, 10.15, 9.69 [19:32:58] PROBLEM - wiki.slimeapedia.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.slimeapedia.org All nameservers failed to answer the query. [19:33:19] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.27, 9.75, 10.09 [19:36:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.04, 9.65, 9.64 [19:37:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.22, 1.52, 1.66 [19:38:04] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.37, 6.77, 6.52 [19:40:04] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.79, 6.32, 6.38 [19:50:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.44, 10.51, 8.94 [19:52:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.86, 9.96, 8.95 [19:54:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.67, 10.35, 10.02 [19:55:24] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.62, 3.28, 2.78 [19:57:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.82, 1.56, 1.46 [19:59:15] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.18, 3.69, 3.05 [20:00:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.21, 9.97, 9.98 [20:01:10] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.61, 3.26, 2.97 [20:01:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.40, 2.27, 1.72 [20:02:26] PROBLEM - wiki.slimeapedia.org - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/ [20:02:27] on3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query slimeapedia.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:03:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.42, 10.47, 9.98 [20:03:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.68, 1.96, 1.68 [20:05:27] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.84, 9.90, 9.82 [20:07:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.30, 1.70, 1.64 [20:10:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.76, 11.38, 10.54 [20:10:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.22, 10.20, 9.43 [20:12:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.86, 11.01, 10.49 [20:12:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.58, 9.19, 9.16 [20:19:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.05, 1.83, 1.64 [20:20:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.20, 10.41, 10.00 [20:20:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.26, 9.97, 10.19 [20:22:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.34, 9.42, 9.68 [20:23:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.73, 1.86, 1.71 [20:25:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.06, 2.21, 1.85 [20:25:36] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.50, 7.53, 6.87 [20:26:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.70, 10.29, 10.28 [20:29:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.04, 1.75, 1.76 [20:31:55] RECOVERY - wiki.slimeapedia.org - reverse DNS on sslhost is OK: SSL OK - wiki.slimeapedia.org reverse DNS resolves to cp21.miraheze.org - CNAME OK [20:33:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.44, 3.38, 3.23 [20:33:27] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.40, 7.99, 7.30 [20:33:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.85, 1.37, 1.61 [20:34:56] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.69, 3.12, 3.15 [20:38:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.34, 10.24, 9.78 [20:40:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.98, 9.69, 9.63 [20:40:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.66, 9.43, 10.03 [20:43:17] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.43, 10.82, 9.58 [20:44:34] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.41, 3.04, 2.99 [20:45:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.98, 10.68, 9.69 [20:46:32] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.36, 3.15, 3.03 [20:47:07] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.21, 10.17, 9.64 [20:47:17] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:47:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.47, 1.85, 1.67 [20:49:19] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 5.545 second response time [20:49:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.06, 1.53, 1.57 [20:50:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.32, 10.80, 10.16 [20:55:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.08, 1.99, 1.71 [20:56:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.68, 10.46, 10.10 [20:57:01] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.48, 1.88, 1.39 [20:57:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.89, 1.57, 1.59 [20:58:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.52, 10.18, 10.04 [21:00:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.39, 1.80, 1.48 [21:02:34] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.30, 11.92, 10.83 [21:02:57] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.84, 2.01, 1.58 [21:04:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.63, 11.83, 10.93 [21:04:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.26, 1.68, 1.51 [21:06:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.52, 10.41, 10.01 [21:06:48] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.46, 7.42, 7.93 [21:08:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.69, 9.67, 9.78 [21:10:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:14:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.31, 9.36, 10.14 [21:14:39] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.33, 5.46, 6.80 [21:14:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 3.60, 3.99 [21:16:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.82, 4.02 [21:18:32] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.79, 6.29, 6.84 [21:18:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.64, 3.44, 3.86 [21:19:57] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.88, 2.39, 1.48 [21:20:30] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.55, 6.82, 6.95 [21:21:56] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.33, 1.88, 1.40 [21:22:27] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.15, 6.45, 6.80 [21:23:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.85, 1.54, 1.33 [21:24:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.62, 10.80, 10.43 [21:26:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.94, 9.84, 10.13 [21:26:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.61, 3.75 [21:28:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.34, 3.65 [21:29:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.42, 10.83, 10.12 [21:31:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.04, 9.97, 9.90 [21:32:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.42, 10.55, 10.34 [21:36:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.25, 9.80, 10.09 [21:36:48] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.54, 3.65, 3.15 [21:37:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.77, 10.45, 10.16 [21:38:43] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.83, 3.50, 3.14 [21:39:44] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.79, 9.14, 9.72 [21:40:05] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.97, 6.56, 6.35 [21:40:39] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.03, 3.76, 3.28 [21:42:04] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.75, 7.47, 6.72 [21:42:34] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.85, 3.38, 3.20 [21:42:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.21, 2.87, 3.28 [21:44:04] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.36, 7.13, 6.68 [21:44:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.61, 10.06, 9.39 [21:46:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.82, 9.68, 9.32 [21:47:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.41, 3.34, 3.99 [21:48:32] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.67, 3.31, 3.19 [21:50:04] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.92, 6.66, 6.65 [21:50:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.21, 3.74, 3.37 [21:50:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.00, 9.38, 8.69 [21:51:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.42, 10.87, 10.01 [21:52:32] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.18, 3.50, 3.32 [21:52:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.76, 9.58, 8.86 [21:54:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.48, 3.96 [21:54:32] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 1.84, 3.02, 3.18 [21:54:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.02, 10.91, 10.24 [21:55:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.32, 3.53, 3.83 [21:56:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.69, 4.17, 4.14 [21:57:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.68, 3.87 [21:58:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.30, 9.91, 10.02 [22:00:04] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.86, 7.00, 6.73 [22:01:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.83, 3.89 [22:02:04] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 10.90, 8.27, 7.21 [22:02:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 5.47, 3.87, 3.34 [22:03:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.07, 10.99, 10.22 [22:04:32] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.96, 3.83, 3.39 [22:05:20] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.26, 9.88, 9.90 [22:05:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.58, 3.33 [22:05:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.05, 3.78, 3.89 [22:06:32] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.06, 3.18, 3.21 [22:07:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.92, 3.39, 3.29 [22:09:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.21, 10.70, 10.22 [22:09:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.75, 3.82 [22:11:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.93, 9.67, 9.90 [22:14:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.87, 3.43, 3.36 [22:15:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.47, 3.55, 3.78 [22:16:04] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.53, 7.73, 7.72 [22:16:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.39, 3.65, 4.00 [22:16:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.21, 3.31, 3.32 [22:20:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.49, 9.78, 9.56 [22:22:04] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.87, 5.35, 6.68 [22:22:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.58, 3.80 [22:24:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.96, 3.64, 3.79 [22:26:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.14, 3.14 [22:26:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 4.02, 3.92 [22:27:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.78, 3.07, 3.36 [22:28:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.36, 9.77, 9.78 [22:30:07] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.21, 3.33, 3.21 [22:32:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.84, 3.88 [22:32:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.58, 10.39, 10.03 [22:36:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.84, 3.85 [22:36:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.85, 9.82, 9.87 [22:38:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.69, 3.80 [22:39:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.22, 1.74, 1.37 [22:40:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.68, 4.13, 3.92 [22:40:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.29, 10.41, 10.07 [22:41:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.16, 9.63, 8.12 [22:42:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.88, 9.65, 8.57 [22:43:09] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.20, 9.57, 8.30 [22:43:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.93, 3.62, 3.40 [22:43:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.01, 1.57, 1.40 [22:45:12] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.50, 3.97, 3.55 [22:45:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.65, 3.33 [22:46:24] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.50, 2.18, 1.46 [22:47:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.52, 3.43 [22:47:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.00, 3.06, 3.15 [22:48:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.44, 1.77, 1.39 [22:48:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.30, 8.83, 8.68 [22:49:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.85, 3.23, 3.33 [22:50:22] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.07, 1.49, 1.33 [22:51:32] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.14, 2.35, 1.78 [22:52:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.56, 9.64, 10.02 [22:52:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 4.00, 3.62, 3.46 [22:53:31] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.86, 1.76, 1.63 [22:54:30] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [22:54:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.78, 3.25, 3.35 [22:55:29] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.44, 1.35, 1.50 [23:00:02] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:05] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.10, 3.56, 3.44 [23:02:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.15, 3.75, 3.52 [23:04:17] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.34, 1.56, 1.26 [23:04:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.54, 3.46 [23:06:16] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.09, 1.41, 1.24 [23:10:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.37, 4.43, 3.79 [23:13:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.78, 3.67 [23:15:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.68, 4.26, 3.84 [23:20:30] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:21:11] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.58, 2.20, 1.58 [23:22:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.75, 3.96 [23:23:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.96, 1.71, 1.47 [23:24:11] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-5/±2] 13https://github.com/miraheze/ssl/compare/3d7be147dc59...0b8644dd8e4a [23:24:11] [url] Comparing 3d7be147dc59...0b8644dd8e4a · miraheze/ssl · GitHub | github.com [23:24:12] [02miraheze/ssl] 07Universal-Omega 030b8644d - Remove petrawiki.org (#523) [23:24:14] [02ssl] 07MacFan4000 closed pull request 03#523: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/523 [23:24:14] [url] Page not found · GitHub · GitHub | github.com [23:24:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.68, 3.89 [23:25:10] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.54, 1.30, 1.35 [23:26:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.45, 3.77 [23:26:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.61, 10.06, 9.33 [23:28:01] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1679s [23:28:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:45] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/en.petrawiki.org.key],File[/etc/ssl/private/files.petrawiki.org.key],File[/etc/ssl/private/translate.petrawiki.org.key],File[/etc/ssl/private/hr.petrawiki.org.key] [23:29:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.43, 3.34 [23:30:19] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/en.petrawiki.org.key],File[/etc/ssl/private/files.petrawiki.org.key],File[/etc/ssl/private/translate.petrawiki.org.key],File[/etc/ssl/private/hr.petrawiki.org.key] [23:32:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.55, 3.96, 3.80 [23:32:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.24, 9.67, 9.46 [23:33:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.27, 3.01, 3.18 [23:36:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.89, 3.84 [23:44:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.03, 5.88, 4.57 [23:44:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.75, 3.46 [23:46:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.86, 3.53 [23:48:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.19, 3.67, 3.50 [23:49:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.26, 1.99, 1.43 [23:50:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.31, 3.21, 3.36 [23:51:00] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.04, 1.66, 1.38 [23:52:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.22, 9.97, 9.52 [23:54:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 3.51, 3.99 [23:54:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.62, 9.64, 9.45 [23:55:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.85, 3.10, 3.83 [23:56:43] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:58:27] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures