[00:00:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.34, 1.72, 1.93 [00:02:44] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.72, 1.49, 1.33 [00:04:09] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 10480 MB (26% inode=96%); [00:04:44] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.07, 1.27, 1.26 [00:05:17] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 10311 MB (26% inode=96%); [00:10:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.31, 1.40, 1.70 [00:20:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.99, 1.64, 1.62 [00:22:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.27, 3.99 [00:24:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.41, 1.58, 1.61 [00:24:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.17, 3.94, 4.14 [00:28:49] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [00:33:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.17, 1.72, 1.46 [00:37:04] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.62, 1.32, 1.36 [00:40:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.48, 3.09, 3.84 [00:44:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.32, 3.61, 3.82 [00:46:27] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.50, 4.37, 2.52 [00:46:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.39, 3.36, 3.69 [00:49:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.56, 2.07, 1.80 [00:50:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.85, 3.71, 3.72 [00:51:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.50, 3.94 [00:52:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.60, 3.67 [00:55:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.73, 3.92 [00:57:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.02, 3.57, 3.85 [00:58:39] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.res [00:58:39] tname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406923294067383 seconds [00:59:04] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.67, 1.98, 1.95 [00:59:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.97, 1.57, 1.95 [01:04:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.41, 3.66, 3.68 [01:05:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.84, 4.27, 4.03 [01:07:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.84, 1.19, 1.63 [01:10:41] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.95, 1.90 [01:10:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.24, 3.67, 3.69 [01:12:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.68, 1.85, 1.88 [01:16:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.40, 3.79, 3.70 [01:18:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.70, 2.24, 2.02 [01:18:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.42, 3.95 [01:18:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.39, 3.56 [01:19:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.27, 3.71, 3.99 [01:20:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.10, 3.35, 3.83 [01:20:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.51, 3.64, 3.62 [01:21:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.22, 4.06, 4.07 [01:22:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.13, 3.70 [01:22:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.45, 3.12, 3.43 [01:23:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.76, 3.47, 3.85 [01:23:14] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.89, 1.70 [01:25:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.89, 4.09, 4.01 [01:26:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 3.70, 3.58 [01:27:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.04, 1.57, 1.62 [01:28:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.59, 3.55 [01:32:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.06, 3.75, 3.61 [01:33:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.09, 3.83, 3.99 [01:34:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.23, 2.59, 3.25 [01:34:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.59, 1.82, 1.77 [01:34:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.17, 3.17, 3.42 [01:36:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.62, 1.50, 1.66 [01:38:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.44, 1.83, 1.99 [01:40:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.53, 3.10, 3.34 [01:41:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 3.91, 3.85 [01:45:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.91, 3.89 [01:47:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.34, 4.03, 3.93 [01:48:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.04, 1.78, 1.88 [01:50:44] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.10, 1.83, 1.64 [01:50:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.64, 3.96, 3.59 [01:52:44] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.39, 1.66, 1.60 [01:53:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.21, 3.47, 3.33 [01:54:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.39, 1.82, 1.91 [01:55:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.12, 3.23, 3.25 [01:58:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.93, 3.81 [01:58:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.19, 3.64, 3.46 [02:00:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.85, 3.37, 3.38 [02:02:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.34, 1.43, 1.68 [02:04:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.59, 3.83, 3.76 [02:06:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.48, 3.46, 3.64 [02:07:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.87, 3.37, 3.87 [02:10:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.39, 3.33 [02:13:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.76, 3.99, 3.96 [02:15:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.82, 3.96, 3.96 [02:16:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.84, 3.25, 3.32 [02:20:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.12, 1.98, 1.73 [02:20:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.06, 2.86, 3.35 [02:24:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.79, 1.94, 1.79 [02:24:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.67, 3.56, 3.39 [02:24:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.38, 3.48 [02:26:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.11, 3.29, 3.30 [02:28:08] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [02:28:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.89, 3.16, 3.37 [02:29:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.74, 2.97, 3.36 [02:34:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.23, 1.83, 1.77 [02:37:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.92, 3.52 [02:37:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.48, 3.44 [02:37:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.86, 3.42, 3.35 [02:38:09] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.98, 1.91, 1.82 [02:39:39] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.13, 3.15, 3.33 [02:39:53] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.96, 2.89, 3.17 [02:40:05] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.42, 2.01, 1.86 [02:41:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.85, 3.35, 3.43 [02:43:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.22, 3.17, 3.37 [02:44:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.38, 1.72, 1.78 [02:47:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.44, 3.76, 3.34 [02:48:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.49, 1.99, 1.87 [02:49:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.30, 3.22 [02:49:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.77, 3.78, 3.41 [02:51:13] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.22, 3.61, 3.33 [02:51:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.90, 3.49 [02:52:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.36, 3.28 [02:54:13] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.09, 1.69, 1.47 [02:55:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.91, 4.00, 3.58 [02:55:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.90, 3.60 [02:56:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.04, 1.41, 1.39 [02:56:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.49, 3.34 [02:58:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.22, 3.30, 3.28 [03:01:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.74, 2.08, 1.67 [03:02:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.20, 3.09, 3.32 [03:03:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.20, 3.02, 3.33 [03:03:54] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.39, 1.90, 1.66 [03:04:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.65, 1.89, 1.97 [03:05:50] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.13, 2.02, 1.73 [03:07:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.29, 1.80, 1.68 [03:09:43] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.59, 1.68, 1.65 [03:10:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.86, 1.93 [03:12:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.55, 1.75, 1.88 [03:18:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.29, 1.97, 1.93 [03:19:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.60, 3.81, 3.35 [03:20:24] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.65, 1.58 [03:21:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.46, 3.60, 3.35 [03:22:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.39, 1.49, 1.53 [03:24:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.60, 1.87, 1.91 [03:27:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.11, 4.00, 3.55 [03:29:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.96, 3.61, 3.47 [03:33:30] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.49, 3.09, 3.28 [03:33:57] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.72, 2.41, 1.98 [03:37:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.99, 1.80, 1.84 [03:39:45] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.99, 2.91, 2.22 [03:43:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.80, 1.67, 1.85 [03:45:33] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.20, 1.72, 1.83 [03:46:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.57, 1.58, 1.68 [03:47:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.04, 1.45, 1.72 [03:49:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.27, 1.61 [03:50:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.60, 2.04, 1.85 [03:50:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.33, 3.22, 2.86 [03:52:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.97, 3.34, 2.94 [03:53:19] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.72, 1.74, 1.73 [03:54:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 3.83, 3.36 [03:55:15] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.58, 1.61, 1.68 [03:56:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.97, 1.90 [03:56:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.48, 3.28 [03:58:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 3.12, 2.86 [04:00:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.71, 2.93, 2.81 [04:02:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.48, 3.32 [04:02:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.38, 3.33, 3.20 [04:04:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.77, 3.27, 3.27 [04:16:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.53, 1.62, 1.70 [04:20:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.47, 2.44, 2.00 [04:24:21] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.80, 6.54, 4.30 [04:27:21] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7933 MB (5% inode=99%); [04:28:21] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.40, 5.12, 4.26 [04:28:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.81, 4.02, 3.38 [04:32:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.40, 1.84, 2.00 [04:32:21] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 4.37, 5.03, 4.43 [04:32:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.12, 3.48, 3.33 [04:36:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.83, 10.06, 8.89 [04:36:21] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 8.35, 6.28, 5.03 [04:36:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.47, 3.30, 3.31 [04:37:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.63, 3.38 [04:38:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.85, 9.25, 8.74 [04:38:22] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.38, 5.42, 4.86 [04:40:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.25, 1.92, 1.96 [04:40:21] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.99, 4.85, 4.72 [04:41:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.95, 2.98, 3.20 [04:42:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.91, 1.86, 1.93 [04:44:21] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.42, 5.25, 4.90 [04:44:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.36, 3.24 [04:45:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 3.82, 3.48 [04:46:21] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.71, 4.56, 4.68 [04:46:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.39, 3.26 [04:47:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.41, 3.68, 3.47 [04:48:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.27, 2.89, 3.09 [04:50:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.20, 1.98, 1.93 [04:50:21] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.46, 5.89, 5.23 [04:51:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.15, 4.04, 3.62 [04:52:21] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 11.25, 7.47, 5.85 [04:53:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.60, 3.50 [04:54:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.82, 1.97, 1.95 [04:54:21] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.15, 5.67, 5.38 [04:55:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.89, 3.84, 3.58 [04:57:10] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.res [04:57:11] tname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404033660888672 seconds [04:58:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.96, 4.94, 3.93 [04:59:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.78, 5.60, 4.22 [05:01:21] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 9443 MB (7% inode=99%); [05:02:21] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 15.71, 8.00, 6.17 [05:03:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.55, 3.89, 3.81 [05:05:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.89, 3.94 [05:07:13] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.43, 3.55, 3.65 [05:07:21] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7356 MB (5% inode=99%); [05:08:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.92, 3.87, 3.88 [05:09:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.69, 3.80 [05:10:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.57, 1.94, 1.87 [05:12:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.82, 1.86, 1.84 [05:12:59] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.81, 3.75 [05:14:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 4.31, 3.96 [05:15:43] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [05:16:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.77, 3.76, 3.80 [05:18:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.45, 2.04, 1.91 [05:19:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.68, 3.88 [05:20:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 2.68, 3.34 [05:20:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.59, 2.84, 3.39 [05:24:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 2.00, 1.96 [05:26:57] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [05:29:21] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 9617 MB (7% inode=99%); [05:30:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.31, 3.55, 3.43 [05:33:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 3.77, 3.69 [05:35:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.94, 3.78, 3.70 [05:36:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.38, 3.28, 3.39 [05:38:21] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 0.68, 3.71, 5.75 [05:39:25] [02dns] 07Universal-Omega opened pull request 03#280: Add feathercare.net zone - 13https://github.com/miraheze/dns/pull/280 [05:39:26] [url] Page not found · GitHub · GitHub | github.com [05:40:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.76, 1.80 [05:41:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.32, 2.98, 3.35 [05:42:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.54, 1.67, 1.76 [05:42:21] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.37, 2.88, 4.91 [05:44:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.27, 1.50, 1.68 [05:46:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.65, 5.25, 4.00 [05:47:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.69, 5.34, 4.20 [05:47:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.53, 4.45, 3.63 [05:48:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.56, 1.92, 1.81 [05:51:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.63, 3.70, 3.83 [05:51:53] [02dns] 07Reception123 closed pull request 03#280: Add feathercare.net zone - 13https://github.com/miraheze/dns/pull/280 [05:51:53] [url] Page not found · GitHub · GitHub | github.com [05:51:54] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/129d8c028cc0...a63cb29ff1e3 [05:51:55] [url] Comparing 129d8c028cc0...a63cb29ff1e3 · miraheze/dns · GitHub | github.com [05:51:56] [02miraheze/dns] 07Universal-Omega 03a63cb29 - Add feathercare.net zone (#280) [05:52:21] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/93d8c058fcc8...14f5117c7c71 [05:52:21] [url] Comparing 93d8c058fcc8...14f5117c7c71 · miraheze/puppet · GitHub | github.com [05:52:22] [02miraheze/puppet] 07Universal-Omega 0314f5117 - deploy-mediawiki: add arguments for branch and rebuildLC language (#2587) [05:52:24] [02puppet] 07Reception123 closed pull request 03#2587: deploy-mediawiki: add arguments for branch and rebuildLC language - 13https://github.com/miraheze/puppet/pull/2587 [05:52:24] [url] Page not found · GitHub · GitHub | github.com [05:53:13] [02puppet] 07Universal-Omega synchronize pull request 03#2606: LetsEncrypt: fix flake8 violations and add to CI - 13https://github.com/miraheze/puppet/pull/2606 [05:53:14] ... [05:53:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.82, 3.62 [05:54:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.53, 1.86, 1.84 [05:55:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.15, 3.84, 3.64 [05:55:29] [02puppet] 07Reception123 closed pull request 03#2606: LetsEncrypt: fix flake8 violations and add to CI - 13https://github.com/miraheze/puppet/pull/2606 [05:55:29] [url] Page not found · GitHub · GitHub | github.com [05:55:30] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/14f5117c7c71...e3ccd78fc1a1 [05:55:31] [url] Comparing 14f5117c7c71...e3ccd78fc1a1 · miraheze/puppet · GitHub | github.com [05:55:32] [02miraheze/puppet] 07Universal-Omega 03e3ccd78 - LetsEncrypt: fix flake8 violations and add to CI (#2606) [05:55:43] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [05:56:51] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.res [05:56:51] tname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407343864440918 seconds [05:58:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.81, 1.82 [05:58:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.92, 3.95 [05:59:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 3.87, 3.79 [06:00:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.77, 1.81 [06:02:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.12, 1.86, 1.83 [06:03:08] [02puppet] 07Universal-Omega closed pull request 03#2562: ssl-certificate: remove `--no-verify-ssl` (T8830) - 13https://github.com/miraheze/puppet/pull/2562 [06:03:08] [url] Page not found · GitHub · GitHub | github.com [06:03:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.93, 3.87 [06:05:36] [02puppet] 07Universal-Omega opened pull request 03#2607: ssl-certificate: fix print output directory - 13https://github.com/miraheze/puppet/pull/2607 [06:05:36] [url] Page not found · GitHub · GitHub | github.com [06:06:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.79, 3.73, 3.76 [06:07:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.37, 3.88, 3.82 [06:07:46] [02puppet] 07Universal-Omega opened pull request 03#2608: ssl-certificate: remove duplicate `--no-verify-ssl` - 13https://github.com/miraheze/puppet/pull/2608 [06:07:47] [url] Page not found · GitHub · GitHub | github.com [06:08:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.58, 3.70 [06:09:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 3.58, 3.71 [06:10:38] [02puppet] 07Universal-Omega synchronize pull request 03#2608: ssl-certificate: remove duplicate `--no-verify-ssl` - 13https://github.com/miraheze/puppet/pull/2608 [06:10:39] [url] Page not found · GitHub · GitHub | github.com [06:13:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.55, 4.66, 4.05 [06:14:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.74, 4.16, 3.85 [06:15:48] [02dns] 07Universal-Omega opened pull request 03#281: Create .gitattributes - 13https://github.com/miraheze/dns/pull/281 [06:15:48] [url] Page not found · GitHub · GitHub | github.com [06:16:18] [02dns] 07Universal-Omega synchronize pull request 03#281: Create .gitattributes - 13https://github.com/miraheze/dns/pull/281 [06:16:18] ... [06:16:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.02, 3.12, 3.81 [06:16:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.57, 3.44, 3.62 [06:17:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.96, 3.93 [06:21:00] [02dns] 07Universal-Omega synchronize pull request 03#281: Create .gitattributes - 13https://github.com/miraheze/dns/pull/281 [06:21:00] [url] Page not found · GitHub · GitHub | github.com [06:21:35] [02dns] 07Universal-Omega edited pull request 03#281: Create .gitattributes - 13https://github.com/miraheze/dns/pull/281 [06:21:36] ... [06:21:45] [02dns] 07Universal-Omega edited pull request 03#281: Set display language of all zone files - 13https://github.com/miraheze/dns/pull/281 [06:21:45] ... [06:21:55] [02dns] 07Universal-Omega edited pull request 03#281: Set display language of all DNS zone files - 13https://github.com/miraheze/dns/pull/281 [06:22:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.53, 3.00, 3.40 [06:24:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.18, 2.83, 3.35 [06:27:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.11, 2.81, 3.32 [06:52:44] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.96, 1.31 [06:54:44] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.83, 1.57, 1.24 [06:56:26] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [06:58:02] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.70, 2.25, 1.43 [07:00:02] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.06, 1.81, 1.36 [07:02:03] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.04, 1.51, 1.30 [07:07:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.14, 3.55, 3.32 [07:09:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.84, 3.02, 3.16 [07:13:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.55, 3.77, 3.55 [07:17:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.71, 4.28, 3.81 [07:18:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 3.76, 3.26 [07:23:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.71, 3.88, 3.86 [07:24:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.89, 3.53 [07:25:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.46, 4.43, 4.03 [07:26:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 4.28, 3.73 [07:27:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.83, 3.86 [07:28:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.90, 1.64, 1.33 [07:30:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.23, 1.39, 1.27 [07:32:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.87, 3.78, 3.69 [07:33:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.77, 3.43, 3.17 [07:35:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 4.07, 3.94 [07:36:07] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.95, 1.74, 1.45 [07:37:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.13, 3.11 [07:37:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 3.99, 3.93 [07:38:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.41, 2.10, 1.63 [07:40:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.45, 1.87, 1.60 [07:41:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.92, 4.07, 3.95 [07:42:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.21, 1.66, 1.56 [07:44:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.04, 3.16, 3.40 [07:47:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.78, 3.79, 3.91 [07:48:11] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (START) [07:48:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:49:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.82, 3.90 [07:50:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.87, 1.63, 1.57 [07:52:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.46, 1.77, 1.62 [07:54:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.61, 3.64, 3.27 [07:56:04] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.res [07:56:04] tname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406260251998901 seconds [07:56:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.48, 3.25, 3.18 [07:56:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.55, 4.08, 3.63 [07:58:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.00, 1.76, 1.74 [07:58:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.69, 3.92, 3.63 [08:01:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 3.66, 3.34 [08:02:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.27, 1.62, 1.69 [08:03:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.32, 3.19, 3.21 [08:04:43] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (END - exit=2) [08:04:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:05:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.89, 3.51, 3.91 [08:06:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.29, 3.85, 3.65 [08:07:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.46, 4.42, 4.20 [08:07:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.92, 3.69, 3.41 [08:08:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.49, 3.76, 3.65 [08:09:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.35, 3.26, 3.29 [08:11:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.09, 3.33, 3.82 [08:12:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.72, 3.01, 3.35 [08:14:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.42, 1.84, 1.71 [08:16:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 13.41, 7.57, 3.88 [08:17:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.25, 2.57, 3.29 [08:20:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.73, 4.49, 3.57 [08:21:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.89, 4.15, 3.72 [08:22:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.83, 3.43 [08:23:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.89, 3.56, 3.55 [08:24:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.05, 3.37, 3.30 [08:27:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.59, 3.09, 3.37 [08:27:27] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.03, 1.89, 1.50 [08:29:21] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.86, 1.86, 1.54 [08:30:33] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [08:30:33] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:33:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.60, 1.39, 1.43 [08:43:47] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.89, 1.92, 1.66 [08:47:34] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.14, 1.61, 1.60 [08:47:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.48, 3.32 [08:51:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.32, 3.70, 3.42 [08:52:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.01, 1.53, 1.90 [08:54:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.63, 2.38, 2.14 [08:55:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.56, 3.42 [08:57:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.78, 2.90, 3.20 [09:05:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.38, 3.12 [09:05:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.80, 3.77, 3.47 [09:07:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.86, 3.22, 3.10 [09:07:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.67, 3.48 [09:11:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.38, 3.08, 3.29 [09:12:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.28, 1.61, 2.00 [09:17:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.66, 1.85, 1.61 [09:18:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.06, 1.81, 1.96 [09:21:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.84, 1.98, 1.73 [09:25:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.27, 1.89, 1.74 [09:28:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.99, 1.95, 2.00 [09:28:33] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [09:31:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.07, 1.98, 1.90 [09:32:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.88, 1.94 [09:41:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.52, 3.93, 3.20 [09:41:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.02, 3.32, 2.89 [09:42:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.87, 1.95 [09:43:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.84, 3.68, 3.19 [09:43:25] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.90, 2.90, 2.77 [09:47:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.93, 1.31, 1.61 [09:47:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 3.18, 3.11 [09:48:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.37, 1.98, 1.96 [09:53:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.05, 3.48, 3.27 [09:53:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.83, 1.59, 1.62 [09:53:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.35, 2.83 [09:55:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.49, 1.43, 1.55 [09:55:25] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.95, 3.06, 2.78 [09:56:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.80, 1.97, 2.00 [09:57:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.82, 3.12, 3.16 [09:58:33] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [09:59:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.18, 2.52, 1.96 [10:03:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.78, 2.00, 1.87 [10:04:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.15, 1.84, 1.91 [10:05:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.48, 2.14, 1.94 [10:06:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.68, 1.76, 1.87 [10:07:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.23, 1.77, 1.82 [10:11:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.23, 1.48, 1.68 [10:16:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.22, 1.62, 1.93 [10:17:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.77, 3.21, 3.13 [10:18:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.50, 2.01, 1.91 [10:18:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.34, 1.87, 1.98 [10:19:30] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.71, 2.70, 2.96 [10:20:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.93, 1.88, 1.97 [10:26:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.67, 1.94, 1.95 [10:26:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.90, 2.33, 2.08 [10:27:34] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.79, 1.91, 1.76 [10:28:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.45, 1.99, 1.99 [10:29:28] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.65, 1.47, 1.61 [10:40:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.42, 2.18, 1.96 [10:42:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.53, 1.58, 1.70 [10:42:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.07, 1.86, 1.88 [10:43:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.72, 3.26 [10:44:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.49, 2.10, 1.96 [10:45:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.36, 3.21, 3.13 [10:50:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.54, 1.97, 1.79 [10:53:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.26, 3.13 [10:55:03] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [10:55:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.73, 3.92, 3.35 [10:57:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.08, 3.45, 3.25 [10:58:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.92, 1.90 [11:00:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.04, 1.99, 1.93 [11:00:37] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.76, 2.09, 1.66 [11:01:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.22, 3.12, 3.18 [11:02:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.90, 1.92, 1.91 [11:02:31] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.18, 1.76, 1.59 [11:04:12] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:04:25] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.40, 2.00, 1.70 [11:05:41] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [11:06:06] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.371 second response time [11:06:59] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.85, 7.04, 6.47 [11:08:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.85, 1.93, 1.75 [11:08:57] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.88, 6.27, 6.25 [11:12:00] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.96, 2.15, 1.88 [11:13:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.98, 3.33, 3.10 [11:13:53] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.61, 1.96, 1.84 [11:15:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.66, 3.16, 2.75 [11:15:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.64, 3.25 [11:16:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.80, 1.81 [11:16:58] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 2.85, 2.68 [11:17:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.43, 3.22 [11:19:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.53, 3.16, 3.15 [11:21:28] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.62, 1.24, 1.57 [11:24:53] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.res [11:24:53] tname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404387474060059 seconds [11:28:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.58, 1.87, 1.96 [11:29:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 4.20, 3.61 [11:29:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.90, 2.10, 1.84 [11:32:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.79, 3.80, 3.26 [11:34:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.68, 3.60, 3.26 [11:34:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.55, 3.35, 3.16 [11:36:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.75, 3.31, 3.20 [11:37:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.52, 3.71, 3.68 [11:39:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.81, 3.70 [11:41:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.80, 3.73 [11:42:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.09, 3.44, 3.28 [11:42:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.34, 3.54, 3.29 [11:46:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.31, 2.02, 1.96 [11:46:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.97, 3.19, 3.22 [11:46:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.73, 3.08, 3.16 [11:50:25] Was in car [11:51:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.60, 2.65, 3.22 [11:51:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.28, 1.54, 1.95 [11:51:56] Echo [11:52:07] .mh SAL [11:52:07] https://meta.miraheze.org/wiki/SAL [11:53:00] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (START) [11:53:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:53:24] I'm not sure why so many echo notifications [11:53:28] It'll clear [11:53:42] We really need to make all faster somehow [11:55:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.40, 2.07, 2.06 [11:56:36] I filed a task [12:00:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.73, 1.86, 1.99 [12:05:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.12, 1.78, 1.99 [12:08:52] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.69, 3.17, 2.01 [12:10:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.02, 1.81, 1.87 [12:12:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.46, 1.68, 1.82 [12:13:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.91, 1.26, 1.65 [12:16:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.75, 3.40 [12:16:52] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.03, 2.98, 2.48 [12:18:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.42, 1.98, 1.88 [12:18:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.52, 3.35 [12:20:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.22, 3.37, 3.31 [12:22:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.61, 3.27 [12:23:20] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 314 bytes in 1.773 second response time [12:23:43] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:46] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.21, 10.99, 8.30 [12:23:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.98, 9.75, 7.42 [12:23:48] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.52, 3.42 [12:24:32] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [12:24:48] It's not workers [12:24:54] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.81, 3.22, 3.16 [12:25:26] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [12:25:44] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.07, 10.00, 8.29 [12:25:47] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.28, 8.33, 7.18 [12:25:51] php-fpm is out on 11 [12:25:53] 101 [12:26:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.65, 1.91, 1.93 [12:26:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 9.81, 5.39, 4.08 [12:26:17] Grafana was laggy [12:26:42] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:00] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:10] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:13] Fun [12:27:17] I'm mobile [12:27:30] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 3.618 second response time [12:27:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 9.67, 5.92, 4.02 [12:28:14] the dashboards are slow [12:28:37] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14656 bytes in 0.026 second response time [12:28:51] We have 43 idle workers [12:28:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 9.32, 7.14, 4.82 [12:29:00] But some servers exhausted [12:29:01] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14642 bytes in 5.419 second response time [12:29:10] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22337 bytes in 2.810 second response time [12:29:26] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 13 backends are healthy [12:29:52] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14650 bytes in 1.272 second response time [12:29:53] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14650 bytes in 1.071 second response time [12:30:19] mwtask is running low on disk [12:30:55] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14650 bytes in 0.017 second response time [12:31:01] why so many warnings today? [12:33:13] ugochimobi: everything blowing up [12:33:20] Reception is running task out of disk [12:33:28] ah [12:33:29] And the job queue wants to cry [12:33:34] And normal load isn't helping [12:33:40] ah [12:34:09] what are you (guys) going to do? [12:34:33] JobQueue I'm trying to solve [12:34:38] Can't do much about load [12:34:46] Don't have permissions to stop reception [12:34:53] ah [12:35:06] JohnLewis welcome [12:35:20] JohnLewis: hi, php workers just ran out everywhere [12:35:25] Task is low on disk I've seen [12:35:37] And the job queue is giving up [12:36:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.27, 1.77, 1.80 [12:37:33] My general impression seems to be job queue has become unmanageable after the recent change to it as it seems since that change was pushed, it’s daily there is a problem [12:38:06] JohnLewis: it was bad anyway but Krinkle's change has not done anything [12:38:12] Reverting it is probably a good idea [12:38:25] if you want to push that then feel free [12:38:36] I think we need to reconsider how to approach that [12:40:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.95, 1.85, 1.82 [12:43:16] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.36, 7.12, 6.38 [12:45:13] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.19, 6.44, 6.22 [12:45:14] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.89, 1.66, 1.60 [12:45:26] JohnLewis: can you press button if I revert [12:46:27] [02puppet] 07RhinosF1 opened pull request 03#2609: Revert "jobrunner: set maxjobs to 1" - 13https://github.com/miraheze/puppet/pull/2609 [12:46:28] [url] Page not found · GitHub · GitHub | github.com [12:47:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.45, 1.86, 1.68 [12:47:30] PR sent [12:48:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.49, 2.01, 1.86 [12:48:10] Yeah [12:48:22] JohnLewis: please press [12:48:27] [02puppet] 07JohnFLewis closed pull request 03#2609: Revert "jobrunner: set maxjobs to 1" - 13https://github.com/miraheze/puppet/pull/2609 [12:48:27] [url] Page not found · GitHub · GitHub | github.com [12:48:28] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e3ccd78fc1a1...7917c485fa8e [12:48:29] [url] Comparing e3ccd78fc1a1...7917c485fa8e · miraheze/puppet · GitHub | github.com [12:48:30] [02miraheze/puppet] 07RhinosF1 037917c48 - Revert "jobrunner: set maxjobs to 1 (#2581)" (#2609) [12:48:30] I can deploy to jobrunner [12:48:55] .in 10minutes check jobrunner puppet [12:48:55] RhinosF1: Okay, will remind at 2022-05-29 - 13:58:55BST [12:49:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.56, 1.79, 1.68 [12:50:44] It's all servers [12:51:05] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.49, 6.93, 6.47 [12:51:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.96, 2.56, 1.97 [12:51:42] Puppet should notify [12:52:53] I guess we wait half an hour [12:53:00] Hopefully jobs will then calm down [12:53:02] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.15, 6.66, 6.42 [12:53:07] I'll clean up my screens this evening [12:54:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.98, 1.92 [12:54:22] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.res [12:54:22] tname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.403839111328125 seconds [12:56:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.33, 2.09, 1.97 [12:58:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.85, 1.97, 1.94 [12:58:57] RhinosF1: check jobrunner puppet [13:02:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.30, 2.02, 1.95 [13:10:04] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.98, 1.96, 1.97 [13:10:41] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [13:16:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.18, 1.87, 1.92 [13:21:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.37, 1.67, 2.00 [13:22:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.71, 3.24, 3.88 [13:24:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 0.66, 1.71, 1.92 [13:24:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.37, 3.92, 4.04 [13:28:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 0.27, 0.98, 1.57 [13:31:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.86, 2.02, 1.92 [13:33:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.10, 1.57, 1.76 [13:33:41] PROBLEM - mwtask111 Disk Space on mwtask111 is WARNING: DISK WARNING - free space: / 6980 MB (10% inode=86%); [13:35:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.00, 1.46, 1.70 [13:35:41] RECOVERY - mwtask111 Disk Space on mwtask111 is OK: DISK OK - free space: / 21460 MB (32% inode=88%); [13:37:26] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:38:02] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:38:02] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:38:03] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:38:40] why only 111 [13:38:46] * RhinosF1 looks [13:39:08] It's all [13:39:13] It'll recover [13:39:25] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14656 bytes in 0.307 second response time [13:39:56] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.383 second response time [13:39:56] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14650 bytes in 0.015 second response time [13:40:03] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14656 bytes in 0.336 second response time [13:40:37] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw111 [13:42:37] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 13 backends are healthy [13:45:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.09, 1.91, 1.74 [13:47:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.69, 1.77, 1.70 [13:49:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.67, 2.12, 1.84 [13:50:12] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.69, 1.82, 1.26 [13:51:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.36, 1.78, 1.75 [13:52:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.19, 1.54, 1.93 [13:52:08] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.69, 1.38, 1.16 [13:53:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.74, 1.44, 1.63 [13:54:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.00, 2.08, 2.08 [14:00:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.97, 3.67, 3.99 [14:02:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.09, 1.67, 1.94 [14:02:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.36, 3.99, 4.07 [14:02:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.62, 3.79, 4.00 [14:04:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.35, 1.84, 1.96 [14:04:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.68, 3.92 [14:06:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.88, 1.94, 1.99 [14:06:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.61, 3.87 [14:08:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.20, 2.09, 2.04 [14:10:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.84, 1.65, 1.89 [14:10:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.44, 3.86 [14:12:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.57, 2.00, 1.98 [14:12:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.22, 3.97, 4.01 [14:14:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 3.60, 3.85 [14:16:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.61, 1.86, 1.94 [14:22:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.58, 2.80, 3.36 [14:22:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.53, 2.81, 3.33 [14:23:50] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [14:24:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 5.02, 2.39, 2.08 [14:25:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.47, 3.96 [14:26:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.03, 3.70, 3.59 [14:27:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.65, 3.96 [14:27:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.50, 1.83, 1.54 [14:28:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 3.56, 3.47 [14:29:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.84, 3.74, 3.95 [14:30:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.24, 3.21, 3.44 [14:30:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.39, 3.59, 3.49 [14:32:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.04, 3.88, 3.62 [14:33:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 3.90, 3.94 [14:33:42] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.43, 7.01, 6.22 [14:34:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 3.91, 3.66 [14:37:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.91, 3.96 [14:37:35] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.37, 6.64, 6.26 [14:39:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.80, 4.12, 4.01 [14:40:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.03, 2.94, 3.30 [14:42:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.67, 4.26, 3.76 [14:43:27] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 8.00, 7.14, 6.52 [14:44:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.90, 4.60, 3.85 [14:45:24] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.07, 6.47, 6.36 [14:46:43] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (END - exit=0) [14:46:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:52:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.69, 3.64, 3.80 [14:54:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.83, 3.60, 3.80 [14:56:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.13, 4.10, 3.95 [14:56:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.35, 4.12, 3.92 [14:58:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.65, 3.80 [14:58:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.38, 3.78, 3.82 [15:00:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.47, 4.17, 3.96 [15:00:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.88, 4.57, 4.08 [15:02:27] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/9518a175aae3 [15:02:28] [02miraheze/ssl] 07MacFan4000 039518a17 - add feathercare.net [15:02:30] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [15:02:30] [url] Page not found · GitHub · GitHub | github.com [15:03:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.05, 1.70, 1.94 [15:03:40] [02ssl] 07MacFan4000 opened pull request 03#534: add 2 certs - 13https://github.com/miraheze/ssl/pull/534 [15:03:41] [url] Page not found · GitHub · GitHub | github.com [15:07:15] PROBLEM - db111 APT on db111 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [15:09:17] PROBLEM - db111 APT on db111 is CRITICAL: APT CRITICAL: 51 packages available for upgrade (6 critical updates). [15:11:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.29, 2.05, 1.93 [15:12:56] [02ssl] 07MacFan4000 edited pull request 03#534: add feathercare.net cert - 13https://github.com/miraheze/ssl/pull/534 [15:12:57] [url] Page not found · GitHub · GitHub | github.com [15:13:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.95, 1.96, 1.91 [15:14:20] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9518a175aae3...48324c0dcacf [15:14:21] [url] Comparing 9518a175aae3...48324c0dcacf · miraheze/ssl · GitHub | github.com [15:14:22] [02miraheze/ssl] 07MacFan4000 0348324c0 - Update certs.yaml [15:14:23] [02ssl] 07MacFan4000 synchronize pull request 03#534: add feathercare.net cert - 13https://github.com/miraheze/ssl/pull/534 [15:14:24] [url] Page not found · GitHub · GitHub | github.com [15:14:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.32, 3.76, 4.00 [15:15:01] [02ssl] 07MacFan4000 closed pull request 03#534: add feathercare.net cert - 13https://github.com/miraheze/ssl/pull/534 [15:15:01] [url] Page not found · GitHub · GitHub | github.com [15:15:02] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/aa6d0d18918c...68725371dcd7 [15:15:03] [url] Comparing aa6d0d18918c...68725371dcd7 · miraheze/ssl · GitHub | github.com [15:15:04] [02miraheze/ssl] 07MacFan4000 036872537 - add feathercare.net cert (#534) [15:15:05] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [15:15:07] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [15:15:07] ... [15:15:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.16, 1.97, 1.92 [15:16:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.22, 4.04, 4.08 [15:17:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.74, 1.91, 1.90 [15:18:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.12, 3.82, 4.00 [15:20:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.85, 3.46, 3.94 [15:22:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.24, 1.64, 2.00 [15:24:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.56, 1.93, 2.05 [15:25:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.38, 1.42, 1.66 [15:32:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.02, 3.44, 3.94 [15:35:04] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.56, 3.09, 3.37 [15:36:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.80, 3.96, 3.74 [15:38:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.60, 3.62 [15:43:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.41, 3.96, 3.59 [15:44:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.80, 4.68, 4.16 [15:44:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.99, 4.01, 3.69 [15:45:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.39, 3.64, 3.51 [15:49:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.29, 4.38, 3.81 [15:50:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.86, 3.72 [15:52:07] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.69, 3.58, 3.09 [15:52:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.89, 4.13, 3.85 [15:53:18] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.res [15:53:18] tname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407266616821289 seconds [15:54:01] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.52, 3.20, 3.01 [15:57:51] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.17, 3.57, 3.19 [15:59:45] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.55, 3.38, 3.16 [16:01:40] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.77, 3.10, 3.08 [16:03:32] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.02, 1.64, 1.93 [16:07:37] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.39, 7.45, 6.07 [16:09:25] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.50, 1.83, 1.88 [16:09:36] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.50, 7.00, 6.07 [16:11:25] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.37, 1.62, 1.80 [16:11:36] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.41, 6.32, 5.93 [16:11:52] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.87, 1.63, 1.51 [16:13:25] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 11.10, 8.73, 4.65 [16:13:52] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.18, 1.77, 1.57 [16:15:52] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.27, 1.63, 1.55 [16:17:33] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.94, 6.81, 6.26 [16:21:52] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.81, 1.69, 1.60 [16:23:03] RECOVERY - wikipedia.yogfront.ooo - reverse DNS on sslhost is OK: SSL OK - wikipedia.yogfront.ooo reverse DNS resolves to cp21.miraheze.org - CNAME OK [16:23:30] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.91, 6.68, 6.43 [16:23:52] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.19, 1.54, 1.55 [16:29:27] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.90, 7.39, 6.81 [16:29:52] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.76, 1.73, 1.63 [16:31:26] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.87, 6.37, 6.51 [16:31:52] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.31, 1.50, 1.56 [16:43:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.49, 3.67, 3.98 [16:45:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 3.92, 4.04 [16:45:55] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.51, 1.06 [16:47:55] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.44, 1.85, 1.24 [16:57:55] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.94, 1.65 [17:01:35] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.24, 1.73, 1.51 [17:03:35] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.15, 1.53, 1.46 [17:03:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.33, 2.98, 1.82 [17:07:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.66, 1.82, 1.61 [17:07:55] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.54, 1.66, 1.63 [17:09:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.36, 1.60, 1.55 [17:13:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/1a97f2767ade [17:13:46] [url] Page not found · GitHub · GitHub | github.com [17:13:48] [02miraheze/mw-config] 07Universal-Omega 031a97f27 - Add `$smwgNamespacesWithSemanticLinks` to ManageWikiNamespaces [17:13:49] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [17:13:49] ... [17:13:51] [02mw-config] 07Universal-Omega opened pull request 03#4717: Add `$smwgNamespacesWithSemanticLinks` to ManageWikiNamespaces - 13https://github.com/miraheze/mw-config/pull/4717 [17:13:51] [url] Page not found · GitHub · GitHub | github.com [17:14:55] miraheze/mw-config - Universal-Omega the build passed. [17:18:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1a97f2767ade...2a4bdbb50759 [17:18:33] [url] Comparing 1a97f2767ade...2a4bdbb50759 · miraheze/mw-config · GitHub | github.com [17:18:33] [02miraheze/mw-config] 07Universal-Omega 032a4bdbb - Update SemanticMediaWiki.php [17:18:35] [02mw-config] 07Universal-Omega synchronize pull request 03#4717: Add `$smwgNamespacesWithSemanticLinks` to ManageWikiNamespaces - 13https://github.com/miraheze/mw-config/pull/4717 [17:18:35] [url] Page not found · GitHub · GitHub | github.com [17:19:13] [02mw-config] 07Universal-Omega edited pull request 03#4717: Add `$smwgNamespacesWithSemanticLinks` to ManageWikiNamespaces - 13https://github.com/miraheze/mw-config/pull/4717 [17:19:13] [url] Page not found · GitHub · GitHub | github.com [17:19:34] miraheze/mw-config - Universal-Omega the build has errored. [17:19:55] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.84, 2.26, 1.89 [17:20:05] [02mw-config] 07Universal-Omega synchronize pull request 03#4717: Add `$smwgNamespacesWithSemanticLinks` to ManageWikiNamespaces - 13https://github.com/miraheze/mw-config/pull/4717 [17:20:05] [url] Page not found · GitHub · GitHub | github.com [17:20:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2a4bdbb50759...a9eb55769d82 [17:20:07] [url] Comparing 2a4bdbb50759...a9eb55769d82 · miraheze/mw-config · GitHub | github.com [17:20:08] [02miraheze/mw-config] 07Universal-Omega 03a9eb557 - Update SemanticMediaWiki.php [17:21:03] miraheze/mw-config - Universal-Omega the build passed. [17:28:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.84, 1.59, 1.48 [17:29:55] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.97, 1.96 [17:30:07] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.82, 1.32, 1.39 [17:38:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.46, 3.12, 3.86 [17:39:25] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.63, 1.79, 1.99 [17:40:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.50, 3.90 [17:42:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.91, 3.70, 3.93 [17:43:25] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.02, 1.71, 1.89 [17:44:46] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 0.83, 2.04, 1.58 [17:45:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.23, 3.89 [17:46:44] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.67, 1.95, 1.59 [17:46:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.18, 4.25, 4.04 [17:47:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.44, 3.57, 3.92 [17:47:55] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.53, 1.91, 1.84 [17:48:41] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.90, 1.57, 1.50 [17:49:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.65, 3.44, 3.82 [17:50:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.99, 3.91, 3.93 [17:51:25] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.81, 2.00, 2.00 [17:53:25] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.54, 2.22, 2.08 [17:55:25] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.57, 1.94, 1.99 [17:57:25] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.82, 2.39, 2.16 [17:57:37] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.67, 1.84, 1.50 [17:58:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.77, 4.16, 3.99 [17:59:55] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.66, 1.88, 1.94 [18:00:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.86, 3.75, 3.87 [18:01:34] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.61, 1.82, 1.57 [18:01:48] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.31, 6.58, 5.88 [18:02:22] [02mw-config] 07Universal-Omega closed pull request 03#4717: Add `$smwgNamespacesWithSemanticLinks` to ManageWikiNamespaces - 13https://github.com/miraheze/mw-config/pull/4717 [18:02:22] [url] Page not found · GitHub · GitHub | github.com [18:02:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/c48775dcf573...265e5c0cdfca [18:02:24] [url] Comparing c48775dcf573...265e5c0cdfca · miraheze/mw-config · GitHub | github.com [18:02:25] [02miraheze/mw-config] 07Universal-Omega 03265e5c0 - Add `$smwgNamespacesWithSemanticLinks` to ManageWikiNamespaces (#4717) [18:02:26] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [18:02:28] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [18:02:28] ... [18:03:30] miraheze/mw-config - Universal-Omega the build passed. [18:03:48] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.73, 6.14, 5.80 [18:05:31] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.84, 1.56, 1.54 [18:05:55] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.55, 1.91, 1.90 [18:07:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.97, 4.70, 4.08 [18:08:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.52, 4.57, 4.14 [18:15:42] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 10.96, 7.83, 6.59 [18:19:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.55, 3.57, 3.96 [18:19:40] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.33, 7.46, 6.79 [18:21:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.71, 4.23, 4.16 [18:21:40] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.10, 7.76, 6.98 [18:23:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.61, 7.41, 6.96 [18:24:06] !log [@mwtask111] starting deploy of {'config': True} to all [18:24:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:18] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 11s [18:24:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.54, 3.65, 3.98 [18:25:22] !log [@test101] starting deploy of {'config': True} to all [18:25:23] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [18:25:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:38] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.11, 6.72, 6.77 [18:28:27] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.80, 1.81, 1.35 [18:30:27] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.67, 1.37, 1.24 [18:32:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.77, 4.38, 4.08 [18:36:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.16, 3.64, 3.89 [18:37:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/e3daa24d1b3e [18:37:28] [02miraheze/mw-config] 07Universal-Omega 03e3daa24 - Update SemanticMediaWiki.php [18:37:29] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [18:37:30] [url] Page not found · GitHub · GitHub | github.com [18:38:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.02, 4.09, 4.01 [18:41:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.68, 3.99 [18:47:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.33, 2.02, 1.35 [18:48:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.49, 3.81 [18:49:34] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.02, 1.59, 1.26 [18:49:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.61, 3.95 [18:50:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.39, 3.63, 3.81 [18:51:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e3daa24d1b3e...38dc405e9bb1 [18:51:24] [url] Comparing e3daa24d1b3e...38dc405e9bb1 · miraheze/mw-config · GitHub | github.com [18:51:25] [02miraheze/mw-config] 07Universal-Omega 0338dc405 - Update .phpcs.xml [18:51:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.16, 3.50 [18:51:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/38dc405e9bb1...80d8a1d13470 [18:51:42] [url] Comparing 38dc405e9bb1...80d8a1d13470 · miraheze/mw-config · GitHub | github.com [18:51:43] [02miraheze/mw-config] 07Universal-Omega 0380d8a1d - Update MirahezeFunctions.php [18:51:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.19, 4.09, 4.07 [18:53:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/80d8a1d13470...1cc26c1e86d4 [18:53:15] [url] Comparing 80d8a1d13470...1cc26c1e86d4 · miraheze/mw-config · GitHub | github.com [18:53:16] [02miraheze/mw-config] 07Universal-Omega 031cc26c1 - Update ManageWikiSettings.php [18:53:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.44, 3.57 [18:55:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.59, 3.62, 3.91 [18:56:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.64, 3.19, 3.61 [18:58:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1cc26c1e86d4...8d5f7dcfb8b7 [18:58:11] [url] Comparing 1cc26c1e86d4...8d5f7dcfb8b7 · miraheze/mw-config · GitHub | github.com [18:58:12] [02miraheze/mw-config] 07Universal-Omega 038d5f7dc - Update GlobalSettings.php [18:59:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.29, 3.44, 3.47 [19:01:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 3.47, 3.48 [19:05:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.51, 3.11, 3.35 [19:06:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.58, 3.06, 3.39 [19:11:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.50, 3.77, 3.66 [19:13:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.62, 3.62 [19:15:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.68, 4.11, 3.78 [19:18:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.11, 3.80, 3.49 [19:18:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.71, 3.51 [19:20:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.50, 3.42 [19:20:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.39, 3.21, 3.35 [19:21:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.74, 3.81 [19:22:11] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.32, 3.32, 3.36 [19:23:25] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.32, 1.75, 1.99 [19:23:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.39, 4.48, 4.05 [19:25:25] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.39, 1.99, 2.04 [19:26:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.57, 4.99, 4.05 [19:26:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.33, 4.82, 3.97 [19:27:25] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.47, 1.80, 1.97 [19:30:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.67, 3.99, 3.81 [19:32:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.84, 4.42, 3.99 [19:35:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.57, 3.75, 3.89 [19:36:03] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.63, 2.46, 1.74 [19:37:25] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.49, 2.01, 1.94 [19:38:01] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.45, 2.00, 1.65 [19:39:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.67, 4.09, 3.96 [19:40:00] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.21, 1.98, 1.67 [19:41:02] [02puppet] 07Reception123 closed pull request 03#2607: ssl-certificate: fix print output directory - 13https://github.com/miraheze/puppet/pull/2607 [19:41:02] [url] Page not found · GitHub · GitHub | github.com [19:41:04] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7917c485fa8e...8aecb586a0c9 [19:41:04] [url] Comparing 7917c485fa8e...8aecb586a0c9 · miraheze/puppet · GitHub | github.com [19:41:05] [02miraheze/puppet] 07Universal-Omega 038aecb58 - ssl-certificate: fix print output directory (#2607) [19:41:11] [02puppet] 07Reception123 closed pull request 03#2608: ssl-certificate: remove duplicate `--no-verify-ssl` - 13https://github.com/miraheze/puppet/pull/2608 [19:41:11] [url] Page not found · GitHub · GitHub | github.com [19:41:12] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8aecb586a0c9...c0e401860bdf [19:41:12] [url] Comparing 8aecb586a0c9...c0e401860bdf · miraheze/puppet · GitHub | github.com [19:41:14] [02miraheze/puppet] 07Universal-Omega 03c0e4018 - ssl-certificate: remove duplicate `--no-verify-ssl` (#2608) [19:41:22] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.77, 1.91, 1.33 [19:41:58] [discord] CosmicAlpha: hello [19:43:22] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.68, 1.41, 1.22 [19:46:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.10, 3.77, 3.95 [19:48:03] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.67, 6.81, 6.07 [19:50:02] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.67, 7.38, 6.36 [19:51:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.81, 3.99 [19:53:52] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.92, 1.81, 1.87 [19:54:00] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.73, 7.91, 6.86 [19:54:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 3.72, 3.82 [19:57:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.22, 3.70, 3.84 [19:57:52] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.16, 1.94, 1.90 [19:59:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.83, 3.66, 3.80 [19:59:49] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [19:59:50] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:59:58] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.44, 6.71, 6.71 [20:00:16] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.85, 1.51, 1.21 [20:01:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.15, 3.70, 3.79 [20:02:15] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.47, 1.13, 1.11 [20:02:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 3.84, 3.92 [20:03:07] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.54, 3.32, 2.97 [20:03:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.61, 3.37, 3.66 [20:03:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.15, 7.32, 6.98 [20:05:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.89, 3.72, 3.74 [20:06:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.62, 4.18, 4.01 [20:07:07] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.13, 2.91, 2.91 [20:07:54] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.88, 5.94, 6.54 [20:08:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.77, 3.88 [20:11:52] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.25, 1.69, 1.97 [20:12:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.15, 4.07, 3.95 [20:15:22] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.40, 1.54, 1.04 [20:15:52] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.79, 1.19, 1.70 [20:17:22] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.89, 1.24, 0.99 [20:24:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.69, 3.96 [20:25:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.59, 3.57, 4.00 [20:26:11] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.21, 1.84, 1.48 [20:28:11] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.53, 1.39, 1.35 [20:28:52] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [20:29:25] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.48, 1.75, 2.00 [20:31:25] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.02, 1.95, 2.05 [20:31:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 3.66, 3.83 [20:32:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.82, 3.82 [20:33:25] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.15, 1.66, 1.93 [20:33:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.18, 3.55, 3.77 [20:34:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.26, 3.73, 3.80 [20:35:41] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [20:36:23] Here [20:37:18] It's links [20:37:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8d5f7dcfb8b7...865eac675127 [20:37:30] [url] Comparing 8d5f7dcfb8b7...865eac675127 · miraheze/mw-config · GitHub | github.com [20:37:31] [02miraheze/mw-config] 07Universal-Omega 03865eac6 - Update MirahezeFunctions.php [20:37:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.14, 4.17, 3.96 [20:38:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/865eac675127...337a6d53a508 [20:38:34] [url] Comparing 865eac675127...337a6d53a508 · miraheze/mw-config · GitHub | github.com [20:38:35] [02miraheze/mw-config] 07Universal-Omega 03337a6d5 - Update MirahezeFunctions.php [20:38:56] CosmicAlpha: access? [20:39:21] I can't see any imports [20:39:31] [02mw-config] 07Universal-Omega opened pull request 03#4718: Improve loading of extensions - 13https://github.com/miraheze/mw-config/pull/4718 [20:39:32] [url] Page not found · GitHub · GitHub | github.com [20:39:56] I'll start a run again [20:40:06] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (START) [20:40:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:40:16] :( [20:40:42] miraheze/mw-config - Universal-Omega the build has errored. [20:40:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.05, 4.62, 4.11 [20:43:03] JohnLewis: revert didn't do much [20:43:51] refreshLinks is the job this time though [20:44:05] JohnLewis: yep [20:44:15] which id only expect in a import [20:44:22] We really need to stop spikes [20:44:32] back in a bit [20:45:25] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.93, 1.30, 1.62 [20:45:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/337a6d53a508...c6659c9e52aa [20:45:36] [url] Comparing 337a6d53a508...c6659c9e52aa · miraheze/mw-config · GitHub | github.com [20:45:37] [02miraheze/mw-config] 07Universal-Omega 03c6659c9 - Update SettingsTest.php [20:45:39] [02mw-config] 07Universal-Omega synchronize pull request 03#4718: Improve loading of extensions - 13https://github.com/miraheze/mw-config/pull/4718 [20:45:39] [url] Page not found · GitHub · GitHub | github.com [20:45:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c6659c9e52aa...d4c45528d07a [20:45:57] [url] Comparing c6659c9e52aa...d4c45528d07a · miraheze/mw-config · GitHub | github.com [20:45:58] [02miraheze/mw-config] 07Universal-Omega 03d4c4552 - Update ManageWikiTestCase.php [20:46:00] [02mw-config] 07Universal-Omega synchronize pull request 03#4718: Improve loading of extensions - 13https://github.com/miraheze/mw-config/pull/4718 [20:46:00] [url] Page not found · GitHub · GitHub | github.com [20:46:42] miraheze/mw-config - Universal-Omega the build passed. [20:46:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.98, 3.97 [20:47:13] miraheze/mw-config - Universal-Omega the build passed. [20:47:20] [02puppet] 07paladox commented on pull request 03#2564: varnish: support redirecting paths in SSL redirects.yaml - 13https://github.com/miraheze/puppet/pull/2564#issuecomment-1140521867 [20:47:21] [url] varnish: support redirecting paths in SSL redirects.yaml by Universal-Omega · Pull Request #2564 · miraheze/puppet · GitHub | github.com [20:47:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.49, 3.90 [20:49:08] RhinosF1: no, on mobile but I did just do https://github.com/miraheze/mw-config/pull/4718, which hopefully will fix enabling SemanticMediaWiki without running setupStore.php on all MediaWiki appservers, but even if not, it improves it a bit I think. [20:49:08] [url] Improve loading of extensions by Universal-Omega · Pull Request #4718 · miraheze/mw-config · GitHub | github.com [20:49:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.61, 3.79, 3.93 [20:49:44] It removes SiteConfiguration dependencies for loading extensions. [20:51:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.75, 3.27, 3.72 [20:57:06] JohnLewis: back [20:57:23] CosmicAlpha: ack [20:57:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.53, 3.66 [20:57:38] Have you seen my few tasks? Any opinions? [20:57:55] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [20:59:02] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.80, 1.72, 1.67 [20:59:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.27, 3.39, 3.59 [21:00:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 3.50, 3.54 [21:00:57] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.27, 1.53, 1.61 [21:02:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.88, 3.46, 3.51 [21:03:30] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.43, 6.83, 5.95 [21:03:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.68, 3.56, 3.94 [21:04:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.89, 3.15, 3.38 [21:05:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.36, 2.76, 3.27 [21:05:29] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.41, 6.03, 5.77 [21:05:58] RhinosF1: I've seen, I'll be on PC soon, then will look into something if I can. [21:06:48] PROBLEM - cp21 Disk Space on cp21 is WARNING: DISK WARNING - free space: / 4290 MB (10% inode=96%); [21:07:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.78, 4.00, 4.04 [21:11:07] CosmicAlpha: I tagged the open dblist with ManageWiki but I don't care how it's done. If mwscript or something is easier than do that. [21:11:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.04, 4.10, 3.64 [21:11:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.06, 4.69, 3.87 [21:13:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.17, 3.73, 3.56 [21:13:41] RhinosF1: what do you mean? I don't know what you mean tagged the open dblist? [21:13:55] Oh [21:13:58] I see. [21:14:08] I didn't get an email for that task unless I missed it. [21:15:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.13, 3.71, 3.64 [21:17:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.82, 3.69 [21:19:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.30, 3.52, 3.59 [21:19:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.83, 3.21, 3.37 [21:22:57] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.07, 3.10, 3.39 [21:28:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 4.26, 3.81 [21:29:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.56, 4.43, 3.80 [21:29:30] CosmicAlpha: ok [21:30:18] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 11.05, 7.58, 6.45 [21:30:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.69, 3.66 [21:31:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.91, 3.67 [21:33:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.28, 4.31, 3.83 [21:33:55] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.58, 1.68, 1.97 [21:34:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.74, 4.54, 4.01 [21:36:16] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.57, 7.56, 6.92 [21:36:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.23, 3.66, 3.75 [21:37:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/a7277a1ecb36 [21:37:53] [02miraheze/mw-config] 07Universal-Omega 03a7277a1 - Add database list for active wikis and improve others [21:37:54] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [21:37:55] [url] Page not found · GitHub · GitHub | github.com [21:37:56] [02mw-config] 07Universal-Omega opened pull request 03#4719: Add database list for active wikis and improve others - 13https://github.com/miraheze/mw-config/pull/4719 [21:37:56] [url] Page not found · GitHub · GitHub | github.com [21:38:15] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.32, 6.46, 6.59 [21:38:32] RhinosF1: ^ [21:38:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.91, 3.93, 3.82 [21:39:04] miraheze/mw-config - Universal-Omega the build passed. [21:40:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.80, 3.80 [21:41:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.18, 3.83, 3.89 [21:41:55] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.83, 1.90 [21:42:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.81, 3.79 [21:44:01] CosmicAlpha: you're amazing, can you add it to mwscript [21:44:48] RhinosF1: sure, once I test it. Doing that soon. [21:45:55] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.97, 1.97, 1.94 [21:46:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.61, 3.24, 3.56 [21:47:37] CosmicAlpha: thank you :) [21:47:42] I sleep now! [21:47:55] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.59, 2.15, 2.01 [21:49:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.49, 2.91, 3.37 [21:49:34] PROBLEM - cp20 Disk Space on cp20 is WARNING: DISK WARNING - free space: / 4284 MB (10% inode=96%); [21:51:44] !log [@test101] starting deploy of {'config': True} to all [21:51:46] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [21:51:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:51:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:52:20] [02puppet] 07Universal-Omega opened pull request 03#2610: deploy-mediawiki: fix `--branch` - 13https://github.com/miraheze/puppet/pull/2610 [21:52:20] [url] Page not found · GitHub · GitHub | github.com [21:52:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.30, 3.54, 3.57 [21:52:57] !log [universalomega@test101] starting deploy of {'config': True} to all [21:53:03] !log [universalomega@test101] finished deploy of {'config': True} to all - SUCCESS in 5s [21:53:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:53:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:53:13] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 13.87, 6.43, 3.56 [21:53:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.19, 4.49, 3.90 [21:54:32] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:54:55] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:11] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:17] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:17] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:34] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:56:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.85, 1.56, 1.34 [21:56:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.57, 3.62 [21:57:44] paladox: can you merge https://github.com/miraheze/puppet/pull/2610 if around? [21:57:44] [url] deploy-mediawiki: fix `--branch` by Universal-Omega · Pull Request #2610 · miraheze/puppet · GitHub | github.com [21:57:55] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 0.74, 1.62, 1.95 [21:58:29] !log [universalomega@test101] starting deploy of {'config': True} to all [21:58:29] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 0.916 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [21:58:30] !log [universalomega@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [21:58:30] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.91, 1.29, 1.26 [21:58:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:58:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:58:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.73, 3.87, 3.71 [21:59:13] [02mw-config] 07Universal-Omega closed pull request 03#4719: Add database list for active wikis and improve others - 13https://github.com/miraheze/mw-config/pull/4719 [21:59:14] [url] Page not found · GitHub · GitHub | github.com [21:59:15] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [21:59:15] ... [21:59:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/265e5c0cdfca...d2e83ca13b1d [21:59:17] [url] Comparing 265e5c0cdfca...d2e83ca13b1d · miraheze/mw-config · GitHub | github.com [21:59:18] [02miraheze/mw-config] 07Universal-Omega 03d2e83ca - Add database list for active wikis and improve others (#4719) [21:59:19] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [21:59:45] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [21:59:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:00:19] miraheze/mw-config - Universal-Omega the build passed. [22:00:27] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 42s [22:00:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:00:49] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22345 bytes in 1.164 second response time [22:01:02] !log [universalomega@test101] starting deploy of {'config': True} to all [22:01:03] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [22:01:10] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14654 bytes in 0.687 second response time [22:01:12] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14654 bytes in 0.034 second response time [22:01:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:01:30] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14654 bytes in 0.044 second response time [22:01:30] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14654 bytes in 0.340 second response time [22:01:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:01:55] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 0.64, 1.13, 1.67 [22:02:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.65, 3.66 [22:03:33] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 341 bytes in 0.273 second response time [22:04:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.36, 1.71, 1.29 [22:05:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.79, 3.60, 3.90 [22:05:34] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [22:05:35] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [22:05:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:05:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:06] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.17, 1.54, 1.27 [22:07:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.92, 4.27, 4.09 [22:08:44] [02puppet] 07paladox closed pull request 03#2610: deploy-mediawiki: fix `--branch` - 13https://github.com/miraheze/puppet/pull/2610 [22:08:45] [url] Page not found · GitHub · GitHub | github.com [22:08:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c0e401860bdf...5502cbada4f8 [22:08:46] [url] Comparing c0e401860bdf...5502cbada4f8 · miraheze/puppet · GitHub | github.com [22:08:47] [02miraheze/puppet] 07Universal-Omega 035502cba - deploy-mediawiki: fix `--branch` (#2610) [22:08:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.34, 3.81, 3.63 [22:11:11] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:11:17] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:11:30] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:11:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.96, 3.40, 3.97 [22:11:45] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:11:47] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:12:43] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:13:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.86, 4.19, 4.18 [22:14:43] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:15:14] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 3.305 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [22:16:37] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 13 minutes ago with 0 failures [22:16:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.22, 3.51, 3.68 [22:17:13] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14654 bytes in 0.030 second response time [22:17:33] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22344 bytes in 0.728 second response time [22:17:44] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14660 bytes in 0.342 second response time [22:17:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.62, 3.96 [22:17:58] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14660 bytes in 0.023 second response time [22:18:00] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14660 bytes in 0.312 second response time [22:19:42] !log [@test101] starting deploy of {'config': True} to all [22:19:43] !log [@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [22:19:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:19:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:20:55] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d4c45528d07a...e1cf2f08b6ff [22:20:55] [url] Comparing d4c45528d07a...e1cf2f08b6ff · miraheze/mw-config · GitHub | github.com [22:20:56] [02miraheze/mw-config] 07Universal-Omega 03e1cf2f0 - Update MirahezeFunctions.php [22:20:58] [02mw-config] 07Universal-Omega synchronize pull request 03#4718: Improve loading of extensions - 13https://github.com/miraheze/mw-config/pull/4718 [22:20:58] [url] Page not found · GitHub · GitHub | github.com [22:21:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.39, 3.39, 3.95 [22:21:33] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 52092 bytes in 0.589 second response time [22:21:38] minorly semantic but https://phabricator.miraheze.org/T9301 how can this be an UBN but all of its sub tasks are of lower priority than it? [22:21:39] [url] ⚓ T9301 JobQueue spiralling out of control (Late May 2022) | phabricator.miraheze.org [22:22:09] miraheze/mw-config - Universal-Omega the build has errored. [22:22:27] Its essentially going "we need to fix this ASAP, but none of the work to fix it needs to be done now, and one can be done in a few months without problems" [22:23:12] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [22:25:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 3.89, 3.87 [22:26:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.64, 2.91, 3.31 [22:27:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.54, 3.74 [22:31:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.68, 2.71, 3.33 [22:35:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.29, 3.49 [22:37:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.36, 3.49 [22:41:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.23, 3.07, 3.36 [22:43:07] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.76, 1.95, 3.85 [22:45:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.30, 3.37 [22:47:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.28, 3.25, 3.34 [22:49:30] !log [@test101] starting deploy of {'config': True} to all [22:49:31] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [22:49:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:49:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:51:07] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.45, 2.39, 3.30 [22:51:07] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:51:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.92, 3.99, 3.64 [22:52:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.82, 3.41 [22:55:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.48, 3.97, 3.66 [22:57:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.01, 3.40, 3.48 [23:00:05] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.15, 3.09, 3.26 [23:00:36] [02mw-config] 07Universal-Omega synchronize pull request 03#4718: Improve loading of extensions - 13https://github.com/miraheze/mw-config/pull/4718 [23:00:37] [url] Page not found · GitHub · GitHub | github.com [23:00:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e1cf2f08b6ff...c0a5a64d39c9 [23:00:38] [url] Comparing e1cf2f08b6ff...c0a5a64d39c9 · miraheze/mw-config · GitHub | github.com [23:00:39] [02miraheze/mw-config] 07Universal-Omega 03c0a5a64 - Update MirahezeFunctions.php [23:01:40] miraheze/mw-config - Universal-Omega the build passed. [23:01:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.85, 3.25, 3.40 [23:14:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.03, 4.01, 3.46 [23:18:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.18, 3.94, 3.58 [23:20:01] !log [@test101] starting deploy of {'config': True} to all [23:20:02] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [23:20:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:20:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:20:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.78, 3.18, 3.34 [23:26:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.07, 4.25, 3.68 [23:26:21] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.36, 3.07 [23:28:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.00, 4.34, 3.45 [23:31:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.77, 3.45 [23:32:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.46, 3.68, 3.43 [23:33:47] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.39, 3.35 [23:34:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.84, 3.79 [23:41:59] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.82, 3.29, 3.36 [23:45:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.49, 3.54 [23:46:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.63, 3.82, 3.49 [23:47:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.98, 3.35, 3.46 [23:48:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.59, 3.43 [23:49:40] !log [@test101] starting deploy of {'config': True} to all [23:49:41] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [23:49:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.98, 3.04, 3.32 [23:49:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:49:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.23, 3.03, 3.24 [23:55:35] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 3.67, 3.42 [23:55:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.68, 3.54 [23:57:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.96, 3.66, 3.44 [23:59:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 3.13, 3.36 [23:59:56] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.84, 3.55, 2.96