[00:00:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.84, 8.39, 8.37 [00:02:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:02:48] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 414 failures. Last run 2 minutes ago with 414 failures. Failed resources (up to 3 shown): File[history.sdtef.org_private],File[wiki.rebirthofthenight.com],File[wiki.rebirthofthenight.com_private],File[beaconspace.unrestrictedlorefare.com] [00:05:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.58, 4.65, 3.34 [00:05:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.18, 9.46, 9.58 [00:08:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.50, 8.42, 8.44 [00:08:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.95, 8.63, 8.15 [00:08:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.67, 3.23 [00:09:43] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [00:10:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.48, 8.40, 8.12 [00:12:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:12:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.84, 3.29, 3.16 [00:14:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.25, 8.88, 8.35 [00:18:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.32, 9.39, 8.63 [00:19:28] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.11, 9.48, 9.20 [00:20:27] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.05, 7.04, 6.42 [00:21:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.00, 8.50, 8.88 [00:21:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.07, 8.54, 8.40 [00:22:26] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.05, 6.96, 6.49 [00:23:29] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.68, 9.00, 8.56 [00:24:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.32, 9.34, 8.96 [00:25:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.55, 8.76, 7.94 [00:25:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.64, 9.13, 8.66 [00:27:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.97, 8.40, 7.90 [00:27:26] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.31, 8.43, 8.46 [00:30:03] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.20, 8.11, 8.49 [00:31:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.32, 2.77 [00:32:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:32:43] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:07] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.04, 3.20, 2.79 [00:34:23] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.94, 6.52, 6.60 [00:35:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:37:08] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.46, 7.24, 8.20 [00:46:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.00, 10.03, 9.08 [00:48:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.30, 9.05, 8.85 [00:56:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.52, 7.89, 8.35 [00:58:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.64, 7.85, 7.41 [01:00:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:00:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.77, 7.91, 7.50 [01:00:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.51, 9.51, 8.94 [01:01:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.94, 8.79, 8.33 [01:02:14] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [01:02:14] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407097816467285 seconds [01:02:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.44, 9.03, 8.35 [01:03:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:03:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.23, 8.48, 8.29 [01:06:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.82, 9.95, 9.43 [01:08:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.53, 7.89, 8.10 [01:10:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.24, 8.58, 8.01 [01:12:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.85, 8.62, 8.19 [01:12:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.24, 7.81, 7.78 [01:13:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.91, 10.03, 8.91 [01:14:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.09, 8.82, 8.30 [01:16:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.97, 3.55, 2.87 [01:17:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.53, 8.87, 8.02 [01:17:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.45, 9.52, 8.94 [01:18:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.42, 8.42, 8.14 [01:18:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.97, 3.88, 3.05 [01:20:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.98, 8.08, 8.04 [01:20:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.22, 3.65, 3.07 [01:20:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.33, 9.43, 9.36 [01:20:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.43, 2.82 [01:21:59] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.56, 2.94 [01:22:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.50, 8.22, 8.03 [01:22:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.44, 9.51, 9.41 [01:22:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.63, 3.24, 2.83 [01:23:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.41, 9.32, 8.93 [01:23:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.48, 2.90, 2.78 [01:24:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.27, 3.10, 2.99 [01:26:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.24, 9.00, 8.40 [01:27:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.73, 9.96, 9.23 [01:27:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.19, 9.73, 9.09 [01:27:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.20, 8.62, 8.24 [01:28:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.80, 8.36, 8.23 [01:29:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.41, 8.11, 8.10 [01:29:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.83, 9.34, 9.08 [01:31:14] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [01:31:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.84, 9.43, 9.19 [01:32:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.58, 8.94, 8.48 [01:33:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.33, 9.23, 9.07 [01:34:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.97, 8.50, 8.37 [01:36:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.60, 8.17, 7.74 [01:36:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.34, 9.49, 9.20 [01:37:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.51, 8.87, 9.00 [01:38:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.51, 7.92, 7.70 [01:38:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.88, 8.65, 8.94 [01:39:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.35, 9.27, 9.16 [01:43:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.15, 9.57, 9.33 [01:44:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.68, 9.75, 9.26 [01:44:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.73, 3.51, 2.95 [01:45:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.31, 9.72, 9.40 [01:46:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.42, 9.00, 9.06 [01:46:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.67, 3.18, 2.90 [01:47:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.74, 9.19, 9.26 [01:49:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.39, 9.89, 9.24 [01:51:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.01, 9.28, 9.28 [01:53:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.05, 9.00, 9.17 [01:53:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.80, 9.78, 9.38 [01:53:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:57:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.18, 9.47, 9.29 [01:59:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.67, 9.26, 9.22 [01:59:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:00:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.47, 9.49, 8.97 [02:01:16] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.48, 7.66, 8.41 [02:04:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.47, 9.12, 9.02 [02:09:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:11:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.92, 7.33, 8.30 [02:18:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:18:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.17, 7.73, 8.36 [02:20:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.78, 9.23, 8.77 [02:22:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.16, 9.61, 8.97 [02:23:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:24:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.06, 8.76, 8.74 [02:30:13] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.39, 8.07, 8.42 [02:30:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:45:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:46:36] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.70, 8.25, 7.79 [02:48:34] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.99, 8.33, 7.89 [02:48:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:49:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.64, 8.73, 8.05 [02:51:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.10, 9.12, 8.27 [02:53:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.15, 9.07, 8.35 [03:02:15] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.06, 10.10, 8.80 [03:04:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.71, 9.66, 8.80 [03:07:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.39, 8.02, 8.26 [03:12:04] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.71, 7.98, 8.36 [03:13:26] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.94, 8.96, 8.15 [03:13:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:13:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.74, 8.61, 7.81 [03:15:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.87, 8.41, 7.83 [03:15:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.83, 8.86, 8.62 [03:16:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:19:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.17, 7.84, 8.00 [03:19:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.19, 1.67, 1.32 [03:19:54] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.74, 7.71, 8.23 [03:21:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.43, 1.58, 1.33 [03:22:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.68, 9.24, 8.48 [03:27:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.79, 8.96, 8.50 [03:28:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.44, 9.35, 8.78 [03:29:41] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.16, 8.38, 8.35 [03:32:13] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.11, 8.07, 8.37 [03:36:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.61, 8.88, 8.43 [03:38:04] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.35, 8.20, 8.23 [03:41:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:43:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.86, 9.00, 8.52 [03:45:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:47:55] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.27, 7.88, 8.21 [03:49:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.66, 9.04, 8.24 [03:51:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.26, 8.56, 8.41 [03:53:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.38, 8.46, 8.23 [03:53:50] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.46, 8.45, 8.38 [03:55:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:56:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 3.52, 3.03 [03:58:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.80, 3.08, 2.92 [04:00:02] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.58, 9.40, 8.41 [04:01:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:02:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.85, 8.84, 8.34 [04:03:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.35, 8.19, 8.17 [04:06:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:15:10] !log sudo -u www-data rsync -ur /mnt/mediawiki-static/tolololpediawiki/*/ /mnt/mediawiki-static/tampvanwiki/ [04:15:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:24:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.59, 8.00, 7.28 [04:26:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.90, 7.76, 7.27 [04:32:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.77, 8.66, 7.88 [04:34:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.16, 8.44, 7.92 [04:40:19] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.12, 1.49, 0.92 [04:42:19] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.58, 1.56, 1.02 [04:46:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.00, 8.37, 7.16 [04:48:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.47, 7.67, 7.05 [04:50:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.43, 8.91, 8.05 [04:51:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.26, 8.99, 8.04 [04:52:08] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.64, 8.14, 7.86 [04:52:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.42, 8.71, 8.06 [04:53:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:54:54] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.42, 1.90, 1.13 [04:54:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.78, 9.52, 8.43 [04:56:54] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.30, 1.54, 1.09 [04:58:02] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.10, 9.24, 8.40 [04:58:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.64, 8.84, 8.48 [05:00:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.77, 8.23, 8.13 [05:00:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.57, 8.15, 8.28 [05:03:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:05:19] PROBLEM - inourownwords.online - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/ [05:05:19] on3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query inourownwords.online. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:13:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.98, 8.56, 8.01 [05:15:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:15:48] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.47, 8.30, 7.98 [05:20:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:22:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.13, 2.75 [05:23:30] [02miraheze/RemovePII] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/checkout-3 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/commit/97cab67d872e [05:23:31] [02miraheze/RemovePII] 07dependabot[bot] 0397cab67 - Bump actions/checkout from 2 to 3 [05:23:33] [02RemovePII] 07dependabot[bot] created branch 03dependabot/github_actions/actions/checkout-3 - 13https://github.com/miraheze/RemovePII [05:23:34] [02RemovePII] 07dependabot[bot] opened pull request 03#51: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/RemovePII/pull/51 [05:23:36] [02RemovePII] 07dependabot[bot] labeled pull request 03#51: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/RemovePII/pull/51 [05:23:37] [02RemovePII] 07dependabot[bot] labeled pull request 03#51: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/RemovePII/pull/51 [05:24:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.70, 2.57, 2.59 [05:26:01] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [05:26:01] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407224655151367 seconds [05:28:19] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.58, 1.74, 1.46 [05:28:32] miraheze/RemovePII - dependabot[bot] the build passed. [05:30:19] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.10, 1.51, 1.41 [05:34:08] RECOVERY - inourownwords.online - reverse DNS on sslhost is OK: SSL OK - inourownwords.online reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [06:08:19] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.78, 2.07, 1.57 [06:12:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:17:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:18:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.52, 9.71, 7.83 [06:19:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 16.48, 10.40, 7.66 [06:19:13] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.61, 9.41, 6.95 [06:19:19] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:19:20] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:19:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 18.51, 11.17, 7.56 [06:19:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.72, 1.52, 1.30 [06:19:57] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:20:13] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:20:18] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:20:22] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:20:39] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:20:55] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:20:57] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:21:40] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:21:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.27, 1.42, 1.28 [06:21:50] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:04] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:22:57] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:11] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:22] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:24:04] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.754 second response time [06:24:18] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:24:25] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.116 second response time [06:24:26] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.703 second response time [06:24:29] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.325 second response time [06:24:47] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22295 bytes in 0.243 second response time [06:24:53] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [06:24:55] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:24:57] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22295 bytes in 0.422 second response time [06:25:06] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.018 second response time [06:25:22] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.319 second response time [06:25:36] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.015 second response time [06:25:40] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.330 second response time [06:25:53] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [06:26:08] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.313 second response time [06:26:12] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.015 second response time [06:26:50] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.322 second response time [06:26:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.43, 9.39, 8.66 [06:26:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.53, 9.25, 8.65 [06:27:02] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22295 bytes in 1.200 second response time [06:27:21] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.308 second response time [06:27:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:28:43] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.54, 3.42, 2.02 [06:28:54] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.12, 7.95, 8.22 [06:29:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.20, 9.65, 8.82 [06:30:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.59, 7.74, 8.15 [06:31:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.24, 8.51, 8.81 [06:33:13] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.00, 7.82, 8.50 [06:33:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.45, 9.69, 8.93 [06:34:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.54, 1.58, 1.65 [06:35:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 5.89, 8.42, 8.58 [06:37:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.18, 7.73, 8.30 [06:40:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.62, 7.99, 7.80 [06:42:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.28, 7.28, 7.56 [06:49:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.97, 1.70, 1.36 [06:51:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.42, 1.57, 1.36 [06:54:11] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1117073596 secs [06:55:58] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [07:16:14] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.08728018403 secs [07:34:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.70, 9.79, 7.11 [07:35:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.23, 7.61, 6.46 [07:37:02] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 4.88, 6.43, 6.16 [07:42:19] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.10, 1.60, 1.90 [07:42:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.22, 8.87, 8.31 [07:44:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.96, 7.53, 7.89 [07:46:19] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.04, 1.29, 1.70 [07:46:58] [02CreateWiki] 07lens0021 opened pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310 [07:49:17] [02CreateWiki] 07lens0021 synchronize pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310 [07:49:20] miraheze/CreateWiki - lens0021 the build has errored. [07:52:25] miraheze/CreateWiki - lens0021 the build has errored. [07:55:10] [02CreateWiki] 07lens0021 synchronize pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310 [07:58:27] miraheze/CreateWiki - lens0021 the build has errored. [07:59:02] [02CreateWiki] 07lens0021 synchronize pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310 [08:02:06] miraheze/CreateWiki - lens0021 the build has errored. [08:02:52] [02CreateWiki] 07lens0021 synchronize pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310 [08:02:55] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.99, 1.57, 1.52 [08:04:50] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.07, 1.74, 1.59 [08:06:46] miraheze/CreateWiki - lens0021 the build has errored. [08:10:01] [02CreateWiki] 07lens0021 synchronize pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310 [08:12:29] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.76, 1.94, 1.78 [08:14:24] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.10, 1.62, 1.68 [08:15:48] miraheze/CreateWiki - lens0021 the build passed. [08:19:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:19:39] [02CreateWiki] 07lens0021 synchronize pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310 [08:24:19] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.72, 2.13, 1.82 [08:24:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:24:42] miraheze/CreateWiki - lens0021 the build passed. [08:40:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.31, 10.09, 8.07 [08:42:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.86, 8.78, 7.84 [08:44:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.66, 7.59, 7.50 [08:56:01] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [08:56:01] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.408379077911377 seconds [09:19:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.85, 1.47, 1.24 [09:21:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.13, 1.33, 1.21 [09:24:19] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.38, 0.95, 1.93 [09:25:57] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [09:28:19] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.45, 0.63, 1.56 [09:56:01] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [09:56:01] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.40478777885437 seconds [10:19:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.27, 1.87, 1.43 [10:21:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.34, 1.68, 1.41 [11:25:58] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [11:28:35] [02miraheze/YouTube] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/checkout-3 [+0/-0/±1] 13https://github.com/miraheze/YouTube/commit/4fe72716a44a [11:28:37] [02miraheze/YouTube] 07dependabot[bot] 034fe7271 - Bump actions/checkout from 2 to 3 [11:28:38] [02YouTube] 07dependabot[bot] created branch 03dependabot/github_actions/actions/checkout-3 - 13https://github.com/miraheze/YouTube [11:28:40] [02YouTube] 07dependabot[bot] opened pull request 03#18: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/YouTube/pull/18 [11:28:41] [02YouTube] 07dependabot[bot] labeled pull request 03#18: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/YouTube/pull/18 [11:28:43] [02YouTube] 07dependabot[bot] labeled pull request 03#18: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/YouTube/pull/18 [11:33:06] miraheze/YouTube - dependabot[bot] the build passed. [12:22:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:26:01] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [12:26:01] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.403635501861572 seconds [12:32:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:53:10] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.13, 1.73, 1.48 [12:54:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:55:05] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.65, 1.69, 1.50 [12:56:00] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [12:59:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:01:32] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [13:01:32] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:02:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:03:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.40, 8.83, 7.57 [13:05:00] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.38, 8.49, 7.59 [13:07:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:22:05] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.21, 8.36, 7.37 [13:22:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.13, 9.07, 8.12 [13:23:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.07, 7.83, 6.95 [13:24:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.86, 8.71, 7.61 [13:24:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.45, 9.21, 8.29 [13:26:01] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [13:26:01] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4056642055511475 seconds [13:27:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.88, 7.99, 7.22 [13:28:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.96, 8.31, 7.76 [13:28:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.31, 9.42, 8.58 [13:30:55] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:30:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.21, 8.96, 8.51 [13:32:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.01, 8.46, 8.37 [13:35:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.40, 8.74, 8.09 [13:37:54] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.96, 8.11, 7.94 [13:38:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:43:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:49:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.87, 1.72, 1.38 [13:51:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.23, 1.54, 1.35 [13:51:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.21, 8.17, 7.92 [13:53:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.55, 7.85, 7.17 [13:53:40] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.03, 8.49, 8.14 [13:53:42] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.44, 8.88, 8.20 [13:55:13] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.96, 7.85, 7.26 [13:55:38] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.37, 8.39, 8.14 [13:55:58] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [13:57:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.20, 9.56, 8.70 [13:57:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:59:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.09, 9.65, 8.67 [14:01:33] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.58, 10.00, 9.01 [14:01:33] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.82, 9.66, 8.80 [14:03:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.60, 9.53, 8.97 [14:04:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.90, 8.32, 7.62 [14:05:30] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.44, 10.36, 9.27 [14:06:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.14, 7.11, 7.26 [14:07:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.98, 9.27, 9.02 [14:07:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:09:28] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.06, 8.11, 8.49 [14:09:28] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.30, 9.65, 9.18 [14:11:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.91, 9.70, 9.26 [14:14:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:15:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.54, 7.46, 7.02 [14:15:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.61, 8.35, 7.82 [14:17:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.38, 8.20, 7.85 [14:19:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.67, 8.95, 8.54 [14:19:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.74, 8.85, 7.65 [14:19:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:21:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.34, 8.48, 7.65 [14:22:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:25:08] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.07, 9.23, 8.76 [14:27:35] [02miraheze/mw-config] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/checkout-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/9fec23404079 [14:27:36] [02miraheze/mw-config] 07dependabot[bot] 039fec234 - Bump actions/checkout from 2 to 3 [14:27:38] [02mw-config] 07dependabot[bot] opened pull request 03#4486: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/mw-config/pull/4486 [14:27:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:27:39] [02mw-config] 07dependabot[bot] created branch 03dependabot/github_actions/actions/checkout-3 - 13https://github.com/miraheze/mw-config [14:27:41] [02mw-config] 07dependabot[bot] labeled pull request 03#4486: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/mw-config/pull/4486 [14:27:42] [02mw-config] 07dependabot[bot] labeled pull request 03#4486: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/mw-config/pull/4486 [14:28:42] miraheze/mw-config - dependabot[bot] the build passed. [14:29:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.63, 8.72, 8.21 [14:31:13] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.97, 7.89, 8.45 [14:31:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.01, 8.29, 8.10 [14:32:39]