[00:00:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.27, 9.31, 9.01 [00:01:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.37, 9.67, 9.53 [00:02:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.63, 9.33, 9.05 [00:02:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.78, 9.09, 8.92 [00:02:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.29, 8.64, 7.88 [00:03:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:03:51] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 442 failures. Last run 3 minutes ago with 442 failures. Failed resources (up to 3 shown): File[nonciclopedia.org],File[nonciclopedia.org_private],File[monarchists.wiki],File[monarchists.wiki_private] [00:04:21] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.07, 9.12, 8.90 [00:04:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.38, 8.10, 8.58 [00:06:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.49, 9.40, 9.04 [00:06:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.50, 4.13, 3.51 [00:06:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.27, 8.48, 8.06 [00:07:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.56, 9.67, 9.51 [00:08:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.49, 2.97 [00:08:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.18, 8.67, 8.66 [00:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:09:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.75, 9.11, 9.31 [00:10:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.25, 3.10, 2.89 [00:10:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.68, 3.98, 3.66 [00:10:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.46, 10.19, 8.83 [00:11:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.85, 9.90, 9.58 [00:12:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.88, 9.47, 8.75 [00:13:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.59, 9.51, 9.49 [00:14:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.18, 3.89, 3.68 [00:14:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.02, 1.60, 1.09 [00:15:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.95, 3.35, 3.05 [00:16:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.15, 3.62, 3.60 [00:16:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.80, 1.35, 1.06 [00:16:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.86, 8.35, 8.48 [00:17:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.56, 9.97, 9.61 [00:17:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.52, 2.91, 2.92 [00:18:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.47, 9.12, 9.20 [00:19:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.49, 8.76, 9.22 [00:19:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.02, 3.76, 3.62 [00:20:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.20, 9.03, 8.33 [00:20:20] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.91, 7.71, 8.34 [00:20:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.51, 9.19, 8.76 [00:22:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.90, 7.45, 8.22 [00:22:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.47, 8.76, 8.64 [00:23:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.27, 3.58, 3.57 [00:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:24:55] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.01, 7.71, 8.49 [00:25:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.00, 3.08, 3.39 [00:25:52] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.96, 8.02, 8.14 [00:26:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.16, 8.16, 8.43 [00:29:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.30, 7.36, 8.43 [00:31:44] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [00:33:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:38:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:42:31] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [00:43:43] !log [void@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/populateContentTables.php --wiki=wikicryptoswiki (END - exit=0) [00:43:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:56:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:05:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.26, 7.98, 7.30 [01:06:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:07:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.61, 7.20, 7.11 [01:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:10:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.61, 8.31, 7.73 [01:12:14] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.07, 7.80, 7.61 [01:12:32] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.res [01:12:32] tname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4048004150390625 seconds [01:14:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:30:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:31:18] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.50, 1.93, 1.36 [01:33:16] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.83, 1.55, 1.29 [01:34:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.42, 8.51, 7.66 [01:35:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:35:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.58, 8.08, 7.47 [01:36:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.56, 8.24, 7.68 [01:37:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.70, 7.78, 7.42 [01:38:59] !log INSERT INTO content_models (model_id, model_name) VALUES (6, 'sanitized-css'); INSERT INTO content_models (model_id, model_name) VALUES (7, 'text'); (wikicryptoswiki) [01:39:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.95, 8.40, 7.66 [01:39:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:39:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.36, 8.86, 7.75 [01:41:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.63, 8.80, 7.87 [01:41:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.81, 8.37, 7.75 [01:42:21] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.27, 10.02, 8.60 [01:43:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:43:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.15, 9.70, 8.39 [01:43:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.25, 8.90, 7.64 [01:44:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.32, 9.47, 8.56 [01:45:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.67, 8.81, 8.14 [01:45:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.64, 9.78, 8.46 [01:46:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.97, 8.33, 8.26 [01:47:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.31, 8.35, 8.03 [01:47:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.87, 9.70, 8.59 [01:49:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.95, 10.14, 8.89 [01:50:21] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.45, 9.41, 8.68 [01:51:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.38, 10.83, 9.17 [01:51:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.92, 9.95, 8.97 [01:51:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.18, 8.27, 7.90 [01:52:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.89, 8.88, 8.58 [01:53:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:54:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.85, 8.18, 8.36 [01:55:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.70, 9.15, 8.89 [01:57:06] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.42, 8.41, 8.40 [01:57:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.06, 8.16, 8.49 [01:58:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.08, 8.51, 8.43 [02:00:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.20, 8.53, 8.04 [02:00:21] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.71, 8.92, 8.56 [02:00:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:02:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.02, 8.11, 7.95 [02:02:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.97, 9.50, 8.84 [02:02:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.15, 9.31, 8.71 [02:03:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.12, 1.63, 1.13 [02:04:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.45, 8.88, 8.64 [02:05:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.15, 9.71, 9.05 [02:05:47] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.91, 1.31, 1.07 [02:06:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.50, 8.54, 8.20 [02:06:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.71, 9.31, 8.81 [02:07:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.70, 8.90, 8.83 [02:09:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.55, 8.28, 8.18 [02:10:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.45, 7.95, 8.37 [02:10:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.25, 9.57, 9.04 [02:12:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.60, 8.68, 8.11 [02:13:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.59, 9.98, 9.33 [02:14:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.70, 8.50, 8.50 [02:16:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.93, 8.22, 8.39 [02:16:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.06, 9.34, 9.07 [02:18:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.59, 8.29, 8.17 [02:18:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.96, 8.59, 8.83 [02:19:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.25, 10.15, 9.24 [02:19:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.43, 9.59, 9.48 [02:21:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.45, 8.96, 8.91 [02:22:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.26, 9.06, 8.48 [02:22:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.03, 8.93, 8.89 [02:23:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.50, 8.87, 8.30 [02:24:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.06, 8.60, 8.38 [02:24:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.41, 9.17, 8.99 [02:25:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.20, 9.07, 8.42 [02:26:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.27, 9.28, 8.65 [02:26:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.03, 9.68, 9.19 [02:27:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.36, 8.32, 8.22 [02:28:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.92, 8.67, 8.50 [02:28:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.89, 9.15, 9.06 [02:29:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.11, 7.35, 8.29 [02:30:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:30:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.39, 8.30, 8.39 [02:31:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.83, 7.51, 8.47 [02:35:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.34, 8.69, 8.74 [02:36:55] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.74, 7.86, 8.46 [02:37:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:39:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.96, 8.00, 8.46 [02:42:31] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [02:44:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.27, 2.77 [02:45:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.98, 8.10, 7.78 [02:45:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.55, 9.17, 8.77 [02:46:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.00, 3.23, 2.82 [02:47:31] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.37, 9.18, 8.24 [02:47:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.43, 8.65, 8.62 [02:49:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.21, 8.80, 8.21 [02:49:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.63, 8.27, 8.48 [02:50:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.84, 8.50, 8.30 [02:51:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.68, 8.66, 8.06 [02:52:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:53:06] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.36, 7.69, 7.77 [02:53:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.23, 8.74, 8.53 [02:53:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.53, 8.88, 8.67 [02:54:30] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.70, 8.19, 8.26 [02:55:17] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.04, 9.28, 8.74 [02:57:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.80, 8.52, 8.54 [02:57:16] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.80, 7.98, 8.06 [02:58:00] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.67, 2.09, 1.57 [02:58:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.86, 3.47, 3.03 [02:59:06] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.99, 7.79, 8.25 [02:59:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:59:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.00, 8.19, 8.48 [02:59:58] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.01, 1.83, 1.54 [03:00:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.89, 3.10, 2.94 [03:01:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.13, 8.75, 8.36 [03:01:56] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.55, 1.37, 1.41 [03:03:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.87, 8.35, 8.26 [03:06:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.74, 8.61, 8.40 [03:09:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:10:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.96, 8.46, 8.40 [03:15:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:19:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.82, 9.05, 8.51 [03:28:55] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.69, 8.08, 8.38 [03:30:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:39:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.03, 8.73, 8.32 [03:41:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.22, 8.34, 7.87 [03:43:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.22, 9.45, 8.73 [03:43:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.68, 8.75, 8.52 [03:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:45:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.76, 8.50, 8.11 [03:50:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.94, 9.52, 8.46 [03:51:06] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.45, 8.15, 8.39 [03:53:10] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.83, 8.58, 8.27 [03:54:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.03, 9.33, 8.80 [03:55:09] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.79, 8.03, 8.11 [03:55:24] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.80, 1.73, 1.18 [03:56:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.44, 8.91, 8.73 [03:56:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.72, 8.67, 8.63 [03:57:20] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.03, 1.46, 1.14 [03:58:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.52, 8.49, 8.42 [04:00:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.61, 9.17, 8.93 [04:00:55] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.83, 8.20, 8.48 [04:04:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:06:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.69, 7.72, 8.37 [04:09:12] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 2.92, 2.29 [04:11:12] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.77, 3.01, 2.41 [04:42:32] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.res [04:42:32] tname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.40663743019104 seconds [04:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:54:16] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.62, 1.71, 1.40 [04:54:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:56:16] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.19, 1.52, 1.37 [05:15:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:19:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.01, 1.79, 1.58 [05:20:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:21:07] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.88, 1.72, 1.57 [05:32:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.56, 3.23, 2.59 [05:34:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.75, 3.08, 2.62 [05:34:55] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.44, 1.68, 1.68 [05:36:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.67, 8.37, 7.55 [05:38:16] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.78, 7.95, 7.49 [05:40:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.65, 1.71, 1.70 [05:51:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:52:38] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.47, 1.97, 1.82 [05:56:35] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.69, 1.95, 1.87 [05:56:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:06:26] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.71, 2.07, 1.91 [06:12:29] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [06:13:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:18:36] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:31:22] PROBLEM - wiki.projectdiablo2.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.projectdiablo2.com' expires in 15 day(s) (Thu 03 Mar 2022 06:03:47 GMT +0000). [06:33:17] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3ebaf3c98c54...1b27c9b55b52 [06:33:18] [02miraheze/ssl] 07MirahezeSSLBot 031b27c9b - Bot: Update SSL cert for wiki.projectdiablo2.com [06:45:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:50:36] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:57:32] [02CreateWiki] 07amire80 opened pull request 03#293: Add a missing apostrophe - 13https://github.com/miraheze/CreateWiki/pull/293 [06:58:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:00:43] RECOVERY - wiki.projectdiablo2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.projectdiablo2.com' will expire on Mon 16 May 2022 05:33:08 GMT +0000. [07:03:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:16:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.85, 8.47, 7.14 [07:16:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.66, 8.90, 7.61 [07:18:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.91, 7.61, 6.97 [07:18:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.38, 7.92, 7.40 [07:19:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:21:38] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.43, 2.05, 1.42 [07:23:32] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.78, 1.62, 1.34 [07:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:41:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.92, 8.60, 7.64 [07:43:53] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.13, 7.63, 7.41 [08:02:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.24, 6.80, 5.35 [08:04:50] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.70, 6.32, 5.34 [08:12:32] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.res [08:12:32] tname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406828880310059 seconds [08:21:20] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-5/±1] 13https://github.com/miraheze/ssl/compare/1b27c9b55b52...f13b4c7602d6 [08:21:22] [02miraheze/ssl] 07Reception123 03f13b4c7 - wiki.yunachannel.com wiki.jkr.productions wiki.homiedb.party wiki.ripto.gq wiki.yapsavun.xyz (not pointing) [08:22:25] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/f13b4c7602d6...a70a4fd93d47 [08:22:26] [02miraheze/ssl] 07Reception123 03a70a4fd - remove wikipedia.yogfront.ooo (not pointing) [08:33:30] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7219c757b031...36304a7584aa [08:33:31] [02miraheze/mw-config] 07Reception123 0336304a7 - temporarily up throttle limits to avoid issues on retry (T8335) [08:34:39] miraheze/mw-config - Reception123 the build passed. [08:41:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:42:31] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [08:44:43] Reception123: there's at least 6 domains with issues [08:44:57] I just didn't file tasks for each because so many [08:45:09] https://icinga.miraheze.org/monitoring/host/show?host=sslhost [08:45:11] [url] Icinga Web 2 Login | icinga.miraheze.org [08:45:15] RhinosF1: the ones with CF, etc. should be dealt with in the normal way not directly removed [08:45:24] Reception123: can you please send notices then [08:45:48] We need to keep on top [08:46:07] yeah, though I wouldn't call it high prio either necessarily especially since it's mostly us waiting for user response [08:46:28] SSL WARNING - rDNS OK but records conflict. {'NS': ['amanda.ns.cloudflare.com.', 'augustus.ns.cloudflare.com.'], 'CNAME': 'miraheze.wgytcraft.cf.'} [08:46:29] is strange [08:46:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:48:07] !log [@test101] starting deploy of {'config': True} to skip [08:48:08] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [08:48:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:48:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:53:33] !log [@mwtask111] starting deploy of {'config': True} to all [08:53:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:53:44] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 11s [08:53:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:56:13] RECOVERY - wiki.yunachannel.com - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:04:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.34, 8.25, 7.45 [09:05:55] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.13, 7.81, 7.38 [09:06:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:11:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:16:30] RhinosF1: any reason you couldn’t send the notices and support Reception123 in keeping on top of managing them? It’s not just his responsibility, it’s that of the MediaWiki team [09:49:18] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.06, 1.32, 2.00 [09:51:48] !log reception@mwtask111:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=metawiki --u="Reception123" --r="[[Special:Diff/236748|Requested]], unused" /home/reception/metadel.txt [09:51:48] https://meta.miraheze.org/wiki/Special:Diff/236748 [09:51:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:53:18] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.33, 0.85, 1.65 [10:11:04] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [10:26:35] [02mw-config] 07ugochimobi opened pull request 03#4475: propertyOrderUrl - 13https://github.com/miraheze/mw-config/pull/4475 [10:26:39] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.89, 2.43, 1.51 [10:27:40] miraheze/mw-config - ugochimobi the build passed. [10:28:33] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.35, 1.95, 1.44 [10:30:27] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.85, 1.56, 1.36 [10:31:40] JohnLewis: I can but so far Reception123 has dealt with SSL mostly themself [10:31:55] 2 of them domains have never worked on this infra as far as I can tell and were added mid switch [10:39:03] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [11:01:55] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.87, 1.59, 1.39 [11:03:49] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.63, 1.66, 1.45 [11:41:53] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.50, 4.63, 2.76 [11:49:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.74, 8.17, 7.13 [11:50:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.05, 8.82, 7.41 [11:51:23] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.31, 1.24, 1.85 [11:51:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.10, 8.06, 7.22 [11:52:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.05, 7.74, 7.21 [11:53:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:55:13] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.35, 1.23, 1.70 [11:58:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:08:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:11:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.23, 8.65, 7.72 [12:15:57] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.06, 8.64, 7.79 [12:16:06] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.43, 8.54, 7.93 [12:17:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.96, 8.49, 7.98 [12:17:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.42, 8.68, 7.90 [12:17:59] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.30, 8.39, 7.94 [12:18:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:19:55] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.42, 8.28, 7.88 [12:21:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.50, 8.55, 8.13 [12:23:41] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.41, 7.79, 7.90 [12:35:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.10, 8.49, 8.02 [12:36:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.33, 2.64 [12:38:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.02, 3.24, 2.69 [12:39:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.18, 8.28, 8.06 [12:41:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.81, 8.86, 7.92 [12:43:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.65, 8.93, 8.06 [12:44:02] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.05, 1.75, 1.46 [12:45:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:46:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.84, 3.15 [12:47:50] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.85, 1.89, 1.59 [12:48:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.70, 8.86, 8.45 [12:48:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.75, 3.15, 2.99 [12:49:44] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.86, 1.60, 1.53 [12:50:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.21, 8.50, 8.37 [12:50:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.25, 8.66, 7.78 [12:50:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:51:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.22, 9.76, 8.72 [12:52:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.56, 8.15, 7.69 [12:53:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.87, 9.83, 8.88 [12:53:33] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.39, 2.00, 1.70 [12:57:21] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.21, 1.75, 1.68 [12:59:15] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.27, 1.53, 1.60 [13:01:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.51, 7.88, 8.32 [13:04:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.75, 8.89, 8.36 [13:08:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.70, 7.69, 8.04 [13:11:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.35, 9.27, 8.72 [13:11:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.60, 9.05, 8.46 [13:12:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.49, 10.11, 8.88 [13:13:51] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.03, 8.34, 8.25 [13:14:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.95, 9.29, 8.74 [13:15:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.83, 9.36, 8.86 [13:17:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.59, 8.64, 8.04 [13:18:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.53, 9.78, 9.03 [13:18:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:19:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.71, 8.53, 8.33 [13:20:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.42, 9.32, 8.93 [13:23:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.56, 9.63, 9.10 [13:23:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.86, 8.18, 8.24 [13:27:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.51, 9.25, 9.13 [13:28:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.22, 9.06, 8.13 [13:29:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.79, 10.58, 9.20 [13:30:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.63, 9.59, 9.20 [13:30:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.10, 9.06, 8.24 [13:31:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.45, 9.01, 8.56 [13:32:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.36, 9.17, 9.07 [13:33:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.35, 10.56, 9.68 [13:33:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.42, 3.04, 2.49 [13:33:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.15, 8.14, 8.31 [13:34:23] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.45, 2.57 [13:37:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.13, 3.00, 2.58 [13:37:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.53, 8.61, 8.46 [13:38:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 3.07, 2.65 [13:38:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:39:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.35, 8.44, 8.42 [13:40:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.19, 7.92, 8.09 [13:41:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.96, 9.68, 9.65 [13:43:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:43:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.60, 9.29, 8.80 [13:44:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.90, 9.19, 8.96 [13:46:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.70, 8.84, 8.85 [13:46:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.65, 8.90, 8.46 [13:49:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.71, 10.13, 9.31 [13:50:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.89, 9.51, 9.07 [13:51:11] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.90, 9.16, 8.13 [13:51:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.07, 9.93, 9.36 [13:52:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.83, 8.03, 8.22 [13:53:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.00, 9.29, 8.33 [13:55:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.11, 10.21, 9.59 [13:57:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.28, 9.95, 9.58 [13:58:59] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.21, 8.28, 8.17 [13:59:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.07, 9.42, 9.28 [14:01:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.81, 9.55, 9.97 [14:02:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.29, 9.85, 9.66 [14:03:10] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.59, 9.81, 9.54 [14:04:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.20, 10.14, 9.78 [14:08:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.25, 9.39, 9.56 [14:08:37] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:09:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.14, 9.06, 9.06 [14:10:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.51, 10.19, 9.83 [14:11:06]