[00:00:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.53, 3.75, 3.58 [00:00:41] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [00:01:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.93, 3.20, 3.09 [00:04:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.48, 5.17, 4.10 [00:04:13] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:39] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 502 failures. Last run 3 minutes ago with 502 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wiki.qadrishattari.xyz.crt],File[/etc/ssl/private/wiki.qadrishattari.xyz.key],File[/etc/ssl/localcerts/www.christipedia.nl.crt],File[/etc/ssl/private/www.christipedia.nl.key] [00:05:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.76, 4.57, 3.67 [00:07:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.26, 3.94, 3.55 [00:10:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.81, 3.81 [00:11:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.15, 3.16, 3.33 [00:13:29] !log [universalomega@mwtask111] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all [00:13:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:13:58] !log [universalomega@test101] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all [00:14:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 3.86, 3.80 [00:14:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.81, 3.75, 3.54 [00:15:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.95, 3.42, 3.99 [00:17:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.06, 3.07, 3.31 [00:18:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.67, 3.47, 3.69 [00:20:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.79, 3.92, 3.80 [00:20:00] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.78, 2.33, 1.46 [00:21:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.20, 4.12, 3.66 [00:21:53] !log [universalomega@mwtask111] finished deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all - SUCCESS in 504s [00:22:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:01] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:24:06] !log [@mwtask111] starting deploy of {'config': True} to all [00:24:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:15] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 8s [00:24:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:25] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [00:24:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/31ae7c4d8b7b...49ebe7df931b [00:24:58] [02miraheze/mw-config] 07Universal-Omega 0349ebe7d - Set $smwgUpgradeKey [00:25:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.48, 3.50, 3.51 [00:25:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.01, 2.66, 3.32 [00:26:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.82, 3.85 [00:26:00] miraheze/mw-config - Universal-Omega the build passed. [00:27:04] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/49ebe7df931b...56599b2eba5c [00:27:06] [02miraheze/mw-config] 07Universal-Omega 0356599b2 - test without enableSemantics [00:27:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.13, 2.92, 3.29 [00:28:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.63, 3.74, 3.79 [00:28:01] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:28:05] miraheze/mw-config - Universal-Omega the build passed. [00:30:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.15, 3.30, 3.61 [00:30:13] !log sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=smwwikibeta on test101 [00:30:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.07, 3.85, 3.77 [00:32:05] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [00:32:06] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:32:13] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:57] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:01] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:33:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.36, 3.36 [00:33:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.59, 3.57, 3.53 [00:34:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.41, 3.62 [00:35:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.49, 2.87, 3.17 [00:35:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.75, 4.14, 3.74 [00:36:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.63, 3.66 [00:37:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.99, 3.96, 3.72 [00:38:01] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:39:31] !log [universalomega@test101] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [00:39:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:41] !log [universalomega@test101] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all [00:39:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.80, 4.27, 3.89 [00:39:55] !log [universalomega@test101] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [00:40:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.87, 3.75 [00:40:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:00] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 0.86, 1.53, 1.94 [00:44:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 3.85, 3.75 [00:46:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.83, 3.76 [00:49:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.52, 3.50, 3.26 [00:50:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 4.32, 3.99 [00:52:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.86, 3.86 [00:52:15] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [00:52:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.66, 3.95, 3.53 [00:53:01] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:53:19] !log [@mwtask111] starting deploy of {'config': True} to all [00:53:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:27] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 8s [00:53:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:00] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 0.69, 1.16, 1.58 [00:58:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.71, 3.74 [00:58:01] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:00:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.83, 3.61, 3.70 [01:00:41] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [01:02:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.28, 3.13, 3.33 [01:08:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.00, 2.60, 3.21 [01:08:01] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:14:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.24, 3.59, 3.37 [01:18:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.25, 3.21, 3.29 [01:18:01] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:25:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.16, 3.96 [01:27:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.03, 4.76, 4.43 [01:30:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.95, 3.41, 3.24 [01:32:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.62, 4.03, 3.47 [01:33:01] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:34:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.75, 3.44 [01:40:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.73, 3.52 [01:42:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.62, 3.50 [01:43:01] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:44:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.73, 3.54 [01:48:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.69, 3.09, 3.36 [01:48:01] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:53:23] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [01:53:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 3.87, 3.28 [01:54:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.58, 3.47 [01:55:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.81, 3.34 [01:56:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.55, 4.19, 3.71 [01:59:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 4.17, 3.59 [02:02:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.53, 3.69, 3.69 [02:02:05] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [02:02:05] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:03:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.69, 3.98, 3.64 [02:05:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.95, 3.67 [02:06:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 4.12, 3.85 [02:07:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.65, 3.59 [02:10:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.81, 3.79 [02:13:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.04, 4.11, 3.77 [02:14:00] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.37, 1.77, 1.37 [02:14:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.32, 4.39, 3.98 [02:15:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.68, 2.10, 1.40 [02:16:00] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.23, 1.62, 1.37 [02:17:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.74, 3.53, 3.63 [02:18:01] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:18:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.10, 10.33, 8.48 [02:19:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.86, 3.63, 3.63 [02:20:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.92, 9.81, 8.50 [02:21:23] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [02:21:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.29, 1.93, 1.58 [02:22:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.88, 3.90 [02:23:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.93, 3.30, 3.51 [02:23:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.57, 1.49 [02:24:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.95, 3.92 [02:26:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.65, 3.81 [02:28:01] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:29:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.18, 3.12, 3.35 [02:30:41] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [02:31:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.34, 3.32, 3.98 [02:33:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.96, 4.43, 4.26 [02:34:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.05, 3.79, 3.75 [02:34:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.69, 11.00, 9.81 [02:35:59] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.65, 9.43, 9.38 [02:36:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.76, 3.25, 3.55 [02:38:01] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:40:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.79, 3.05, 3.40 [02:46:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.86, 4.01, 3.66 [02:48:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.96, 3.70 [02:55:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.27, 3.07 [02:57:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.71, 3.26 [02:57:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.82, 1.68, 1.30 [02:58:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.19, 3.04, 3.38 [02:59:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.69, 3.25, 3.14 [02:59:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.21, 1.51, 1.28 [03:15:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.88, 3.55, 3.34 [03:21:06] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.09, 3.30, 3.35 [03:23:01] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:33:01] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:33:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.98, 3.55, 3.97 [03:37:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.96, 3.50 [03:37:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.68, 3.89, 4.02 [03:39:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.27, 3.65, 3.44 [03:40:14] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.30, 2.93 [03:40:20] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 15.92, 11.16, 9.06 [03:41:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.62, 3.37, 3.79 [03:42:10] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.29, 3.19, 2.93 [03:42:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.98, 10.24, 8.98 [03:42:53] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.19, 3.28, 3.36 [03:43:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:43:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.23, 3.64, 3.83 [03:44:13] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.96, 9.53, 8.88 [03:49:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.38, 3.92, 3.97 [03:50:00] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.12, 1.55, 1.22 [03:51:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:51:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 4.37, 4.13 [03:52:00] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.25, 1.43, 1.21 [03:54:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 3.62, 3.38 [03:56:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.52, 3.27, 3.29 [04:00:15] Hdy [04:00:19] Hi [04:01:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.32, 10.42, 9.10 [04:01:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.35, 9.82, 8.87 [04:01:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:03:18] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.49, 10.47, 9.21 [04:05:17] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.23, 10.05, 9.26 [04:05:18] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.50, 9.80, 9.13 [04:17:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.37, 4.01, 3.59 [04:19:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.75, 3.54 [04:21:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.56, 4.09, 3.69 [04:21:31] U [04:21:36] riddle me this [04:21:44] What does a liar do when he is dead [04:23:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.54, 3.56, 3.55 [04:24:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:26:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.99, 3.72 [04:28:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.68, 3.65 [04:29:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:32:05] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [04:32:05] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:38:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.14, 2.74, 3.23 [04:41:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.42, 3.20, 3.96 [04:51:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.00, 2.43, 3.22 [05:00:41] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [05:08:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.79, 4.48, 3.71 [05:15:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:18:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.80, 3.39, 3.72 [05:20:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:22:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.83, 4.12, 3.90 [05:24:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 3.52, 2.95 [05:24:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.33, 3.57, 2.86 [05:25:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.29, 3.05, 2.76 [05:26:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.71, 3.20, 2.90 [05:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:31:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:37:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.21, 5.06, 2.60 [05:38:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:41:04] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [05:41:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.92, 3.29, 1.89 [05:43:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:45:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.73, 1.71, 1.92 [05:45:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.90, 1.91, 1.64 [05:45:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.41, 3.02, 3.77 [05:47:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.60, 1.47, 1.51 [05:49:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.91, 1.17, 1.63 [05:51:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.08, 2.16, 3.16 [06:09:03] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:09:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.15, 4.25, 3.44 [06:11:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.64, 3.70, 3.34 [06:13:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.69, 4.28, 3.57 [06:15:07] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=rolmaciawiki --report 1 --username-prefix=wikia:rolbenigno --no-updates /home/reception/esrolbenigno_pages_full.xml (START) [06:15:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:15:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.34, 4.00, 3.08 [06:16:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.98, 3.78, 2.90 [06:18:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.07, 3.43, 2.88 [06:22:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.81, 3.79, 3.14 [06:23:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.02, 3.88, 3.46 [06:24:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.55, 3.12 [06:25:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.52, 3.85, 3.27 [06:26:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.37, 3.12, 3.24 [06:27:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.98, 3.36, 3.15 [06:34:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.06, 2.89, 3.80 [06:36:12] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=rolmaciawiki --report 1 --username-prefix=wikia:rolbenigno --no-updates /home/reception/esrolbenigno_pages_full.xml (END - exit=0) [06:36:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:37:10] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=rolmaciawiki (START) [06:37:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:41:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.72, 2.51, 3.30 [07:02:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:09:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.63, 3.30 [07:11:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.09, 1.77, 1.35 [07:11:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.79, 2.87, 3.05 [07:12:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:13:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.76, 1.40, 1.26 [07:31:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.38, 3.27, 2.62 [07:32:05] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [07:33:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 53% [07:33:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.11, 2.84, 2.55 [07:34:15] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 5.74, 3.46, 1.81 [07:34:30] PROBLEM - cp20 APT on cp20 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [07:35:13] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.13, 2.55, 1.40 [07:35:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 52% [07:36:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.03, 10.13, 8.80 [07:36:35] RECOVERY - cp20 APT on cp20 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [07:36:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.21, 8.90, 7.35 [07:38:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.79, 9.81, 8.84 [07:39:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [07:39:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:39:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [07:39:55] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=rolmaciawiki (END - exit=0) [07:40:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:40:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.91, 9.04, 7.82 [07:41:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 58% [07:43:03] PROBLEM - cp20 APT on cp20 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [07:43:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.60, 4.19, 3.40 [07:44:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:45:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 54% [07:45:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.32, 3.53, 3.26 [07:47:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [07:47:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [07:47:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.11, 3.94, 3.45 [07:49:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 55% [07:49:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 52% [07:49:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 3.60, 3.39 [07:50:20] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [07:51:21] RECOVERY - cp20 APT on cp20 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [07:51:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.69, 4.42, 3.71 [07:52:20] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [07:53:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:53:38] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.88, 3.42, 2.56 [07:54:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.44, 2.73 [07:55:34] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.50, 2.69 [07:55:42] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [07:56:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.69, 2.92 [07:57:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [07:57:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.19, 3.99, 2.96 [07:57:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 46% [07:58:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 4.07, 3.16 [07:59:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 55% [08:00:41] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [08:00:41] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:01:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [08:01:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [08:03:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.24, 3.87, 3.29 [08:03:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 58% [08:04:08] PROBLEM - cp20 APT on cp20 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [08:05:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [08:05:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 51% [08:06:10] RECOVERY - cp20 APT on cp20 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [08:07:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.92, 3.43 [08:09:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 44% [08:13:21] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 31% [08:15:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [08:17:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [08:19:09] Here [08:19:18] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.16, 10.24, 9.13 [08:19:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.01, 3.33, 3.64 [08:19:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 58% [08:19:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 47% [08:20:19] Weird [08:21:18] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.56, 9.87, 9.13 [08:21:18] Reception123: [08:21:21] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.00, 3.84, 3.78 [08:21:22] paladox: [08:21:41] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 31% [08:23:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.64, 3.72 [08:23:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [08:23:22] around now [08:23:40] Reception123: -security [08:23:46] that's what I imagined [08:25:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 52% [08:26:32] PROBLEM - cp20 APT on cp20 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [08:27:56] PROBLEM - cp21 APT on cp21 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [08:28:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.73, 3.57, 3.93 [08:29:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 51% [08:29:59] RECOVERY - cp21 APT on cp21 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [08:30:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.80, 3.97 [08:30:06] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [08:31:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.91, 2.88, 3.36 [08:31:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [08:32:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.23, 3.38, 3.81 [08:33:21] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 26% [08:33:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 52% [08:35:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.17, 10.73, 9.47 [08:35:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [08:37:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [08:37:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.20, 10.58, 9.58 [08:37:41] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 31% [08:39:04] RECOVERY - cp20 APT on cp20 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [08:39:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.81, 9.98, 9.48 [08:40:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 3.63, 3.69 [08:41:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 44% [08:41:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 54% [08:42:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.85, 3.54, 3.66 [08:43:27] PROBLEM - cp20 APT on cp20 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [08:44:27] !log added an IP to private abuse list [08:44:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:45:21] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 39% [08:46:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 3.73, 3.68 [08:47:36] RECOVERY - cp20 APT on cp20 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [08:47:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.83, 10.47, 9.67 [08:48:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.47, 3.23, 3.50 [08:49:04] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [08:49:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.43, 10.32, 9.74 [08:49:42] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 34% [08:49:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.04, 11.15, 10.03 [08:51:03] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [08:51:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.34, 9.78, 9.60 [08:51:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.34, 10.56, 9.95 [08:52:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.62, 3.60 [08:54:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.51, 3.36, 3.50 [08:55:34] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.27, 9.51, 9.69 [08:56:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.49, 3.11, 3.40 [08:57:20] PROBLEM - puppet111 Current Load on puppet111 is WARNING: WARNING - load average: 5.03, 5.12, 3.31 [08:57:28] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown): File[/etc/puppetlabs/puppet/auth.conf],File[/etc/puppetlabs/puppet/fileserver.conf],File[/etc/puppetlabs/puppet/hiera.yaml],File[/etc/sysctl.d] [08:57:45] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:57:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:58:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:58:34] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:59:04] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:59:07] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:59:20] RECOVERY - puppet111 Current Load on puppet111 is OK: OK - load average: 2.75, 4.50, 3.32 [08:59:23] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:59:23] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:59:27] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:00:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.51, 3.31 [09:01:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.06, 3.38, 3.91 [09:02:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.60, 2.88, 3.11 [09:05:23] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [09:05:23] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [09:07:03] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:07:27] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [09:07:45] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:08:34] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [09:09:07] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:09:11] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.05, 1.01, 1.95 [09:09:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:09:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.78, 3.92, 3.87 [09:10:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.47, 3.36, 3.19 [09:10:47]