[00:00:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.52, 3.60, 3.57 [00:00:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.73, 3.39 [00:01:15] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 1874 MB (19% inode=80%); [00:02:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.82, 3.95, 3.69 [00:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:04:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.18, 4.33, 3.66 [00:04:46] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 515 failures. Last run 3 minutes ago with 515 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:05:40] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:06:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.78, 3.74, 3.53 [00:08:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.90, 3.85 [00:09:26] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.22, 6.48, 5.15 [00:09:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.79, 2.70, 3.26 [00:09:54] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [00:10:17] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.63, 3.23, 3.38 [00:12:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.25, 4.13, 3.90 [00:15:26] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.72, 6.51, 5.65 [00:18:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.72, 3.79 [00:19:26] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.30, 7.13, 6.14 [00:23:26] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 10.37, 7.99, 6.68 [00:25:26] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.01, 6.47, 6.28 [00:28:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.12, 2.74, 3.25 [00:32:43] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:36:18] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.82, 7.67, 6.82 [00:38:14] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.00, 7.08, 6.72 [00:40:09] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.95, 6.63, 6.59 [00:44:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.49, 3.39 [00:46:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.39, 3.25, 3.32 [00:53:59] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:56:05] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [01:18:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:20:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.72, 1.40 [01:21:25] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [01:22:51] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.09, 1.74, 1.43 [01:24:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:26:52] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.28, 1.89, 1.63 [01:27:45] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [01:30:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.82, 2.25, 1.80 [01:32:52] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.27, 1.84, 1.70 [01:34:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:36:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.19, 1.95, 1.78 [01:37:26] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.25, 6.52, 5.86 [01:39:26] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.09, 6.12, 5.78 [01:42:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:42:53] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.15, 1.84, 1.82 [01:44:53] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.45, 1.35, 1.64 [01:46:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.74, 3.98, 3.37 [01:52:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.87, 3.14, 3.25 [01:52:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:54:53] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.22, 2.36, 2.07 [01:57:12] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.68, 7.12, 6.52 [01:58:05] PROBLEM - wiki.emsoc.ie - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.emsoc.ie could not be found [02:00:53] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.29, 1.85, 1.95 [02:01:03] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.09, 6.63, 6.43 [02:05:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:06:54] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.60, 2.23, 2.05 [02:08:54] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.42, 1.86, 1.93 [02:12:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 3.85, 3.36 [02:14:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.81, 3.40 [02:18:54] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.78, 1.23, 1.63 [02:20:54] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.69, 2.22, 1.51 [02:22:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.15, 3.15, 3.30 [02:22:53] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.60, 1.65, 1.39 [02:35:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:39:51] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 3.79, 3.27 [02:41:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.01, 3.57, 3.25 [02:41:57] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 3.63, 3.01 [02:42:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:43:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.56, 3.96, 3.44 [02:45:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.78, 3.44 [02:45:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.51, 3.66, 3.23 [02:47:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:47:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.37, 3.17 [02:48:18] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.27, 0.39, 1.92 [02:49:26] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.68, 3.20, 3.28 [02:50:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.58, 1.84, 1.30 [02:51:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.45, 3.24 [02:52:18] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.19, 0.65, 1.66 [02:53:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.32, 3.34, 3.22 [02:56:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.09, 1.77, 1.50 [02:57:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.80, 4.02, 3.57 [02:58:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.92, 1.45, 1.41 [02:59:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.46, 3.17, 2.70 [02:59:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.89, 3.70, 3.51 [03:00:18] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.73, 1.36, 1.53 [03:01:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.32, 3.98, 3.06 [03:02:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 3.59, 3.28 [03:02:18] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.35, 1.72, 1.64 [03:03:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.93, 3.56, 3.01 [03:04:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.44, 3.26 [03:04:18] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.89, 1.80, 1.68 [03:05:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.58, 4.89, 4.02 [03:06:18] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.87, 1.52, 1.60 [03:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:08:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 2.98, 3.12 [03:11:16] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.11, 3.36, 3.15 [03:12:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.18, 3.72, 3.37 [03:14:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.89, 3.48, 3.31 [03:15:30] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 8.36, 2.68, 1.41 [03:15:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.68, 3.88 [03:17:29] dmehus: I wonder why that rename keeps failing. [03:18:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.32, 3.19, 3.23 [03:18:20] I'm mobile right now or I'd run the script. [03:19:29] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.20, 1.77, 1.32 [03:21:29] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.85, 1.46, 1.25 [03:21:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:21:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.47, 2.35, 3.22 [03:23:46] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/979b0eb1e272...4ccb4f92818e [03:23:47] [02miraheze/ManageWiki] 07Universal-Omega 034ccb4f9 - Update ManageWikiTypes.php [03:23:49] [02ManageWiki] 07Universal-Omega synchronize pull request 03#348: Fix users and wikipages type to always be array, not string - 13https://github.com/miraheze/ManageWiki/pull/348 [03:25:26] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.99, 6.49, 5.95 [03:26:45] [02mw-config] 07Universal-Omega reviewed pull request 03#4545 commit - 13https://github.com/miraheze/mw-config/pull/4545#discussion_r842302598 [03:27:56] miraheze/ManageWiki - Universal-Omega the build passed. [03:29:26] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.40, 6.42, 6.09 [03:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:37:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.43, 3.33 [03:39:26] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.74, 7.32, 6.50 [03:39:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.70, 2.79, 3.11 [03:40:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:41:26] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.81, 7.12, 6.54 [03:43:26] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.32, 6.49, 6.38 [03:43:57] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 7.79, 4.61, 2.41 [03:47:36] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [03:47:36] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:49:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.83, 1.97, 1.91 [03:52:12] PROBLEM - www.obeymewiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.obeymewiki.com All nameservers failed to answer the query. [03:55:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.51, 1.11, 1.56 [04:12:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.85, 3.42, 2.99 [04:14:02] CosmicAlpha, yeah, I think it's related to the job queue changes or something we made back in August/September that's causing the job queue to act up and renames to fail / take forever. Not sure exactly, but worth SRE exploring as time permits [04:14:16] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.66, 3.24, 2.98 [04:14:17] and that's fine, Reception123 or RhinosF1 can run it later [04:14:31] Hopefully @Eytirth doesn't login to their wiki account right now again ;) [04:16:26] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [04:22:03] RECOVERY - www.obeymewiki.com - reverse DNS on sslhost is OK: SSL OK - www.obeymewiki.com reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [04:38:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.70, 4.15, 3.24 [04:40:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.97, 3.31 [04:44:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.95, 2.94, 3.05 [04:47:13] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [04:47:13] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:54:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.78, 3.65, 2.92 [04:58:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.65, 3.27, 2.94 [05:13:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.64, 3.65, 3.18 [05:15:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.51, 3.18 [05:16:03] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [05:17:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.05, 3.32, 3.15 [05:18:10] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.44, 6.95, 6.09 [05:20:05] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.28, 7.04, 6.23 [05:20:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.15, 4.00, 3.49 [05:23:56] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.48, 7.69, 6.67 [05:25:51] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.44, 7.19, 6.61 [05:26:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.38, 2.02, 1.21 [05:27:47] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.84, 6.69, 6.50 [05:28:00] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.92, 1.56, 1.13 [05:28:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.76, 3.71 [05:30:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 4.17, 3.88 [05:33:35] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.34, 7.27, 6.75 [05:34:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.92, 3.70, 3.75 [05:35:30] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.52, 6.74, 6.63 [05:38:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 11.47, 5.60, 4.33 [05:39:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.10, 3.69, 2.83 [05:40:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.76, 4.58, 3.65 [05:41:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.70, 3.23, 2.77 [05:45:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.91, 5.44, 3.79 [05:49:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.31, 3.43, 3.37 [05:51:08] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=bigforestwiki /srv/mediawiki/w/extensions/VoteNY/sql/vote.mysql [05:51:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:51:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.52, 3.19, 3.29 [05:56:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.22, 3.30, 3.96 [06:00:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.70, 2.64, 3.78 [06:02:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.31, 2.48, 3.36 [06:06:07] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 3.48, 3.74 [06:08:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.61, 3.78 [06:08:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 3.25, 3.39 [06:10:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.75, 2.60, 3.14 [06:12:03] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.22, 2.47, 1.45 [06:14:03] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.45, 1.71, 1.29 [06:16:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.07, 1.51, 1.27 [06:17:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.90, 3.03, 3.38 [06:18:18] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.06, 1.68, 1.25 [06:20:18] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.87, 1.73, 1.32 [06:24:18] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.79, 2.13, 1.57 [06:25:32] PROBLEM - wiki.xysspon.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.xysspon.com could not be found [06:28:18] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.22, 1.74, 1.54 [06:32:18] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.06, 1.84, 1.62 [06:34:18] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.94, 1.89, 1.67 [06:36:18] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.03, 1.94, 1.71 [06:38:18] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.66, 1.82, 1.70 [06:44:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.51, 4.07, 3.40 [06:46:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.07, 3.90, 3.43 [06:48:35] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.29, 3.34, 3.28 [06:52:18] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.30, 2.08, 1.89 [06:56:18] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.43, 1.99, 1.92 [06:57:35] [02mw-config] 07Universal-Omega edited pull request 03#4546: Only explode if string - 13https://github.com/miraheze/mw-config/pull/4546 [06:59:26] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.26, 7.02, 5.84 [07:01:26] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.72, 7.14, 6.02 [07:04:18] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.02, 1.87, 1.87 [07:05:26] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.95, 5.97, 5.84 [07:06:18] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.96, 1.89, 1.88 [07:12:18] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.57, 2.06, 1.93 [07:15:17] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [07:40:18] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.47, 1.82, 1.98 [07:44:06] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [07:44:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 2.87, 2.48 [07:46:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.25, 2.55, 2.40 [07:50:18] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.64, 1.30, 1.70 [07:55:14] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [08:00:37] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:21:14] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:28:04] Reception123: we just paged [08:30:07] Reception123: i closed the alert but jobs was high [08:30:37] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:51:10] ack [08:51:26] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.31, 7.21, 6.17 [08:51:50] Reception123: it was right on limit [08:53:26] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.38, 6.61, 6.08 [08:53:42] hmm [09:09:25] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.11, 3.36, 2.66 [09:11:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.98, 2.77, 2.52 [09:14:31] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [09:14:31] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:43:20] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [09:53:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:13:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:24:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.38, 3.04 [10:26:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.70, 3.19 [10:27:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:28:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.69, 3.40, 3.15 [10:32:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.61, 4.45, 3.66 [10:36:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.04, 4.01, 3.25 [10:37:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:38:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.02, 8.08, 7.24 [10:38:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.71, 3.62, 3.20 [10:40:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.17, 3.15, 3.08 [10:42:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.72, 7.42, 7.21 [10:52:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.89, 3.68, 3.98 [10:56:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.11, 3.52, 3.84 [10:58:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.82, 2.97, 3.60 [11:04:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.91, 2.54, 3.26 [11:12:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:14:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.62, 4.36, 3.44 [11:20:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.57, 3.47 [11:22:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.79, 3.78, 3.56 [11:22:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:22:47] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.35, 3.13, 3.32 [11:24:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.17, 3.86, 3.61 [11:26:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.22, 3.71, 3.60 [11:27:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:28:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.89, 4.28, 3.83 [11:32:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.64, 3.67 [11:32:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:34:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.44, 3.28 [11:36:30] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.58, 3.34 [11:38:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.95, 3.53, 3.37 [11:40:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.38, 2.92, 3.35 [11:40:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.25, 3.37, 3.32 [11:41:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:46:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:46:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.10, 3.78, 3.54 [11:48:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.33, 3.14, 3.33 [12:18:18] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.74, 1.76, 1.46 [12:20:18] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.49, 1.97, 1.57 [12:20:54] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.07, 9.31, 8.53 [12:22:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.52, 8.61, 7.63 [12:22:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.54, 9.28, 8.61 [12:26:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.91, 8.18, 7.68 [12:28:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.36, 2.91 [12:30:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.00, 3.16, 2.88 [12:30:37] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:36:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.66, 8.30, 8.48 [12:39:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.18, 8.75, 7.76 [12:41:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.43, 8.06, 7.62 [12:42:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.70, 8.84, 8.41 [12:44:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.87, 8.27, 8.26 [12:45:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:48:13] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.65, 1.80, 1.35 [12:48:30] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.19, 9.00, 8.54 [12:48:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.31, 9.37, 8.57 [12:49:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.63, 9.22, 8.65 [12:50:04] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.40, 3.54, 3.01 [12:50:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.46, 8.60, 8.43 [12:50:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.26, 9.11, 8.58 [12:52:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.00, 3.34, 3.00 [12:52:10] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.93, 1.60, 1.41 [12:52:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.40, 8.14, 8.28 [12:53:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.76, 9.19, 8.76 [12:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:56:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.63, 1.73, 1.52 [12:56:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.22, 9.61, 8.83 [12:56:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.51, 8.41, 8.46 [12:57:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.09, 3.52, 3.16 [12:57:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.08, 8.21, 8.42 [12:58:05] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.47, 1.28, 1.38 [12:58:20] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [12:58:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.91, 9.54, 8.89 [12:59:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.56, 4.41, 3.53 [12:59:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.27, 8.16, 7.78 [12:59:57] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 3.77, 3.16 [13:03:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.03, 9.06, 8.70 [13:03:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.90, 3.03, 3.02 [13:04:22] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.78, 7.96, 8.38 [13:04:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.13, 3.25, 3.35 [13:05:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.97, 9.54, 8.93 [13:07:00] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.55, 8.31, 8.15 [13:07:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.93, 8.95, 8.48 [13:10:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.80, 3.95, 3.39 [13:12:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.24, 3.64, 3.34 [13:15:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.93, 7.95, 8.37 [13:16:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.62, 4.01, 3.58 [13:17:17] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.26, 7.46, 8.35 [13:18:06] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.94, 6.43, 4.87 [13:19:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.18, 3.18 [13:20:06] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.52, 6.15, 4.97 [13:20:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.32, 3.50, 3.50 [13:22:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.02, 4.05, 3.69 [13:23:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 3.82, 3.42 [13:23:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.22, 8.68, 8.50 [13:24:21] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 3.97, 3.34 [13:24:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.66, 3.73, 3.61 [13:26:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.60, 3.95, 3.69 [13:28:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.95, 3.46 [13:28:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.40, 3.72, 3.58 [13:29:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.13, 8.33, 8.49 [13:30:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.23, 4.54, 3.88 [13:31:55] [02puppet] 07RhinosF1 opened pull request 03#2461: jobrunner-high: allow running refreshLinks with higher memory - 13https://github.com/miraheze/puppet/pull/2461 [13:32:11] [02puppet] 07RhinosF1 edited pull request 03#2461: jobrunner-high: allow running refreshLinks with higher memory - 13https://github.com/miraheze/puppet/pull/2461 [13:32:12] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.90, 4.07, 3.59 [13:33:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.51, 9.02, 8.72 [13:34:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.10, 3.78, 3.55 [13:35:41] [02mw-config] 07HM100 synchronize pull request 03#4545: Add more Evelution Configuration settings - 13https://github.com/miraheze/mw-config/pull/4545 [13:35:54] [02mw-config] 07HM100 synchronize pull request 03#4545: Add more Evelution Configuration settings - 13https://github.com/miraheze/mw-config/pull/4545 [13:36:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.97, 4.09, 3.70 [13:36:08] [02mw-config] 07HM100 reviewed pull request 03#4545 commit - 13https://github.com/miraheze/mw-config/pull/4545#discussion_r842794025 [13:39:17] [02puppet] 07paladox closed pull request 03#2461: jobrunner-high: allow running refreshLinks with higher memory - 13https://github.com/miraheze/puppet/pull/2461 [13:39:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/80c4d20fe772...cba1f05cc054 [13:39:20] [02miraheze/puppet] 07RhinosF1 03cba1f05 - jobrunner-high: allow running refreshLinks with higher memory (#2461) [13:39:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.32, 9.65, 9.05 [13:40:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.10, 3.67, 3.63 [13:40:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.96, 3.93 [13:41:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.38, 9.02, 8.90 [13:42:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.13, 3.46, 3.89 [13:44:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.53, 3.06, 3.39 [13:46:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.08, 5.01, 4.28 [13:46:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.17, 4.42, 4.09 [13:47:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.32, 8.93, 8.81 [13:47:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.82, 4.08, 3.72 [13:49:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.81, 3.48, 3.54 [13:51:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.81, 8.95, 8.94 [13:54:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.66, 3.48, 3.87 [13:54:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.48, 8.05, 7.70 [13:56:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.84, 8.00, 7.72 [13:57:26] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.11, 6.80, 6.18 [13:57:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.73, 8.46, 8.24 [13:57:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.88, 3.18, 3.40 [13:58:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.34, 3.85 [14:00:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 3.89, 3.98 [14:01:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.54, 8.13, 7.67 [14:01:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.37, 8.40, 8.28 [14:03:26] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.06, 7.47, 6.73 [14:03:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.04, 3.02, 3.39 [14:05:26] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.19, 7.32, 6.79 [14:05:29] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.58, 8.33, 7.90 [14:07:26] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.74, 6.77, 6.65 [14:07:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.63, 8.64, 8.49 [14:08:03] [02dns] 07MacFan4000 opened pull request 03#262: Add buildabearwiki.info zone - 13https://github.com/miraheze/dns/pull/262 [14:09:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.87, 8.50, 8.46 [14:13:33] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.11, 4.26, 3.65 [14:13:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 4.14, 3.71 [14:14:06] [02dns] 07MacFan4000 opened pull request 03#263: add discordian.wiki zone - 13https://github.com/miraheze/dns/pull/263 [14:17:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.89, 1.38, 0.95 [14:17:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.81, 8.40, 8.24 [14:19:05] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.10, 1.22, 0.94 [14:19:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.79, 3.75, 3.66 [14:19:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.53, 9.52, 9.03 [14:19:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.98, 8.26, 8.21 [14:19:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.80, 3.78, 3.75 [14:20:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.56, 8.97, 8.59 [14:21:25] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.91, 3.72 [14:21:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.91, 9.36, 9.04 [14:22:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.30, 8.26, 8.37 [14:22:52] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6a19642ac994...d22b6e350403 [14:22:53] [02miraheze/ssl] 07MacFan4000 03d22b6e3 - renew wiki.rosestulipsandliberty.com [14:23:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.56, 3.59, 3.63 [14:23:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.62, 4.07, 3.89 [14:25:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.99, 9.52, 9.11 [14:25:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.85, 3.84 [14:27:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 3.84, 3.70 [14:27:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.95, 8.62, 8.84 [14:28:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.72, 9.02, 8.65 [14:29:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.55, 3.50, 3.61 [14:29:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.42, 8.86, 8.42 [14:30:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.27, 8.33, 8.44 [14:31:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.84, 7.80, 8.09 [14:31:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.11, 7.05, 8.18 [14:31:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.95, 4.33, 3.95 [14:34:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:35:10] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.01, 4.84, 4.10 [14:37:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.44, 8.95, 8.36 [14:39:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.80, 8.77, 8.37 [14:41:41]