[00:01:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.46, 3.45, 3.75 [00:01:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.01, 8.47, 8.97 [00:04:05] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 7416 MB (18% inode=96%); [00:04:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.99, 9.23, 9.19 [00:04:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.83, 9.59, 9.73 [00:05:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:05:38] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 0.99, 2.20, 1.58 [00:06:26] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:06:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.95, 2.39, 3.19 [00:07:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.05, 9.99, 9.49 [00:08:34] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.57, 1.51, 1.42 [00:08:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:08:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.85, 9.18, 8.90 [00:10:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.44, 9.29, 9.31 [00:15:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.47, 9.25, 9.01 [00:15:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.62, 8.72, 8.19 [00:15:25] !log [void@mwtask111] starting deploy of {'config': True, 'world': True, 'l10n': True} to all [00:15:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.71, 9.47, 9.90 [00:17:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.04, 3.42, 3.37 [00:18:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.20, 8.08, 8.05 [00:20:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.41, 3.69, 3.45 [00:21:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.01, 7.64, 8.39 [00:23:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.17, 9.31, 9.67 [00:23:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.31, 3.06, 3.25 [00:28:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.25, 9.33, 8.94 [00:28:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.20, 8.41, 8.18 [00:28:51] !log [rhinos@mwtask111] global token reset done [00:29:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:12] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [00:31:49] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.37, 9.00, 8.42 [00:32:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.86, 9.97, 9.94 [00:34:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.42, 9.75, 9.19 [00:34:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.49, 8.70, 8.44 [00:35:10] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.88, 10.41, 9.58 [00:37:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.76, 8.50, 8.79 [00:38:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.82, 9.13, 9.26 [00:38:43] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.01, 1.80, 1.45 [00:40:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.98, 7.65, 8.40 [00:40:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.03, 9.53, 9.61 [00:41:43] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.18, 1.51, 1.40 [00:43:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.91, 7.82, 8.17 [00:46:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.26, 8.61, 9.22 [00:46:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.36, 7.42, 8.36 [00:46:50] !log [void@mwtask111] finished deploy of {'config': True, 'world': True, 'l10n': True} to all - SUCCESS in 1884s [00:46:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:50:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.24, 8.47, 8.35 [00:53:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.82, 7.88, 8.16 [00:54:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 3.53, 3.04 [00:58:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.87, 9.27, 9.11 [01:00:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.25, 3.19, 3.13 [01:01:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.89, 9.01, 9.04 [01:03:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.43, 9.00, 8.99 [01:06:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.17, 9.16, 9.10 [01:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:15:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.74, 9.00, 8.41 [01:18:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.53, 9.52, 9.11 [01:21:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.46, 8.80, 8.89 [01:21:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.58, 7.98, 8.20 [01:26:57] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.33, 7.87, 8.46 [01:33:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:36:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.34, 9.84, 9.21 [01:37:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.39, 10.09, 8.95 [01:39:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.23, 10.84, 9.71 [01:40:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.68, 9.47, 8.91 [01:40:21] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.23, 9.74, 8.03 [01:41:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.61, 10.01, 8.85 [01:43:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.02, 9.49, 9.00 [01:43:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.01, 9.59, 8.30 [01:43:45] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.61, 9.44, 8.39 [01:46:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.23, 8.99, 8.90 [01:49:34] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.50, 10.05, 8.89 [01:49:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.43, 9.97, 9.29 [01:50:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 5.43, 8.70, 9.03 [01:52:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.75, 9.43, 8.85 [01:53:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.15, 9.30, 9.19 [01:55:34] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.13, 8.20, 8.47 [01:55:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.76, 9.82, 9.48 [01:56:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.76, 8.97, 9.08 [01:58:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.29, 9.77, 9.54 [02:02:02] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.25, 9.48, 9.40 [02:05:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.97, 9.16, 9.29 [02:08:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.71, 9.68, 9.45 [02:09:04] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.45, 8.79, 8.75 [02:10:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.66, 9.97, 9.72 [02:13:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.02, 8.89, 9.33 [02:13:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.67, 9.35, 9.43 [02:14:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.24, 9.77, 9.32 [02:19:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.96, 9.99, 9.69 [02:25:50] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.52, 9.53, 9.75 [02:28:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.12, 9.92, 9.87 [02:34:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.54, 9.56, 9.77 [02:37:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.54, 9.44, 9.04 [02:41:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.45, 9.18, 8.88 [02:43:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.86, 9.43, 9.19 [02:44:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.35, 9.49, 9.06 [02:48:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:49:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.28, 7.39, 8.34 [02:49:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.38, 7.35, 8.22 [02:51:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.93, 9.19, 9.88 [02:53:15] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.30, 7.51, 8.32 [02:54:05] PROBLEM - wiki.timewisely.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.timewisely.net could not be found [02:56:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.11, 9.02, 9.51 [03:02:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.64, 9.27, 8.54 [03:02:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.13, 8.45, 9.23 [03:04:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.90, 9.39, 8.74 [03:05:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.84, 9.63, 9.52 [03:08:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.40, 8.90, 9.26 [03:08:38] PROBLEM - wiki.timewisely.net - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.timewisely.net and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:08:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.34, 8.95, 8.27 [03:14:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.18, 8.89, 8.46 [03:20:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.84, 7.70, 8.10 [03:21:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:22:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.40, 7.84, 8.41 [03:24:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.44, 8.69, 8.43 [03:26:14] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 6.73, 3.09, 2.04 [03:26:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:29:13] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.80, 1.98, 1.80 [03:30:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.12, 8.14, 8.40 [03:31:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.17, 7.83, 8.36 [03:32:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.47, 1.37, 1.59 [03:36:10] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.24, 2.12, 1.46 [03:41:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.10, 8.33, 7.97 [03:41:57] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.27, 1.60, 1.43 [03:43:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.63, 9.04, 8.70 [03:44:51] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.35, 8.11, 7.92 [03:48:51] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.49, 8.19, 8.45 [03:58:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.33, 10.48, 9.14 [04:03:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:07:37] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:07:58] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:09:20] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:09:22] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:11:00] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:13:47] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw111 [04:22:40] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 11 backends are healthy [04:24:38] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.018 second response time [04:24:38] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [04:26:00] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.307 second response time [04:26:07] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22220 bytes in 0.617 second response time [04:26:17] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.306 second response time [04:30:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:40:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.96, 8.67, 7.22 [04:40:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:41:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.74, 9.39, 8.15 [04:42:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.13, 8.71, 7.17 [04:43:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.85, 8.76, 8.14 [04:45:08] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.80, 7.96, 7.16 [04:45:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:46:47] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.31, 8.40, 8.10 [04:47:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.93, 8.76, 9.99 [04:53:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.87, 8.60, 8.39 [04:54:59] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.87, 8.08, 7.92 [04:55:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:56:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.05, 7.99, 8.19 [04:57:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.46, 9.38, 8.14 [05:00:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.34, 7.71, 7.71 [05:07:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.57, 7.14, 8.28 [05:14:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:23:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:28:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:32:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:36:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.34, 9.01, 8.02 [05:39:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.00, 7.48, 7.58 [05:42:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.00, 8.58, 8.06 [05:45:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.24, 7.91, 7.89 [05:52:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:33:59] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.23, 1.77, 1.15 [06:36:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.40, 1.20, 1.03 [06:53:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:58:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:30:16] PROBLEM - parapedia.pl - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/pyth [07:30:16] -packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query parapedia.pl. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:57:25] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[postgresql-contrib-13] [07:59:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:05:45] taavi: replied, feel free to Dm if easier [08:09:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:19:17] taavi: should knowing what was in the session be logged then [08:19:25] We can't expect users to provide that [08:19:43] yes, I'd start with looking at all other log messages for the same request id [08:21:13] taavi: ok, I will check again later but I don't remember there being any [08:23:40] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [08:28:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.46, 2.55 [08:28:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.45, 2.66 [08:29:40] RECOVERY - parapedia.pl - reverse DNS on sslhost is OK: SSL OK - parapedia.pl reverse DNS resolves to cp20.miraheze.org - CNAME FLAT [08:31:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.51, 2.75, 2.44 [08:31:30] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.32, 3.17, 2.70 [08:35:26] RhinosF1 can you merge PR in mw-config? [08:35:29] https://github.com/miraheze/mw-config/pull/4469 [08:35:30] [url] T8712: Extented confirmed protection NPWiki by ugochimobi · Pull Request #4469 · miraheze/mw-config · GitHub | github.com [08:37:00] Mobile [08:37:05] o.o [08:54:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:04:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:28:07] Please who can purge the statistics of gratisdatawiki? [09:28:27] Reception123: ^ [09:38:22] ugochimobi: mind reminding me in 3 hours? [09:38:53] very well then, I will. [09:45:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:50:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:16:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:21:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:41:15] Noticed that Matomo seems to have stopped reporting visitor data. Did some numbers based on the numbers my wiki was previously getting and I think the start of the problem coincides roughly with these changes on the 15th of Jan: https://phabricator.miraheze.org/rPUPCd8f0074b6b03ef59caaf6b977c47061f58ba938c https://phabricator.miraheze.org/rPUPC9340fc17c6f42ac70ff993fb9a8f0285a38f411f [10:41:16] https://phabricator.miraheze.org/rPUPCf7dab6a17c25e382a9dfe9b7f1c972c530127121 [10:41:20] [url] rPUPCd8f0074b6b03 | phabricator.miraheze.org [10:41:20] [url] rPUPCf7dab6a17c25 | phabricator.miraheze.org [10:41:22] [url] rPUPC9340fc17c6f4 | phabricator.miraheze.org [10:56:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:01:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:14:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:29:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:34:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:36:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.88, 9.44, 8.36 [11:42:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.86, 8.36, 8.23 [11:48:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:48:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.06, 8.61, 8.43 [11:54:23] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.20, 8.36, 8.42 [11:59:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.44, 8.57, 8.33 [12:02:42] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.51, 7.98, 8.13 [12:03:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.02, 8.47, 8.52 [12:09:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.83, 7.79, 8.31 [12:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:40:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:45:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:57:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.58, 9.17, 8.12 [13:00:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.30, 7.95, 7.83 [13:02:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.58, 8.56, 7.64 [13:06:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.92, 9.46, 8.65 [13:07:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.83, 8.71, 8.08 [13:08:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:12:38] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.96, 9.71, 8.98 [13:13:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.06, 8.62, 7.98 [13:14:50] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.10, 7.91, 7.96 [13:15:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.79, 9.48, 9.06 [13:15:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.78, 8.20, 8.15 [13:16:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.53, 8.02, 7.85 [13:24:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.62, 9.60, 9.22 [13:26:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.77, 8.70, 8.97 [13:28:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:35:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.45, 10.02, 9.36 [13:36:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.31, 9.57, 8.86 [13:38:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.83, 9.18, 9.17 [13:41:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.50, 9.59, 9.32 [13:42:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.31, 9.53, 8.12 [13:43:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:44:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.68, 8.97, 9.15 [13:45:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.28, 8.88, 8.12 [13:45:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 9.87, 10.06, 8.94 [13:47:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.50, 8.42, 8.08 [13:48:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.57, 9.63, 8.98 [13:53:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.49, 9.14, 8.91 [13:54:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.58, 7.77, 8.33 [13:54:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.16, 9.28, 8.90 [13:56:53] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.25, 7.90, 8.44 [13:58:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:03:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.08, 7.87, 8.32 [14:04:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.69, 8.88, 8.76 [14:09:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.64, 8.35, 8.25 [14:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:10:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.14, 8.51, 8.46 [14:11:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.75, 9.03, 8.74 [14:12:18] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.14, 8.27, 8.24 [14:13:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.48, 9.63, 9.64 [14:14:53] Reception123 any luck? 0.0 [14:15:52] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.90, 7.60, 8.14 [14:17:42] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.58, 7.97, 8.34 [14:17:48] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=gratisdatawiki --active --update (END - exit=0) [14:17:49] ugochimobi: ^ [14:17:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:18:44] alright, thank. [14:19:01] when will it hopefully be done? [14:19:12] I mean finish [14:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:22:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.16, 9.26, 8.67 [14:24:53] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.32, 8.72, 8.82 [14:25:21] ugochimobi: it has finished [14:25:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.58, 9.14, 8.71 [14:28:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:30:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.68, 8.90, 9.00 [14:31:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 4.69, 7.44, 8.27 [14:33:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:39:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:39:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.84, 8.95, 8.19 [14:39:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.43, 10.05, 9.49 [14:42:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.79, 9.72, 9.46 [14:42:39] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.89, 8.25, 8.04 [14:42:40] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.52, 8.97, 8.15 [14:43:57] okay [14:44:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.77, 8.49, 7.98 [14:45:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.15, 7.74, 7.82 [14:47:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.69, 8.43, 8.05 [14:49:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:54:13] RhinosF1 still Mobile? [14:55:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:00:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:06:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.16, 9.45, 9.11 [15:07:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:08:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.67, 8.61, 8.06 [15:11:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.03, 8.19, 8.01 [15:12:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.61, 8.79, 8.15 [15:15:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.92, 9.06, 8.20 [15:17:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:18:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.02, 8.17, 8.11 [15:18:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.02, 8.75, 8.38 [15:18:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.72, 8.01, 7.94 [15:19:36] !log MariaDB [ftpwwiki]> ALTER TABLE searchindex ADD FULLTEXT INDEX si_title (si_title), ADD FULLTEXT INDEX si_text (si_text); [15:19:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:20:48] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.04, 8.68, 9.35 [15:21:09] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.12, 8.12, 8.21 [15:31:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.95, 8.49, 8.01 [15:31:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:33:42] JohnLewis: hi, any idea what could be causing https://phabricator.miraheze.org/T8719 ? I can't seem to figure out what's going on [15:33:43] [url] ⚓ T8719 Matomo seems to have stopped reporting | phabricator.miraheze.org [15:34:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.06, 8.77, 8.16 [15:35:31] JohnLewis: visits in real time seem to be displaying but not the other data like visitor logs or the daily visitors graph [15:35:39] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.80, 10.19, 9.37 [15:36:20] seems to be down since 13-14 Jan [15:36:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.81, 4.37, 3.30 [15:37:13] so actually that's just since the migration [15:37:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.90, 8.64, 8.35 [15:37:19] so something went wrong somewhere [15:37:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.35, 2.93 [15:37:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.55, 8.34, 8.12 [15:37:49] cc paladox [15:38:48] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.40, 9.49, 9.27 [15:39:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.94, 3.32 [15:40:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.98, 2.83, 2.82 [15:41:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.16, 10.08, 9.54 [15:42:11] [02mw-config] 07Reception123 closed pull request 03#4469: T8712: Extented confirmed protection NPWiki - 13https://github.com/miraheze/mw-config/pull/4469 [15:42:13] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c736ef6d2ab0...417720f3c087 [15:42:14] [02miraheze/mw-config] 07ugochimobi 03417720f - T8712: Extented confirmed protection NPWiki (#4469) [15:42:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.00, 8.77, 8.35 [15:42:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.36, 3.40, 3.23 [15:43:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.70, 9.94, 9.03 [15:43:34] miraheze/mw-config - Reception123 the build passed. [15:44:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.97, 9.79, 9.51 [15:47:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.81, 10.38, 9.77 [15:48:09] !log [@test101] starting deploy of {'config': True} to skip [15:48:10] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [15:48:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.82, 9.26, 8.64 [15:48:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:48:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:49:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.57, 8.55, 8.01 [15:49:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.30, 9.09, 8.95 [15:51:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.58, 9.77, 8.98 [15:51:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:52:13] Reception123: I won’t have a laptop for a week or a bit. It’s gone in for repairs [15:52:18] So I won’t be able to look [15:52:24] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.08, 8.48, 8.09 [15:52:51] paladox: oh I see :( I guess you don't have any idea what could've happened during the migration for it to not work? [15:52:52] !log [@mwtask111] starting deploy of {'config': True} to all [15:52:56] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [15:52:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:53:06] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 13s [15:53:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:53:20] No, run the script and see if it errors out [15:53:39] which one? [15:53:48] https://github.com/miraheze/puppet/blob/master/modules/matomo/manifests/init.pp#L87 [15:53:49] [url] puppet/init.pp at master · miraheze/puppet · GitHub | github.com [15:54:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.34, 9.63, 9.01 [15:56:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.41, 3.00 [15:58:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:00:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.86, 3.05, 2.93 [16:00:26] ah [16:00:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.44, 9.37, 9.21 [16:02:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.14, 9.08, 8.93 [16:04:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.73, 10.38, 9.68 [16:05:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.73, 9.11, 8.74 [16:06:03] paladox: hmm, shouldn't /usr/local/bin/runMatomoArchive.sh be owned by www-data if www-data is supposed to run it? [16:06:06] I think that may be the issue [16:07:38] Try running it as www-data [16:07:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.63, 9.19, 8.39 [16:07:58] yeah, it says [16:08:00] reception@mon111:~$ sudo -u www-data /usr/local/bin/runMatomoArchive.sh [16:08:00] /usr/local/bin/runMatomoArchive.sh: line 3: /srv/matomo-archive.log: Permission denied [16:08:08] Ah ok [16:08:15] running as root doesn't say anything but it seems to be working then [16:08:37] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:09:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.45, 9.82, 9.54 [16:09:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.18, 10.40, 9.31 [16:09:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/4938db048a4f [16:09:28] [02miraheze/puppet] 07paladox 034938db0 - matomo: fix permissions for runMatomoArchive.sh [16:09:30] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [16:09:31] [02puppet] 07paladox opened pull request 03#2412: matomo: fix permissions for runMatomoArchive.sh - 13https://github.com/miraheze/puppet/pull/2412 [16:09:32] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:09:36] Reception123: ^ [16:10:01] paladox: thanks! will deploy [16:10:39] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:46] [02puppet] 07Reception123 closed pull request 03#2412: matomo: fix permissions for runMatomoArchive.sh - 13https://github.com/miraheze/puppet/pull/2412 [16:10:47] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/356a8ac9e4c3...b6ccaea02ef9 [16:10:49] [02miraheze/puppet] 07paladox 03b6ccaea - matomo: fix permissions for runMatomoArchive.sh (#2412) [16:10:50] [02puppet] 07Reception123 deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [16:10:52] [02miraheze/puppet] 07Reception123 deleted branch 03paladox-patch-5 [16:11:08] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:16] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:12:05]