[00:00:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.99, 3.77, 3.79 [00:01:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.41, 10.52, 9.89 [00:02:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.00, 3.57, 3.47 [00:02:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.61, 1.93, 1.96 [00:03:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.87, 10.64, 10.01 [00:03:14] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.25, 1.75, 1.68 [00:04:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.61, 4.53, 3.84 [00:04:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 5.69, 2.69, 1.91 [00:04:26] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 10268 MB (26% inode=96%); [00:05:12] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.38, 1.71, 1.68 [00:05:52] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 10135 MB (25% inode=96%); [00:06:19] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.18, 1.96, 1.73 [00:06:39] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.stardewmodding.wiki All nameservers failed to answer the query. [00:07:10] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.48, 1.70, 1.68 [00:08:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.66, 1.48, 1.58 [00:09:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.67, 9.93, 9.92 [00:09:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.93, 3.64, 3.29 [00:10:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.24, 1.94, 1.93 [00:11:33] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.67, 3.91, 3.43 [00:13:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.70, 3.91, 3.49 [00:14:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.85, 3.86 [00:14:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.95, 1.96, 1.94 [00:17:27] [02mw-config] 07Universal-Omega closed pull request 03#4711: add Nigerian Pidgin to wgExtraLanguageNames - 13https://github.com/miraheze/mw-config/pull/4711 [00:17:27] [url] Page not found · GitHub · GitHub | github.com [00:17:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f180f9780935...89159e7433da [00:17:29] [url] Comparing f180f9780935...89159e7433da · miraheze/mw-config · GitHub | github.com [00:17:29] [02miraheze/mw-config] 07ugochimobi 0389159e7 - add Nigerian Pidgin to wgExtraLanguageNames (#4711) [00:17:40] PROBLEM - phab121 NTP time on phab121 is WARNING: NTP WARNING: Offset -0.1032290161 secs [00:18:13] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.61, 2.24, 2.05 [00:18:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.32, 2.29, 1.77 [00:18:39] miraheze/mw-config - Universal-Omega the build passed. [00:19:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.14, 3.26, 3.37 [00:20:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.77, 3.48, 3.95 [00:21:01] !log [@test101] starting deploy of {'config': True} to all [00:21:02] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [00:21:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:22:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.23, 2.76, 3.33 [00:23:11] !log [@mwtask111] starting deploy of {'config': True} to all [00:23:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:19] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 8s [00:23:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:35] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - stardewmodding.wiki All nameservers failed to answer the query. [00:24:19] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.76, 1.97, 1.81 [00:24:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.37, 3.74 [00:26:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.66, 3.82 [00:27:04] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.45, 3.10, 2.35 [00:28:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.52, 3.47 [00:28:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.91, 1.46, 1.64 [00:32:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.15, 3.23, 3.36 [00:32:35] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+1/-0/±0] 13https://github.com/miraheze/ErrorPages/commit/8fc517d6f516 [00:32:37] [02miraheze/ErrorPages] 07Universal-Omega 038fc517d - Add 504 error page [00:32:38] [02ErrorPages] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/ErrorPages [00:32:39] [url] Page not found · GitHub · GitHub | github.com [00:32:40] [02ErrorPages] 07Universal-Omega opened pull request 03#7: Add 504 error page - 13https://github.com/miraheze/ErrorPages/pull/7 [00:32:40] ... [00:33:28] miraheze/ErrorPages - Universal-Omega the build passed. [00:33:34] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/8fc517d6f516...a0db331a85bd [00:33:34] [url] Comparing 8fc517d6f516...a0db331a85bd · miraheze/ErrorPages · GitHub | github.com [00:33:35] [02miraheze/ErrorPages] 07Universal-Omega 03a0db331 - Update 504.php [00:33:37] [02ErrorPages] 07Universal-Omega synchronize pull request 03#7: Add 504 error page - 13https://github.com/miraheze/ErrorPages/pull/7 [00:33:37] [url] Page not found · GitHub · GitHub | github.com [00:34:28] miraheze/ErrorPages - Universal-Omega the build passed. [00:35:42] RECOVERY - phab121 NTP time on phab121 is OK: NTP OK: Offset -0.09772092104 secs [00:36:40] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.re [00:36:40] stname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407235622406006 seconds [00:38:19] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/a0db331a85bd...dc6976c36304 [00:38:19] [url] Comparing a0db331a85bd...dc6976c36304 · miraheze/ErrorPages · GitHub | github.com [00:38:20] [02miraheze/ErrorPages] 07Universal-Omega 03dc6976c - Update en.json [00:38:22] [02ErrorPages] 07Universal-Omega synchronize pull request 03#7: Add 504 error page - 13https://github.com/miraheze/ErrorPages/pull/7 [00:38:22] [url] Page not found · GitHub · GitHub | github.com [00:38:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 3.59, 3.61 [00:39:11] miraheze/ErrorPages - Universal-Omega the build passed. [00:39:23] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/dc6976c36304...dc3fd6f152f7 [00:39:23] [url] Comparing dc6976c36304...dc3fd6f152f7 · miraheze/ErrorPages · GitHub | github.com [00:39:24] [02miraheze/ErrorPages] 07Universal-Omega 03dc3fd6f - Update qqq.json [00:39:26] [02ErrorPages] 07Universal-Omega synchronize pull request 03#7: Add 504 error page - 13https://github.com/miraheze/ErrorPages/pull/7 [00:39:26] [url] Page not found · GitHub · GitHub | github.com [00:40:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.58, 3.40, 3.28 [00:40:40] miraheze/ErrorPages - Universal-Omega the build passed. [00:40:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.73, 3.57, 3.27 [00:41:04] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.34, 1.78, 1.99 [00:42:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.69, 3.37, 3.28 [00:44:06] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.46, 1.82, 1.98 [00:44:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.63, 3.39 [00:46:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.93, 3.01, 3.14 [00:48:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.94, 3.91 [00:49:04] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.49, 1.33, 1.68 [00:50:04] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.33, 2.00, 1.98 [00:50:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.31, 3.73, 3.57 [00:52:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.78, 3.08, 3.35 [00:56:04] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.45, 1.84, 1.95 [00:56:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.90, 3.45, 3.56 [00:58:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.79, 3.30, 3.50 [01:02:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.25, 1.39, 1.70 [01:02:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 3.54, 3.53 [01:04:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.27, 3.07, 3.36 [01:15:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.92, 3.76, 3.43 [01:17:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.73, 3.30, 3.30 [01:19:56] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.24, 1.84, 1.67 [01:22:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.80, 3.75, 3.48 [01:23:36] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolv [01:23:36] me, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406234502792358 seconds [01:23:55] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.70, 1.88, 1.74 [01:24:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.62, 3.37, 3.37 [01:27:54] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.45, 1.65, 1.68 [01:33:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.33, 8.79, 7.61 [01:33:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.64, 3.91, 3.54 [01:33:52] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.81, 1.73 [01:35:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.74, 8.72, 7.74 [01:36:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.59, 4.09, 3.29 [01:36:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.84, 3.30 [01:36:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.02, 1.61, 1.28 [01:37:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.99, 10.56, 9.38 [01:37:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.85, 3.63 [01:37:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.99, 3.35 [01:38:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.72, 3.39, 3.19 [01:38:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.34, 1.48, 1.27 [01:39:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.59, 2.13, 1.87 [01:39:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.69, 3.93, 3.38 [01:41:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.02, 10.06, 9.48 [01:41:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.29, 3.29, 3.21 [01:44:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.61, 3.95, 3.70 [01:45:48] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.66, 1.90, 1.87 [01:46:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.67, 3.62 [01:47:48] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.94, 1.89 [01:50:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 4.01, 3.76 [01:52:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.55, 3.61 [01:53:35] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - stardewmodding.wiki All nameservers failed to answer the query. [01:54:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.52, 5.60, 4.36 [01:54:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.98, 4.30, 3.53 [01:55:18] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 4.35, 3.69 [02:05:43] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.93, 2.00, 2.00 [02:09:42] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.37, 2.02, 2.00 [02:13:41] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.87, 1.95 [02:17:40] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.67, 2.12, 2.01 [02:23:05] PROBLEM - wikien.wildterra2.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikien.wildterra2.com' expires in 15 day(s) (Sat 11 Jun 2022 02:10:08 GMT +0000). [02:23:36] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolv [02:23:36] me, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406584739685059 seconds [02:24:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [02:26:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.77, 1.94, 1.51 [02:28:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.72, 1.42, 1.37 [02:29:40] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.67, 1.78, 1.94 [02:30:00] PROBLEM - wiki.warframestat.us - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.warframestat.us' expires in 15 day(s) (Sat 11 Jun 2022 02:27:27 GMT +0000). [02:30:24] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/13d025975bcc...633dc82b7dee [02:30:25] [url] Comparing 13d025975bcc...633dc82b7dee · miraheze/ssl · GitHub | github.com [02:30:26] [02miraheze/ssl] 07MirahezeSSLBot 03633dc82 - Bot: Update SSL cert for wikien.wildterra2.com [02:34:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.85, 1.99, 1.64 [02:34:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [02:35:31] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.43, 1.71, 1.37 [02:35:38] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.17, 1.35, 1.69 [02:37:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/633dc82b7dee...ee744e2db378 [02:37:24] [url] Comparing 633dc82b7dee...ee744e2db378 · miraheze/ssl · GitHub | github.com [02:37:25] [02miraheze/ssl] 07MirahezeSSLBot 03ee744e2 - Bot: Update SSL cert for wiki.warframestat.us [02:37:30] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.90, 1.36, 1.28 [02:38:17] PROBLEM - www.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Sat 11 Jun 2022 02:21:01 GMT +0000). [02:38:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.02, 1.45, 1.49 [02:38:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ee744e2db378...8634faec737f [02:38:54] [url] Comparing ee744e2db378...8634faec737f · miraheze/ssl · GitHub | github.com [02:38:55] [02miraheze/ssl] 07MirahezeSSLBot 038634fae - Bot: Update SSL cert for www.thesimswiki.com [02:44:36] PROBLEM - wiki.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Sat 11 Jun 2022 02:21:01 GMT +0000). [02:47:04] PROBLEM - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'marinebiodiversitymatrix.org' expires in 15 day(s) (Sat 11 Jun 2022 02:30:15 GMT +0000). [02:49:33] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.92, 1.66, 1.61 [02:51:33] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.27, 1.88, 1.70 [02:53:28] PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'marinebiodiversitymatrix.org' expires in 15 day(s) (Sat 11 Jun 2022 02:30:15 GMT +0000). [02:53:32] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.36, 1.65, 1.63 [02:53:33] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - stardewmodding.wiki All nameservers failed to answer the query. [02:55:04] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zymonic.com' expires in 15 day(s) (Sat 11 Jun 2022 02:37:25 GMT +0000). [02:59:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8634faec737f...a4eff34ddba2 [02:59:24] [url] Comparing 8634faec737f...a4eff34ddba2 · miraheze/ssl · GitHub | github.com [02:59:25] [02miraheze/ssl] 07MirahezeSSLBot 03a4eff34 - Bot: Update SSL cert for marinebiodiversitymatrix.org [02:59:29] RECOVERY - wiki.warframestat.us - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.warframestat.us' will expire on Wed 24 Aug 2022 01:37:17 GMT +0000. [03:00:34] PROBLEM - wiki.seredrau.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.seredrau.xyz' expires in 15 day(s) (Sat 11 Jun 2022 02:51:57 GMT +0000). [03:03:58] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.nowchess.org' expires in 15 day(s) (Sat 11 Jun 2022 02:42:56 GMT +0000). [03:07:04] RECOVERY - www.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Wed 24 Aug 2022 01:38:48 GMT +0000. [03:07:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a4eff34ddba2...cede36158b75 [03:07:26] [url] Comparing a4eff34ddba2...cede36158b75 · miraheze/ssl · GitHub | github.com [03:07:27] [02miraheze/ssl] 07MirahezeSSLBot 03cede361 - Bot: Update SSL cert for wiki.zymonic.com [03:08:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.20, 4.00 [03:08:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 2.99, 3.91 [03:10:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.72, 4.11 [03:12:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.99, 3.39, 3.93 [03:12:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cede36158b75...507cb3aad50e [03:12:41] [url] Comparing cede36158b75...507cb3aad50e · miraheze/ssl · GitHub | github.com [03:12:42] [02miraheze/ssl] 07MirahezeSSLBot 03507cb3a - Bot: Update SSL cert for wiki.nowchess.org [03:13:42] RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Wed 24 Aug 2022 01:38:48 GMT +0000. [03:17:28] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/507cb3aad50e...cf8653dcb321 [03:17:29] [url] Comparing 507cb3aad50e...cf8653dcb321 · miraheze/ssl · GitHub | github.com [03:17:30] [02miraheze/ssl] 07MirahezeSSLBot 03cf8653d - Bot: Update SSL cert for wiki.seredrau.xyz [03:18:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.39, 1.96, 1.74 [03:18:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.79, 3.74, 3.79 [03:20:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.06, 3.31, 3.62 [03:21:04] RECOVERY - wikien.wildterra2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikien.wildterra2.com' will expire on Wed 24 Aug 2022 01:30:15 GMT +0000. [03:22:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 3.72, 3.73 [03:26:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.04, 4.17, 3.95 [03:28:05] PROBLEM - wiki.mobilityengineer.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mobilityengineer.com' expires in 15 day(s) (Sat 11 Jun 2022 03:06:20 GMT +0000). [03:28:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.41, 1.89, 1.90 [03:28:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.71, 3.84, 3.79 [03:28:52] PROBLEM - wiki.minecraftathome.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.minecraftathome.com' expires in 15 day(s) (Sat 11 Jun 2022 03:20:24 GMT +0000). [03:28:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cf8653dcb321...40f75b5df5a2 [03:28:59] [url] Comparing cf8653dcb321...40f75b5df5a2 · miraheze/ssl · GitHub | github.com [03:29:00] [02miraheze/ssl] 07MirahezeSSLBot 0340f75b5 - Bot: Update SSL cert for wiki.mobilityengineer.com [03:29:07] RECOVERY - wiki.seredrau.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.seredrau.xyz' will expire on Wed 24 Aug 2022 02:17:23 GMT +0000. [03:32:05] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/40f75b5df5a2...a262ec3547ce [03:32:06] [url] Comparing 40f75b5df5a2...a262ec3547ce · miraheze/ssl · GitHub | github.com [03:32:07] [02miraheze/ssl] 07MirahezeSSLBot 03a262ec3 - Bot: Update SSL cert for wiki.minecraftathome.com [03:32:56] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Wed 24 Aug 2022 02:12:36 GMT +0000. [03:34:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.69, 4.65, 4.07 [03:36:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.91, 1.88 [03:38:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.98, 1.93, 1.90 [03:39:03] PROBLEM - wiki.starship.digital - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.starship.digital' expires in 15 day(s) (Sat 11 Jun 2022 03:19:25 GMT +0000). [03:42:53] PROBLEM - wikiru.wildterra2.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikiru.wildterra2.com' expires in 15 day(s) (Sat 11 Jun 2022 03:12:09 GMT +0000). [03:43:36] PROBLEM - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.rebirthofthenight.com' expires in 15 day(s) (Sat 11 Jun 2022 03:28:59 GMT +0000). [03:44:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a262ec3547ce...9fd7b9da9ab7 [03:44:10] [url] Comparing a262ec3547ce...9fd7b9da9ab7 · miraheze/ssl · GitHub | github.com [03:44:11] [02miraheze/ssl] 07MirahezeSSLBot 039fd7b9d - Bot: Update SSL cert for wiki.starship.digital [03:44:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.26, 2.06, 1.96 [03:45:04] RECOVERY - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Wed 24 Aug 2022 01:59:18 GMT +0000. [03:47:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9fd7b9da9ab7...34ebe4cbc5f8 [03:47:14] [url] Comparing 9fd7b9da9ab7...34ebe4cbc5f8 · miraheze/ssl · GitHub | github.com [03:47:15] [02miraheze/ssl] 07MirahezeSSLBot 0334ebe4c - Bot: Update SSL cert for wikiru.wildterra2.com [03:48:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.31, 3.84, 4.00 [03:49:27] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/34ebe4cbc5f8...768126d926a8 [03:49:27] [url] Comparing 34ebe4cbc5f8...768126d926a8 · miraheze/ssl · GitHub | github.com [03:49:28] [02miraheze/ssl] 07MirahezeSSLBot 03768126d - Bot: Update SSL cert for wiki.rebirthofthenight.com [03:50:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.50, 3.37, 3.85 [03:50:24] PROBLEM - wiki.whentheycry.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.whentheycry.org' expires in 15 day(s) (Sat 11 Jun 2022 03:34:31 GMT +0000). [03:51:28] RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Wed 24 Aug 2022 01:59:18 GMT +0000. [03:52:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.25, 3.84, 3.93 [03:53:12] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Wed 24 Aug 2022 02:07:19 GMT +0000. [03:54:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.67, 3.31, 3.72 [03:56:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.11, 2.58, 3.36 [03:57:10] RECOVERY - wiki.mobilityengineer.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mobilityengineer.com' will expire on Wed 24 Aug 2022 02:28:53 GMT +0000. [03:57:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/768126d926a8...166f09fe43b5 [03:57:43] [url] Comparing 768126d926a8...166f09fe43b5 · miraheze/ssl · GitHub | github.com [03:57:44] [02miraheze/ssl] 07MirahezeSSLBot 03166f09f - Bot: Update SSL cert for wiki.whentheycry.org [03:57:57] PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.villagecollaborative.net' expires in 15 day(s) (Sat 11 Jun 2022 03:28:10 GMT +0000). [03:58:44] RECOVERY - wiki.minecraftathome.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minecraftathome.com' will expire on Wed 24 Aug 2022 02:32:00 GMT +0000. [03:59:12] PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thelonsdalebattalion.co.uk' expires in 15 day(s) (Sat 11 Jun 2022 03:45:11 GMT +0000). [03:59:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/166f09fe43b5...0f044151b0e6 [03:59:40] [url] Comparing 166f09fe43b5...0f044151b0e6 · miraheze/ssl · GitHub | github.com [03:59:41] [02miraheze/ssl] 07MirahezeSSLBot 030f04415 - Bot: Update SSL cert for wiki.villagecollaborative.net [04:00:10] PROBLEM - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mxlinuxusers.de' expires in 15 day(s) (Sat 11 Jun 2022 03:56:48 GMT +0000). [04:00:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.35, 3.12, 3.84 [04:01:06] PROBLEM - wiki.mcpirevival.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mcpirevival.tk' expires in 15 day(s) (Sat 11 Jun 2022 03:37:34 GMT +0000). [04:04:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.54, 3.91, 3.95 [04:04:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.54, 3.99, 3.75 [04:05:47] PROBLEM - wiki.s23.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.s23.org' expires in 15 day(s) (Sat 11 Jun 2022 03:37:20 GMT +0000). [04:06:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0f044151b0e6...fe5e91a9df97 [04:06:24] [url] Comparing 0f044151b0e6...fe5e91a9df97 · miraheze/ssl · GitHub | github.com [04:06:25] [02miraheze/ssl] 07MirahezeSSLBot 03fe5e91a - Bot: Update SSL cert for thelonsdalebattalion.co.uk [04:06:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.71, 3.76, 3.69 [04:08:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fe5e91a9df97...ec24fc1c4c8a [04:08:01] [url] Comparing fe5e91a9df97...ec24fc1c4c8a · miraheze/ssl · GitHub | github.com [04:08:02] [02miraheze/ssl] 07MirahezeSSLBot 03ec24fc1 - Bot: Update SSL cert for wiki.mcpirevival.tk [04:08:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.88, 1.99 [04:08:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ec24fc1c4c8a...adc46e8bd786 [04:08:25] [url] Comparing ec24fc1c4c8a...adc46e8bd786 · miraheze/ssl · GitHub | github.com [04:08:26] [02miraheze/ssl] 07MirahezeSSLBot 03adc46e8 - Bot: Update SSL cert for wiki.s23.org [04:08:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.71, 3.85 [04:08:35] RECOVERY - wiki.starship.digital - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.starship.digital' will expire on Wed 24 Aug 2022 02:44:01 GMT +0000. [04:11:48] RECOVERY - wikiru.wildterra2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiru.wildterra2.com' will expire on Wed 24 Aug 2022 02:47:08 GMT +0000. [04:12:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 4.19, 4.00 [04:14:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.73, 3.51 [04:14:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/adc46e8bd786...f507320a7428 [04:14:37] [url] Comparing adc46e8bd786...f507320a7428 · miraheze/ssl · GitHub | github.com [04:14:38] [02miraheze/ssl] 07MirahezeSSLBot 03f507320 - Bot: Update SSL cert for wiki.mxlinuxusers.de [04:15:10] PROBLEM - wiki.anglish.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.anglish.info' expires in 15 day(s) (Sat 11 Jun 2022 04:03:46 GMT +0000). [04:18:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.07, 3.04, 3.29 [04:18:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f507320a7428...e483dfb8940a [04:18:42] [url] Comparing f507320a7428...e483dfb8940a · miraheze/ssl · GitHub | github.com [04:18:42] [02miraheze/ssl] 07MirahezeSSLBot 03e483dfb - Bot: Update SSL cert for wiki.anglish.info [04:18:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.85, 2.86, 3.30 [04:18:45] PROBLEM - icclopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'icclopedia.org' expires in 15 day(s) (Sat 11 Jun 2022 03:52:40 GMT +0000). [04:20:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.25, 1.96, 1.92 [04:22:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.98, 1.94 [04:22:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.53, 3.84 [04:22:33] PROBLEM - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lefrenchmelee.fr' expires in 15 day(s) (Sat 11 Jun 2022 04:05:18 GMT +0000). [04:22:51] PROBLEM - history.estill.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'history.estill.org' expires in 15 day(s) (Sat 11 Jun 2022 04:20:02 GMT +0000). [04:23:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e483dfb8940a...60e11a1d6bf7 [04:23:11] [url] Comparing e483dfb8940a...60e11a1d6bf7 · miraheze/ssl · GitHub | github.com [04:23:12] [02miraheze/ssl] 07MirahezeSSLBot 0360e11a1 - Bot: Update SSL cert for icclopedia.org [04:26:54] RECOVERY - wiki.villagecollaborative.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Wed 24 Aug 2022 02:59:35 GMT +0000. [04:27:54] RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Wed 24 Aug 2022 03:06:17 GMT +0000. [04:28:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.14, 1.39, 1.68 [04:29:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/60e11a1d6bf7...8399f9fb6a10 [04:29:45] [url] Comparing 60e11a1d6bf7...8399f9fb6a10 · miraheze/ssl · GitHub | github.com [04:29:46] [02miraheze/ssl] 07MirahezeSSLBot 038399f9f - Bot: Update SSL cert for history.estill.org [04:29:50] RECOVERY - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.de' will expire on Wed 24 Aug 2022 03:14:31 GMT +0000. [04:30:12] RECOVERY - wiki.mcpirevival.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mcpirevival.tk' will expire on Wed 24 Aug 2022 03:07:55 GMT +0000. [04:34:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8399f9fb6a10...37d3690d224a [04:34:33] [url] Comparing 8399f9fb6a10...37d3690d224a · miraheze/ssl · GitHub | github.com [04:34:33] [02miraheze/ssl] 07MirahezeSSLBot 0337d3690 - Bot: Update SSL cert for wiki.lefrenchmelee.fr [04:35:04] RECOVERY - wiki.s23.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.s23.org' will expire on Wed 24 Aug 2022 03:08:19 GMT +0000. [04:40:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.00, 2.93, 3.33 [04:42:47] RECOVERY - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rebirthofthenight.com' will expire on Wed 24 Aug 2022 02:49:21 GMT +0000. [04:44:06] PROBLEM - vmcodex.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vmcodex.net' expires in 15 day(s) (Sat 11 Jun 2022 04:15:58 GMT +0000). [04:44:50] RECOVERY - wiki.anglish.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.anglish.info' will expire on Wed 24 Aug 2022 03:18:35 GMT +0000. [04:45:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/37d3690d224a...b06ced124c14 [04:45:47] [url] Comparing 37d3690d224a...b06ced124c14 · miraheze/ssl · GitHub | github.com [04:45:48] [02miraheze/ssl] 07MirahezeSSLBot 03b06ced1 - Bot: Update SSL cert for vmcodex.net [04:46:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.34, 3.36 [04:48:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.54, 1.91, 1.66 [04:48:24] RECOVERY - wiki.whentheycry.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.whentheycry.org' will expire on Wed 24 Aug 2022 02:57:37 GMT +0000. [04:48:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.84, 4.29, 3.70 [04:50:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.22, 2.93 [04:52:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.97, 1.77 [04:52:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.77, 3.98, 3.70 [04:52:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.03, 3.06, 2.90 [04:53:31] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for stardewmodding.wiki could not be found [04:53:58] PROBLEM - wiki.kourouklides.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kourouklides.com' expires in 15 day(s) (Sat 11 Jun 2022 04:34:35 GMT +0000). [04:54:15] PROBLEM - kunwok.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kunwok.org' expires in 15 day(s) (Sat 11 Jun 2022 04:28:19 GMT +0000). [04:55:36] PROBLEM - spiral.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'spiral.wiki' expires in 15 day(s) (Sat 11 Jun 2022 04:24:22 GMT +0000). [04:56:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.08, 3.93, 3.69 [04:56:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b06ced124c14...e070d69c7a46 [04:56:51] [url] Comparing b06ced124c14...e070d69c7a46 · miraheze/ssl · GitHub | github.com [04:56:52] [02miraheze/ssl] 07MirahezeSSLBot 03e070d69 - Bot: Update SSL cert for wiki.kourouklides.com [04:58:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.32, 1.59, 1.66 [04:58:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.61, 3.60 [04:58:36] PROBLEM - wiki.climatechange.ai - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.climatechange.ai' expires in 15 day(s) (Sat 11 Jun 2022 04:34:18 GMT +0000). [04:59:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e070d69c7a46...e66c35b0754d [04:59:24] [url] Comparing e070d69c7a46...e66c35b0754d · miraheze/ssl · GitHub | github.com [04:59:25] [02miraheze/ssl] 07MirahezeSSLBot 03e66c35b - Bot: Update SSL cert for spiral.wiki [05:00:05] PROBLEM - spcodex.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'spcodex.wiki' expires in 15 day(s) (Sat 11 Jun 2022 04:55:11 GMT +0000). [05:02:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.05, 4.01, 3.73 [05:03:35] PROBLEM - bebaskanpengetahuan.id - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bebaskanpengetahuan.id' expires in 15 day(s) (Sat 11 Jun 2022 04:40:03 GMT +0000). [05:04:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e66c35b0754d...7eb1271e67ff [05:04:04] [url] Comparing e66c35b0754d...7eb1271e67ff · miraheze/ssl · GitHub | github.com [05:04:05] [02miraheze/ssl] 07MirahezeSSLBot 037eb1271 - Bot: Update SSL cert for wiki.climatechange.ai [05:04:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.76, 3.67 [05:07:21] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.39, 1.99, 1.79 [05:08:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.07, 4.11, 3.80 [05:10:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.38, 3.56 [05:11:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7eb1271e67ff...b2665d71a5f4 [05:11:06] [url] Comparing 7eb1271e67ff...b2665d71a5f4 · miraheze/ssl · GitHub | github.com [05:11:08] [02miraheze/ssl] 07MirahezeSSLBot 03b2665d7 - Bot: Update SSL cert for bebaskanpengetahuan.id [05:11:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b2665d71a5f4...a7c494b358a8 [05:11:44] [url] Comparing b2665d71a5f4...a7c494b358a8 · miraheze/ssl · GitHub | github.com [05:11:45] [02miraheze/ssl] 07MirahezeSSLBot 03a7c494b - Bot: Update SSL cert for kunwok.org [05:12:18] PROBLEM - miraheze.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.ga' expires in 15 day(s) (Sat 11 Jun 2022 05:08:40 GMT +0000). [05:12:41] RECOVERY - vmcodex.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vmcodex.net' will expire on Wed 24 Aug 2022 03:45:41 GMT +0000. [05:15:19] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.99, 1.89 [05:16:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.58, 3.53 [05:16:45] RECOVERY - icclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'icclopedia.org' will expire on Wed 24 Aug 2022 03:23:05 GMT +0000. [05:17:12] PROBLEM - vise.dayid.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vise.dayid.org' expires in 15 day(s) (Sat 11 Jun 2022 04:48:07 GMT +0000). [05:17:19] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.23, 2.01, 1.91 [05:17:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a7c494b358a8...5147d0ce9604 [05:17:48] [url] Comparing a7c494b358a8...5147d0ce9604 · miraheze/ssl · GitHub | github.com [05:17:49] [02miraheze/ssl] 07MirahezeSSLBot 035147d0c - Bot: Update SSL cert for spcodex.wiki [05:18:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.97, 3.61, 3.55 [05:21:40] RECOVERY - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lefrenchmelee.fr' will expire on Wed 24 Aug 2022 03:34:26 GMT +0000. [05:22:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.09, 4.08, 3.80 [05:22:34] RECOVERY - history.estill.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.estill.org' will expire on Wed 24 Aug 2022 03:29:39 GMT +0000. [05:24:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.04, 3.86, 3.76 [05:25:08] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5147d0ce9604...c554381497a9 [05:25:08] [url] Comparing 5147d0ce9604...c554381497a9 · miraheze/ssl · GitHub | github.com [05:25:09] [02miraheze/ssl] 07MirahezeSSLBot 03c554381 - Bot: Update SSL cert for miraheze.ga [05:27:55] PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonciclopedia.org' expires in 15 day(s) (Sat 11 Jun 2022 05:24:12 GMT +0000). [05:28:31] RECOVERY - wiki.climatechange.ai - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.climatechange.ai' will expire on Wed 24 Aug 2022 04:03:58 GMT +0000. [05:28:45] PROBLEM - runzeppelin.ru - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'runzeppelin.ru' expires in 15 day(s) (Sat 11 Jun 2022 05:01:40 GMT +0000). [05:28:56] PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vedopedia.witches-empire.com' expires in 15 day(s) (Sat 11 Jun 2022 05:15:21 GMT +0000). [05:29:40] RECOVERY - spcodex.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spcodex.wiki' will expire on Wed 24 Aug 2022 04:17:42 GMT +0000. [05:30:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c554381497a9...95d864bfc4aa [05:30:52] [url] Comparing c554381497a9...95d864bfc4aa · miraheze/ssl · GitHub | github.com [05:30:52] [02miraheze/ssl] 07MirahezeSSLBot 0395d864b - Bot: Update SSL cert for vise.dayid.org [05:33:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.82, 1.96, 1.99 [05:33:34] RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Wed 24 Aug 2022 04:11:00 GMT +0000. [05:35:14] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 1.98, 2.02, 2.01 [05:36:11] PROBLEM - savagepedia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'savagepedia.wiki' expires in 15 day(s) (Sat 11 Jun 2022 05:23:12 GMT +0000). [05:36:11] PROBLEM - wiki.triplescripts.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.triplescripts.org' expires in 15 day(s) (Sat 11 Jun 2022 05:07:51 GMT +0000). [05:38:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.11, 2.89, 3.33 [05:38:36] PROBLEM - wiki.nevillepedia.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.nevillepedia.eu' expires in 15 day(s) (Sat 11 Jun 2022 05:29:19 GMT +0000). [05:38:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/95d864bfc4aa...afa6abf62a92 [05:38:53] [url] Comparing 95d864bfc4aa...afa6abf62a92 · miraheze/ssl · GitHub | github.com [05:38:54] [02miraheze/ssl] 07MirahezeSSLBot 03afa6abf - Bot: Update SSL cert for nonciclopedia.org [05:41:12] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.93, 1.99, 1.99 [05:41:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/afa6abf62a92...9431c68c6841 [05:41:26] [url] Comparing afa6abf62a92...9431c68c6841 · miraheze/ssl · GitHub | github.com [05:41:27] [02miraheze/ssl] 07MirahezeSSLBot 039431c68 - Bot: Update SSL cert for wiki.nevillepedia.eu [05:42:17] !log reception@mwtask111:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=45yearwarwiki --logwiki=metawiki D_tsagaris Vardan97 [05:42:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:44:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9431c68c6841...cc18f04bbb5c [05:44:43] [url] Comparing 9431c68c6841...cc18f04bbb5c · miraheze/ssl · GitHub | github.com [05:44:44] [02miraheze/ssl] 07MirahezeSSLBot 03cc18f04 - Bot: Update SSL cert for savagepedia.wiki [05:46:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.78, 3.58 [05:49:10] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.50, 2.46, 2.16 [05:49:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cc18f04bbb5c...739204b25959 [05:49:11] [url] Comparing cc18f04bbb5c...739204b25959 · miraheze/ssl · GitHub | github.com [05:49:12] [02miraheze/ssl] 07MirahezeSSLBot 03739204b - Bot: Update SSL cert for wiki.triplescripts.org [05:50:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 3.83, 3.62 [05:52:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.43, 3.50 [05:52:53] RECOVERY - wiki.kourouklides.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kourouklides.com' will expire on Wed 24 Aug 2022 03:56:44 GMT +0000. [05:53:46] RECOVERY - kunwok.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kunwok.org' will expire on Wed 24 Aug 2022 04:11:39 GMT +0000. [05:54:50] RECOVERY - spiral.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spiral.wiki' will expire on Wed 24 Aug 2022 03:59:18 GMT +0000. [05:55:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/739204b25959...cbbb3be601e7 [05:55:14] [url] Comparing 739204b25959...cbbb3be601e7 · miraheze/ssl · GitHub | github.com [05:55:15] [02miraheze/ssl] 07MirahezeSSLBot 03cbbb3be - Bot: Update SSL cert for vedopedia.witches-empire.com [05:56:50] RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Wed 24 Aug 2022 04:38:47 GMT +0000. [05:58:11] PROBLEM - dc-multiverse.dcwikis.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dc-multiverse.dcwikis.com' expires in 15 day(s) (Sat 11 Jun 2022 05:37:04 GMT +0000). [05:58:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 10.20, 5.86, 4.40 [05:58:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 9.15, 5.82, 4.02 [05:59:07] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.59, 1.76, 1.96 [06:00:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.53, 4.54, 3.47 [06:01:01] PROBLEM - wiki.valkyrienskies.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.valkyrienskies.org' expires in 15 day(s) (Sat 11 Jun 2022 05:52:49 GMT +0000). [06:02:04] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [06:02:04] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:02:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cbbb3be601e7...db63195f67c4 [06:02:51] [url] Comparing cbbb3be601e7...db63195f67c4 · miraheze/ssl · GitHub | github.com [06:02:52] [02miraheze/ssl] 07MirahezeSSLBot 03db63195 - Bot: Update SSL cert for runzeppelin.ru [06:05:51] RECOVERY - savagepedia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'savagepedia.wiki' will expire on Wed 24 Aug 2022 04:44:37 GMT +0000. [06:07:03] PROBLEM - wiki.cyberfurs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cyberfurs.org' expires in 15 day(s) (Sat 11 Jun 2022 05:41:11 GMT +0000). [06:07:05] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.74, 1.94, 1.89 [06:07:42] RECOVERY - wiki.nevillepedia.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nevillepedia.eu' will expire on Wed 24 Aug 2022 04:41:20 GMT +0000. [06:08:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.58, 3.77, 3.60 [06:08:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/db63195f67c4...4bf1b8ec1eca [06:08:59] [url] Comparing db63195f67c4...4bf1b8ec1eca · miraheze/ssl · GitHub | github.com [06:09:00] [02miraheze/ssl] 07MirahezeSSLBot 034bf1b8e - Bot: Update SSL cert for dc-multiverse.dcwikis.com [06:10:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 4.02, 3.72 [06:10:13] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4bf1b8ec1eca...b37b79368634 [06:10:13] [url] Comparing 4bf1b8ec1eca...b37b79368634 · miraheze/ssl · GitHub | github.com [06:10:14] [02miraheze/ssl] 07MirahezeSSLBot 03b37b793 - Bot: Update SSL cert for wiki.cyberfurs.org [06:11:53] RECOVERY - miraheze.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ga' will expire on Wed 24 Aug 2022 04:25:02 GMT +0000. [06:12:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.69, 3.60, 3.61 [06:13:10] PROBLEM - wikipariksha.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikipariksha.com' expires in 15 day(s) (Sat 11 Jun 2022 05:46:34 GMT +0000). [06:14:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b37b79368634...3526a1b3545a [06:14:46] [url] Comparing b37b79368634...3526a1b3545a · miraheze/ssl · GitHub | github.com [06:14:47] [02miraheze/ssl] 07MirahezeSSLBot 033526a1b - Bot: Update SSL cert for wiki.valkyrienskies.org [06:15:12] RECOVERY - vise.dayid.org - LetsEncrypt on sslhost is OK: OK - Certificate 'vise.dayid.org' will expire on Wed 24 Aug 2022 04:30:45 GMT +0000. [06:18:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 3.89, 3.71 [06:18:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3526a1b3545a...a3f121baa957 [06:18:55] [url] Comparing 3526a1b3545a...a3f121baa957 · miraheze/ssl · GitHub | github.com [06:18:56] [02miraheze/ssl] 07MirahezeSSLBot 03a3f121b - Bot: Update SSL cert for wikipariksha.com [06:20:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.94, 3.49, 3.58 [06:26:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.84, 3.96, 3.67 [06:26:24] PROBLEM - wiki.minkyu.kim - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.minkyu.kim' expires in 15 day(s) (Sat 11 Jun 2022 05:57:18 GMT +0000). [06:28:40] RECOVERY - runzeppelin.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'runzeppelin.ru' will expire on Wed 24 Aug 2022 05:02:45 GMT +0000. [06:28:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.17, 1.73, 1.21 [06:28:48] RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Wed 24 Aug 2022 04:55:08 GMT +0000. [06:30:02] RECOVERY - wiki.valkyrienskies.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.valkyrienskies.org' will expire on Wed 24 Aug 2022 05:14:40 GMT +0000. [06:30:38] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [06:30:39] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.34, 1.57, 1.20 [06:33:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a3f121baa957...86000724fa3b [06:33:52] [url] Comparing a3f121baa957...86000724fa3b · miraheze/ssl · GitHub | github.com [06:33:53] [02miraheze/ssl] 07MirahezeSSLBot 038600072 - Bot: Update SSL cert for wiki.minkyu.kim [06:35:32] RECOVERY - wiki.triplescripts.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.triplescripts.org' will expire on Wed 24 Aug 2022 04:49:04 GMT +0000. [06:37:02] RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cyberfurs.org' will expire on Wed 24 Aug 2022 05:10:07 GMT +0000. [06:38:41] PROBLEM - wikicryptos.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikicryptos.org' expires in 15 day(s) (Sat 11 Jun 2022 06:35:29 GMT +0000). [06:42:20] RECOVERY - wikipariksha.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipariksha.com' will expire on Wed 24 Aug 2022 05:18:49 GMT +0000. [06:45:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/86000724fa3b...46cf0db8fe8e [06:45:22] [url] Comparing 86000724fa3b...46cf0db8fe8e · miraheze/ssl · GitHub | github.com [06:45:23] [02miraheze/ssl] 07MirahezeSSLBot 0346cf0db - Bot: Update SSL cert for wikicryptos.org [06:45:54] It's nice to have MirahezeSSLBot back. I bet Reception123 doesn't miss having to manually do SSL cert renewals :p [06:51:15] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.92, 1.82, 1.59 [06:53:12] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.59, 1.64, 1.54 [06:56:33] RECOVERY - dc-multiverse.dcwikis.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dc-multiverse.dcwikis.com' will expire on Wed 24 Aug 2022 05:08:53 GMT +0000. [06:57:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.73, 1.78, 1.63 [06:59:01] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.32, 1.69, 1.61 [07:06:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.52, 3.49, 3.91 [07:07:20] dmehus: heh, that was months ago but yeah it was awful! [07:07:27] soon I hope to find time to get rid of the private key step as well [07:07:51] RECOVERY - wikicryptos.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikicryptos.org' will expire on Wed 24 Aug 2022 05:45:16 GMT +0000. [07:09:07] PROBLEM - kiseki.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kiseki.wiki' expires in 15 day(s) (Sat 11 Jun 2022 06:47:56 GMT +0000). [07:10:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.13, 3.88 [07:11:34] Reception123, yeah [07:12:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.34, 4.03, 4.13 [07:14:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.99, 3.59, 3.95 [07:15:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/46cf0db8fe8e...70143fbbfe78 [07:15:46] [url] Comparing 46cf0db8fe8e...70143fbbfe78 · miraheze/ssl · GitHub | github.com [07:15:47] [02miraheze/ssl] 07MirahezeSSLBot 0370143fb - Bot: Update SSL cert for kiseki.wiki [07:18:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.75, 2.92, 3.40 [07:20:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.81, 2.37, 3.28 [07:22:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.50, 2.99, 3.93 [07:24:24] RECOVERY - wiki.minkyu.kim - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Wed 24 Aug 2022 05:33:46 GMT +0000. [07:24:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.80, 3.58, 4.02 [07:26:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 3.48, 3.95 [07:26:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.25, 3.39 [07:27:24] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 2.00, 1.82, 1.54 [07:28:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.49, 3.65, 3.51 [07:29:21] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.39, 1.60, 1.49 [07:31:29] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.66, 2.05, 1.26 [07:32:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.93, 3.53, 3.52 [07:33:29] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.80, 1.52, 1.16 [07:35:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.46, 3.42, 3.23 [07:37:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.21, 3.00, 3.10 [07:38:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.95, 3.64, 3.73 [07:38:42] RECOVERY - kiseki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kiseki.wiki' will expire on Wed 24 Aug 2022 06:15:40 GMT +0000. [07:38:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.12, 3.24, 3.40 [07:42:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.47, 3.64 [07:44:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.94, 3.80 [07:46:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.53, 3.65 [07:48:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 3.67, 3.68 [07:50:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.18, 3.38, 3.57 [07:51:45] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.72, 1.41, 1.36 [07:53:43] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.65, 1.52, 1.41 [07:57:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.28, 3.76, 3.25 [07:57:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 8.99, 3.42, 2.07 [07:58:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.81, 3.97, 3.69 [08:00:01] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.45, 2.11, 1.64 [08:00:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.05, 5.41, 3.87 [08:01:56] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.41, 1.73, 1.55 [08:05:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.66, 1.25, 1.40 [08:09:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.11, 3.99, 3.95 [08:14:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.43, 4.06, 3.98 [08:16:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.81, 3.89 [08:17:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.30, 1.80, 1.60 [08:18:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.62, 3.75 [08:19:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.68, 1.75, 1.61 [08:20:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.51, 4.30, 3.96 [08:20:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.21, 3.95, 3.87 [08:21:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.47, 1.62, 1.58 [08:22:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.78, 3.58, 3.73 [08:26:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.22, 4.15, 3.87 [08:27:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.75, 2.16, 1.79 [08:32:04] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [08:35:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.46, 1.79, 1.81 [08:39:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.02, 1.46, 1.67 [08:50:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.59, 3.20, 3.91 [08:53:08] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [08:57:14] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.47, 2.02, 1.80 [08:59:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.35, 1.80, 1.75 [09:00:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 3.74, 3.79 [09:00:38] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [09:00:38] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:01:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.75, 1.47, 1.64 [09:02:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.01, 3.08, 3.54 [09:02:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.49, 3.86 [09:04:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.55, 3.52, 3.64 [09:06:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.27, 3.01, 3.44 [09:08:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.32, 3.48, 3.55 [09:08:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.20, 3.36, 3.60 [09:10:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.38, 3.51 [09:11:53] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/lib/p [09:11:53] ist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wiki.mineland.eu. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:12:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.80, 3.40, 3.57 [09:14:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.13, 2.85, 3.27 [09:14:25] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.53, 1.79, 1.63 [09:14:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.28, 2.77, 3.33 [09:18:14] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.45, 1.68, 1.62 [09:18:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.44, 3.29, 4.00 [09:21:07] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [09:28:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.22, 1.64, 1.95 [09:28:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.33, 3.71, 3.73 [09:30:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.82, 3.70, 3.48 [09:32:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.79, 3.84, 3.54 [09:34:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.72, 3.53 [09:36:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.82, 3.89, 3.39 [09:36:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.19, 1.36, 1.69 [09:38:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.34, 3.32, 3.24 [09:38:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.77, 3.23, 3.37 [09:43:05] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.82, 1.57, 1.52 [09:44:59] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.57, 1.42, 1.46 [09:50:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.60, 1.57 [09:50:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.05, 3.43, 3.92 [09:52:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.78, 3.99 [09:54:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.47, 3.85 [09:56:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.22, 1.57, 1.59 [09:56:27] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.18, 1.89, 1.67 [09:56:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.43, 4.14, 4.04 [09:58:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.40, 3.25, 3.02 [10:00:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 3.26, 3.05 [10:00:16] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.93, 1.94, 1.75 [10:02:11] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.14, 2.03, 1.81 [10:02:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.22, 3.77, 3.91 [10:04:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.58, 4.27, 4.06 [10:06:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.89, 3.94 [10:07:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.37, 3.07, 3.10 [10:09:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.19, 1.74, 1.84 [10:10:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.65, 3.78 [10:10:38] RECOVERY - wiki.mineland.eu - reverse DNS on sslhost is OK: SSL OK - wiki.mineland.eu reverse DNS resolves to cp21.miraheze.org - CNAME OK [10:12:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.24, 3.62 [10:15:21] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.90, 1.76, 1.64 [10:16:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.44, 3.59 [10:17:20] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.70, 2.02, 1.74 [10:18:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.14, 3.01, 3.41 [10:20:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.09, 3.52, 3.53 [10:22:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.93, 3.31, 3.44 [10:23:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.03, 1.45, 1.66 [10:24:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.81, 3.76, 3.59 [10:26:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.41, 3.40, 3.46 [10:28:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 3.73, 3.56 [10:29:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.86, 1.90 [10:30:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.90, 3.66 [10:32:46] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.56, 1.71, 1.71 [10:33:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.90, 1.91 [10:34:41] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.18, 1.52, 1.64 [10:35:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.80, 1.88, 1.90 [10:38:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.27, 2.95, 3.35 [10:39:36] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=seccdxmodelwiki --report 1 --no-updates --username-prefix=mh:seccdx /home/reception/seccdxwiki_xml_91d8b72478f28f3271c6.xml (START) [10:39:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:41:14] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.18, 2.01, 1.95 [10:42:17] !log reception@mwtask111:~/seccdxmodel$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=seccdxmodelwiki --search-recursively /home/reception/seccdxmodel --user=Miraheze_maintenance_script --sleep --comment="Imported from seccdx.miraheze.org" [10:42:21] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.35, 1.94, 1.75 [10:42:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:42:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.51, 3.50, 3.50 [10:42:32] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=seccdxmodelwiki --report 1 --no-updates --username-prefix=mh:seccdx /home/reception/seccdxwiki_xml_91d8b72478f28f3271c6.xml (END - exit=0) [10:42:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:43:15] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=seccdxwiki (START) [10:43:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:43:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.13, 3.42, 3.16 [10:43:59] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=seccdxwiki (END - exit=0) [10:44:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:44:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.75, 3.59 [10:45:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.74, 2.83, 2.98 [10:45:54] !log [reception@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (START) [10:46:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:46:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.42, 3.48 [10:48:05] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.64, 1.96, 1.85 [10:48:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.39, 3.59, 3.52 [10:52:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.68, 3.59 [10:55:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.08, 2.04, 1.91 [10:56:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.65, 3.05, 3.34 [11:00:04] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [11:02:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.42, 3.44 [11:04:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.07, 3.96, 3.62 [11:05:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.98, 1.96, 1.98 [11:06:12] PROBLEM - mwtask111 Disk Space on mwtask111 is WARNING: DISK WARNING - free space: / 6446 MB (9% inode=88%); [11:07:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.79, 2.05, 2.02 [11:08:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.77, 3.77, 3.62 [11:09:06] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.90, 2.00 [11:09:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.46, 1.95, 1.99 [11:10:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.22, 3.15, 3.40 [11:11:36] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/landing/compare/91f79a7d8e6d...ed842beb9172 [11:11:37] [url] Comparing 91f79a7d8e6d...ed842beb9172 · miraheze/landing · GitHub | github.com [11:11:38] [02miraheze/landing] 07translatewiki 03ed842be - Localisation updates from https://translatewiki.net. [11:11:38] [url] Main page - translatewiki.net | translatewiki.net [11:12:36] miraheze/landing - translatewiki the build passed. [11:13:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.05, 4.63, 3.68 [11:14:12] PROBLEM - mwtask111 Disk Space on mwtask111 is CRITICAL: DISK CRITICAL - free space: / 2904 MB (4% inode=86%); [11:14:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.46, 3.50 [11:15:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.75, 3.85, 3.51 [11:16:19] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=webkinzguidewiki --report 1 --username-prefix=webkinzpictureguide --no-updates /home/reception/webkinzpictureguideshoutwikicom_w-20220524-history.xml (START) [11:16:20] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=webkinzguidewiki --report 1 --username-prefix=webkinzpictureguide --no-updates /home/reception/webkinzpictureguideshoutwikicom_w-20220524-history.xml (END - exit=256) [11:16:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:16:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:16:43] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=webkinzguidewiki --report 1 --username-prefix=webkinzpictureguide --no-updates /home/reception/webkinzguide/webkinzpictureguideshoutwikicom_w-20220524-wikidump/webkinzpictureguideshoutwikicom_w-20220524-history.xml (START) [11:16:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:17:04] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.12, 1.72, 1.83 [11:17:12] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - fc.songcontests.eu All nameservers failed to answer the query. [11:18:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.82, 2.91, 3.29 [11:19:03] !log reception@mwtask111:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=webkinzguidewiki --search-recursively /home/reception/webkinzguide --user=Miraheze_maintenance_script --sleep --comment="Imported from http://webkinzpictureguide.shoutwiki.com" [11:19:04] [url] Webkinz Pictures | webkinzpictureguide.shoutwiki.com [11:19:10] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.35, 3.17, 3.31 [11:19:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:23:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.44, 1.94, 1.93 [11:23:36] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=webkinzguidewiki --report 1 --username-prefix=webkinzpictureguide --no-updates /home/reception/webkinzguide/webkinzpictureguideshoutwikicom_w-20220524-wikidump/webkinzpictureguideshoutwikicom_w-20220524-history.xml (END - exit=2) [11:23:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:25:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.47, 1.90, 1.82 [11:27:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.91, 1.93, 1.84 [11:29:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.11, 3.11 [11:29:54] PROBLEM - wiki.thunis.eu - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/lib/pyt [11:29:54] t-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wiki.thunis.eu. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:31:00] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.11, 1.35, 1.64 [11:31:57] !log [reception@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php --type=htmlCacheUpdate (START) [11:32:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:32:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 3.59, 3.37 [11:33:38] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.36, 3.16, 3.12 [11:34:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.80, 3.67, 3.43 [11:35:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.22, 1.55, 1.68 [11:39:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.83, 10.49, 8.81 [11:41:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.44, 10.17, 8.92 [11:44:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 4.02, 3.67 [11:45:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.41, 1.89, 1.73 [11:46:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.76, 3.54, 3.53 [11:48:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.10, 3.41, 2.72 [11:48:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.92, 3.68 [11:48:55] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.60, 1.96, 1.71 [11:50:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.33, 10.14, 8.56 [11:52:54] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.64, 1.89, 1.75 [11:52:55] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.87, 9.40, 8.49 [11:52:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 3.76, 3.44 [11:56:11] PROBLEM - www.lab612.at - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.lab612.at All nameservers failed to answer the query. [11:56:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.07, 3.45, 3.08 [11:56:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.27, 3.93, 3.60 [11:56:52] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.89, 1.77 [11:58:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.17, 3.94, 3.49 [11:58:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.39, 3.73, 3.22 [11:58:52] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.84, 1.76 [11:59:17] RECOVERY - wiki.thunis.eu - reverse DNS on sslhost is OK: SSL OK - wiki.thunis.eu reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:00:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.81, 3.71, 3.28 [12:01:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.63, 10.57, 9.93 [12:02:08] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.41, 3.21, 3.29 [12:02:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.17, 3.39, 3.20 [12:02:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.99, 3.73, 3.57 [12:02:51] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.10, 1.91, 1.80 [12:04:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.50, 3.51 [12:05:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.41, 11.08, 10.24 [12:07:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.27, 11.17, 10.38 [12:08:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.54, 3.03, 3.33 [12:11:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.34, 11.79, 10.79 [12:12:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.48, 3.46, 3.36 [12:12:12] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mineland.eu All nameservers failed to answer the query. [12:12:49] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.95, 1.90 [12:13:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.76, 11.62, 10.84 [12:13:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.14, 1.73, 2.00 [12:14:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.70, 3.94, 3.55 [12:14:48] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.94, 1.91 [12:17:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.79, 11.82, 11.06 [12:17:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.58, 9.95, 8.78 [12:17:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.78, 2.01, 2.03 [12:18:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.61, 10.25, 9.18 [12:19:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.58, 11.66, 11.11 [12:19:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.20, 8.90, 8.54 [12:19:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.62, 1.81, 1.95 [12:20:41] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [12:20:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.68, 10.51, 9.42 [12:22:55] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.93, 10.18, 9.42 [12:23:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.27, 11.67, 11.23 [12:25:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.13, 11.08, 11.08 [12:25:19] RECOVERY - www.lab612.at - reverse DNS on sslhost is OK: SSL OK - www.lab612.at reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:27:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.85, 2.27, 2.05 [12:30:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.93, 3.52 [12:30:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.76, 3.24, 3.04 [12:31:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.99, 1.77, 1.92 [12:32:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.02, 3.23, 3.31 [12:32:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.19, 3.05, 2.98 [12:39:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.11, 9.38, 10.16 [12:41:35] RECOVERY - wiki.mineland.eu - reverse DNS on sslhost is OK: SSL OK - wiki.mineland.eu reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:41:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.51, 1.86, 1.83 [12:42:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.04, 3.92, 3.61 [12:43:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.69, 10.83, 10.59 [12:44:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.28, 3.12 [12:44:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.72, 3.82, 3.61 [12:45:12] RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: SSL OK - fc.songcontests.eu reverse DNS resolves to cp20.miraheze.org - CNAME OK [12:45:41] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [12:46:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.24, 2.92, 3.00 [12:46:40] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.92, 2.00 [12:46:56] !log [reception@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php --type=htmlCacheUpdate (END - exit=0) [12:47:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:48:40] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.51, 2.13, 2.06 [12:48:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.65, 3.63, 3.55 [12:49:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.30, 1.82, 1.88 [12:50:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 4.00, 3.68, 3.58 [12:52:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.08, 3.11, 3.38 [12:53:08] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [12:55:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.32, 1.97, 1.90 [12:55:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.88, 10.50, 9.97 [12:57:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.07, 10.76, 10.49 [12:58:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.41, 1.79, 1.99 [12:58:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.60, 3.53 [12:59:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.36, 1.83, 1.89 [13:01:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.75, 10.52, 10.55 [13:02:04] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [13:02:04] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:02:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.51, 4.38, 3.62 [13:03:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.37, 9.46, 9.75 [13:03:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.68, 2.09, 1.97 [13:05:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.26, 11.84, 11.01 [13:05:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.73, 1.86, 1.90 [13:06:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.35, 3.45, 3.43 [13:06:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.81, 3.10, 3.33 [13:07:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.97, 11.98, 11.18 [13:07:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.11, 2.12, 2.00 [13:08:34] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.14, 1.34, 1.66 [13:09:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.61, 1.91, 1.93 [13:10:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.17, 3.95, 3.61 [13:10:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.55, 3.47 [13:12:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.48, 3.48 [13:12:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.22, 3.16, 3.34 [13:14:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.75, 3.26, 3.39 [13:15:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.45, 11.47, 11.10 [13:19:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.75, 11.37, 11.16 [13:19:05] !log [reception@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (END - exit=0) [13:19:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:19:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.10, 11.01, 10.26 [13:20:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.22, 1.82, 1.69 [13:20:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 3.70, 3.46 [13:21:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 3.44, 3.43 [13:21:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.28, 11.62, 11.27 [13:21:07] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [13:21:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.99, 11.10, 10.39 [13:21:46] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.41, 3.59, 3.23 [13:22:30] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.88, 1.73 [13:22:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.03, 3.35, 3.40 [13:23:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.56, 10.88, 11.05 [13:23:40] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.25, 3.75, 3.33 [13:25:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.72, 9.74, 9.99 [13:25:35] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.65, 3.26, 3.19 [13:25:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.02, 1.79, 1.77 [13:26:29] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.34, 1.69, 1.69 [13:26:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.91, 3.63 [13:26:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.91, 3.66 [13:28:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.47, 3.50 [13:28:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.85, 3.66 [13:29:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.79, 11.90, 11.33 [13:30:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.56, 3.51 [13:30:38] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:31:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.22, 11.64, 11.31 [13:33:15] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.73, 3.47, 3.29 [13:33:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.37, 9.68, 8.88 [13:34:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.30, 10.81, 10.30 [13:35:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.32, 11.76, 11.43 [13:35:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.91, 9.31, 8.87 [13:36:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.97, 10.49, 10.25 [13:36:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.23, 3.50, 3.58 [13:37:05] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.76, 3.18, 3.22 [13:38:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.22, 4.27, 3.88 [13:40:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 3.85, 3.77 [13:40:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.68, 3.64, 3.77 [13:41:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.02, 11.84, 11.63 [13:42:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.83, 3.76 [13:42:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.73, 1.68 [13:42:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.51, 3.80, 3.79 [13:43:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.08, 12.11, 11.76 [13:43:42] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.83, 10.04, 10.17 [13:44:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.86, 3.70, 3.73 [13:44:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.64, 1.69, 1.67 [13:44:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.75, 3.41, 3.65 [13:45:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.14, 11.25, 11.48 [13:46:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 2.01, 2.77, 3.88 [13:48:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.22, 2.27, 1.89 [13:48:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.13, 3.58, 3.40 [13:48:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.13, 3.37, 3.52 [13:50:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.29, 4.10, 3.82 [13:50:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.34, 3.50 [13:52:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.47, 3.62 [13:52:37] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.69, 3.12, 3.26 [13:52:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.58, 3.57 [13:54:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 3.47, 3.53 [13:56:08] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.67, 2.73, 3.27 [13:58:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.45, 3.05, 3.39 [14:02:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.14, 3.15, 3.37 [14:06:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.31, 3.51, 3.48 [14:08:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.24, 3.59, 3.50 [14:10:41] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [14:10:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.92, 3.83, 3.61 [14:11:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.00, 9.20, 10.09 [14:11:20] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.34, 10.40, 9.63 [14:12:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.18, 4.19, 3.76 [14:13:20] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.31, 9.60, 9.43 [14:13:23] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.58, 3.43, 3.31 [14:14:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.57, 3.42 [14:14:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.86, 3.69 [14:15:18] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.97, 3.25, 3.25 [14:16:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.47, 3.25, 3.32 [14:17:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.41, 1.81, 2.00 [14:25:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.05, 2.13, 2.01 [14:26:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.26, 4.49, 3.92 [14:28:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.83, 3.55 [14:28:42] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.55, 3.15, 3.06 [14:29:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.27, 1.93, 1.97 [14:30:37] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.07, 3.08, 3.04 [14:30:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.87, 3.81 [14:33:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.46, 10.24, 10.09 [14:34:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.84, 1.99 [14:35:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.01, 1.83, 1.91 [14:36:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.26, 1.99, 2.02 [14:37:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.98, 1.89, 1.92 [14:38:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.89, 1.98 [14:39:13] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.53, 9.84, 10.04 [14:40:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.96, 4.29, 3.76 [14:40:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 3.91, 3.77 [14:42:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.94, 3.97, 3.81 [14:43:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.13, 1.75, 1.83 [14:44:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.22, 3.95, 3.81 [14:45:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.43, 1.56, 1.74 [14:46:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.95, 3.78 [14:47:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.28, 1.41, 1.66 [14:48:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.04, 1.77, 1.82 [14:48:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.80, 3.78 [14:50:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 4.20, 3.91 [14:50:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.77, 4.14, 3.91 [14:52:06] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.45, 3.67, 3.37 [14:52:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.99, 1.93 [14:52:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.62, 3.88, 3.83 [14:53:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.08, 1.70, 1.70 [14:54:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.65, 3.76 [14:54:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.01, 4.26, 3.97 [14:55:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.32, 1.60, 1.67 [14:55:55] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.60, 3.84, 3.51 [14:56:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.85, 5.02, 4.21 [14:58:12] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:13] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.24, 1.46, 1.70 [14:58:45] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:09] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:27] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:48] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:16] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14642 bytes in 4.672 second response time [15:00:26] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:28] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:35] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:41] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [15:00:45] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14642 bytes in 3.265 second response time [15:01:27] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 1.699 second response time [15:01:39] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.94, 3.27, 3.37 [15:01:44] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14656 bytes in 0.840 second response time [15:02:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.66, 1.81, 1.75 [15:04:39] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.62, 1.37, 1.59 [15:07:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.37, 10.69, 9.95 [15:07:29] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14656 bytes in 8.578 second response time [15:08:39] !log [reception@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (START) [15:08:40] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14642 bytes in 9.131 second response time [15:08:50] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14642 bytes in 7.933 second response time [15:08:50] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 8.340 second response time [15:08:56] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14650 bytes in 7.016 second response time [15:09:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.71, 11.42, 10.31 [15:09:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:11:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.31, 10.78, 10.22 [15:11:20] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.15, 1.78, 1.67 [15:11:51] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:55] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14642 bytes in 9.122 second response time [15:16:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.66, 1.62 [15:18:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.13, 1.83, 1.69 [15:21:29] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:32] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 6.466 second response time [15:26:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.99, 1.85 [15:30:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.08, 1.94, 1.85 [15:32:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.95, 1.97, 1.87 [15:32:54] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/46dcf1972dc5 [15:32:56] [02miraheze/mw-config] 07Reception123 0346dcf19 - change default thumbsize to 3 for criticalrolewiki T9282 [15:32:57] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [15:32:57] [url] Page not found · GitHub · GitHub | github.com [15:32:59] [02mw-config] 07Reception123 opened pull request 03#4712: change default thumbsize to 3 for criticalrolewiki T9282 - 13https://github.com/miraheze/mw-config/pull/4712 [15:32:59] [url] Page not found · GitHub · GitHub | github.com [15:34:02] miraheze/mw-config - Reception123 the build passed. [15:34:21] [02mw-config] 07Reception123 closed pull request 03#4712: change default thumbsize to 3 for criticalrolewiki T9282 - 13https://github.com/miraheze/mw-config/pull/4712 [15:34:22] [url] Page not found · GitHub · GitHub | github.com [15:34:23] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/89159e7433da...6f9c5da6d9e0 [15:34:23] [url] Comparing 89159e7433da...6f9c5da6d9e0 · miraheze/mw-config · GitHub | github.com [15:34:24] [02miraheze/mw-config] 07Reception123 036f9c5da - change default thumbsize to 3 for criticalrolewiki T9282 (#4712) [15:34:26] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [15:34:26] [url] Page not found · GitHub · GitHub | github.com [15:34:27] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [15:34:34] !log [reception@mwtask111] starting deploy of {'config': True} to all [15:34:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:34:51] !log [reception@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 17s [15:34:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.41, 12.10, 11.24 [15:35:23] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=obeymewiki (START) [15:35:32] miraheze/mw-config - Reception123 the build passed. [15:35:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:39:22] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.03, 3.70, 3.42 [15:40:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 1.99, 2.02, 1.94 [15:40:41] alerting : [FIRING:1, RESOLVED:1] (mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [15:41:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.12, 11.61, 11.37 [15:41:17] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.77, 3.67, 3.44 [15:42:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.49, 1.81, 1.87 [15:42:31] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.68, 11.92, 11.50 [15:43:12] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.77, 3.34, 3.35 [15:44:32] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14650 bytes in 6.815 second response time [15:47:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.30, 11.25, 11.35 [15:48:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.59, 2.15, 1.99 [15:50:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.79, 3.46, 2.96 [15:50:41] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [15:51:05] !log [@test101] starting deploy of {'config': True} to all [15:51:06] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [15:51:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:51:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:51:50] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.31, 3.44, 3.39 [15:52:09] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 5.56, 4.23, 3.31 [15:53:44] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.78, 3.31, 3.35 [15:54:44] !log [@mwtask111] starting deploy of {'config': True} to all [15:54:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:55:00] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 17s [15:55:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:01:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.22, 11.00, 10.90 [16:03:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.59, 10.49, 10.73 [16:05:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.09, 11.08, 10.92 [16:07:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.36, 10.63, 10.78 [16:10:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.51, 3.68, 3.91 [16:10:59] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.02, 3.72, 3.47 [16:11:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.68, 11.01, 10.83 [16:12:54] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.03, 3.44, 3.39 [16:13:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.52, 9.97, 10.49 [16:16:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.92, 1.94, 1.99 [16:18:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.27, 3.63, 3.70 [16:18:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.70, 2.25, 2.11 [16:18:37] [02puppet] 07Universal-Omega opened pull request 03#2589: Remove arcanist from hieradata - 13https://github.com/miraheze/puppet/pull/2589 [16:18:37] [url] Page not found · GitHub · GitHub | github.com [16:19:55] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-1/±2] 13https://github.com/miraheze/puppet/compare/9dcc1aba82fe...0fdc6745205d [16:19:56] [url] Comparing 9dcc1aba82fe...0fdc6745205d · miraheze/puppet · GitHub | github.com [16:19:57] [02miraheze/puppet] 07Universal-Omega 030fdc674 - Remove arcanist from hieradata (#2589) [16:19:58] [02puppet] 07Reception123 closed pull request 03#2589: Remove arcanist from hieradata - 13https://github.com/miraheze/puppet/pull/2589 [16:19:59] ... [16:20:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.20, 3.48, 3.64 [16:20:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.11, 3.74, 3.53 [16:20:38] [02puppet] 07Universal-Omega opened pull request 03#2590: Remove `use_gluster_new` from hieradata - 13https://github.com/miraheze/puppet/pull/2590 [16:20:39] ... [16:21:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.71, 10.14, 9.47 [16:21:21] [02puppet] 07Reception123 closed pull request 03#2590: Remove `use_gluster_new` from hieradata - 13https://github.com/miraheze/puppet/pull/2590 [16:21:22] [url] Page not found · GitHub · GitHub | github.com [16:21:23] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0fdc6745205d...157a3b8b8ca8 [16:21:23] [url] Comparing 0fdc6745205d...157a3b8b8ca8 · miraheze/puppet · GitHub | github.com [16:21:24] [02miraheze/puppet] 07Universal-Omega 03157a3b8 - Remove `use_gluster_new` from hieradata (#2590) [16:22:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.09, 3.63, 3.67 [16:22:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.77, 3.74, 3.56 [16:22:55] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.86, 9.68, 9.38 [16:23:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.64, 1.67, 2.00 [16:25:41] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [16:25:50] again [16:25:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.72, 2.38, 2.21 [16:25:58] well it's running... [16:26:09] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.86, 3.85, 3.73 [16:27:56] [02puppet] 07Universal-Omega opened pull request 03#2591: Remove `puppetmaster` from hieradata - 13https://github.com/miraheze/puppet/pull/2591 [16:27:56] [url] Page not found · GitHub · GitHub | github.com [16:28:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.11, 1.68, 1.96 [16:29:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.83, 9.56, 10.11 [16:30:37] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.73, 3.14, 3.37 [16:31:19] Reception123: I updated the alert summary, so it was restarted when it updated, so stopped alerting during the time and started up again. It's at like 10,000 unclaimed jobs last I looked. [16:32:03] CosmicAlpha: hmm, well there's nothing more I can do since I'm running jobs right now [16:32:08] so this might not be a good way to do thing [16:32:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.11, 4.12, 3.89 [16:32:10] *things [16:32:55] [discord] Jobrunner acting up again? [16:33:02] As always [16:33:15] This is after Krinkle suggested an improvement [16:33:45] Reception123: are you running on only that wiki with --wait? [16:33:56] [discord] If i remember right, last time, we did a rolling restart of both jobrunners, restarting one at a time [16:34:07] All jobs? [16:34:18] Zppix: it's Reception123 blowing stuff up [16:34:34] [discord] Then we restart the human 😛 [16:35:04] .!log root@reception123: reboot now [16:35:12] RhinosF1: well the initial script was running on all wikis but in effect it's running only on that wiki [16:35:30] Reception123: no, run it on just the one wiki with --wait [16:35:32] I guess I can start a new one maybe to speed things up [16:35:35] All types of jobs [16:35:39] [discord] What job is it that seemingly isnt behaving? [16:35:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.54, 1.61, 1.94 [16:35:56] html cache + links [16:36:07] [discord] Is it all wikis? [16:36:16] Nope [16:36:18] A single wiki [16:36:28] Because reception is importing stuff [16:36:34] [discord] Could it be something on that wiki, that mediawiki doesnt like? [16:36:41] running now, should probably add START/STOP for runjobs with mwscript [16:36:43] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=webkinzguidewiki --wait (END - exit=256) [16:36:48] @Zppix it's because of a change Krinkle suggested [16:36:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:52] Reception123: it does [16:36:54] every time someone does an import jobs go crazy because of it [16:36:57] [discord] Which change is that? [16:37:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.08, 10.35, 10.13 [16:37:08] https://www.irccloud.com/pastebin/wAwa7FWn/ [16:37:10] ^ RhinosF1 well that's fun [16:37:12] adding --maxjobs 1 to the queue [16:37:27] Reception123: you broke the world [16:37:45] [discord] Wouldnt that make it so it would only run 1 job at a time? [16:37:53] and for the record, there's also a refreshLinks running for another wiki [16:37:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.03, 1.80, 1.97 [16:38:01] Yes Zppix [16:38:13] Rather than affect every wiki when one wiki gets too many jobs [16:38:21] chrs: you have ping on testwiki [16:38:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.42, 1.45, 1.70 [16:38:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.81, 3.38, 3.34 [16:39:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.74, 9.86, 9.99 [16:39:11] [discord] Tbh job running on mediawiki needs to be reworked, i don’t really think theres any combination that could fix this [16:39:34] refreshLinks still going; importing images has failed with the socket error now [16:40:07] [discord] Could that be throttling? [16:40:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.64, 3.95, 3.56 [16:41:16] [discord] My interpretation of the error message you posted makes it sound like a connection issue. Since, it’s all internal connections, I could only think that throttling would cause that [16:41:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.65, 1.53, 1.85 [16:42:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.70, 10.12, 9.08 [16:42:29] [discord] Are we still running a 2 jobrunner setup? [16:44:27] nope [16:44:41] it's a all mw*s are jobrunners setup now [16:45:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.73, 10.65, 10.31 [16:45:41] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [16:46:02] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.72, 9.38, 9.05 [16:46:11] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+2/-0/±2] 13https://github.com/miraheze/puppet/compare/c5283d248df1...f235ba7c6794 [16:46:12] [url] Comparing c5283d248df1...f235ba7c6794 · miraheze/puppet · GitHub | github.com [16:46:13] [02miraheze/puppet] 07Reception123 03f235ba7 - add varnish/nginx to puppet111 (moving SSL) [16:46:14] [02puppet] 07Reception123 synchronize pull request 03#2519: remove addthis.com from the CSP (per replacement, see T7876) - 13https://github.com/miraheze/puppet/pull/2519 [16:46:15] [url] Page not found · GitHub · GitHub | github.com [16:46:15] Are you running a content import and a image import simulationusly [16:46:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.92, 3.87, 3.70 [16:47:00] [02puppet] 07Reception123 commented on pull request 03#2519: remove addthis.com from the CSP (per replacement, see T7876) - 13https://github.com/miraheze/puppet/pull/2519#issuecomment-1138773057 [16:47:01] [url] remove addthis.com from the CSP (per replacement, see T7876) by Reception123 · Pull Request #2519 · miraheze/puppet · GitHub | github.com [16:47:02] [02puppet] 07Reception123 closed pull request 03#2519: remove addthis.com from the CSP (per replacement, see T7876) - 13https://github.com/miraheze/puppet/pull/2519 [16:47:02] [url] Page not found · GitHub · GitHub | github.com [16:47:12] Zppix: and a refreshlinks [16:47:23] though I mean it's nothing uncommon, and we can't keep people waiting for that long [16:47:48] [02miraheze/puppet] 07Reception123 created branch 03Reception123-patch-3 13https://github.com/miraheze/puppet/commit/f235ba7c6794afbffd855d4f519888bf88bf562e [16:47:49] [02puppet] 07Reception123 created branch 03Reception123-patch-3 - 13https://github.com/miraheze/puppet [16:47:50] [url] Page not found · GitHub · GitHub | github.com [16:47:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.05, 1.62, 1.77 [16:48:02] I agree, the thing is without a dedicated jobrunner, I’m not sure if it can handle doing those all at the same time [16:48:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.04, 3.89, 3.72 [16:49:00] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f235ba7c6794...e65c80d2980c [16:49:00] [url] Comparing f235ba7c6794...e65c80d2980c · miraheze/puppet · GitHub | github.com [16:49:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.27, 9.83, 10.04 [16:49:01] Doing both at the same time, you are creating more jobs, then you can run at once [16:49:01] [02miraheze/puppet] 07Reception123 03e65c80d - re-add; mistake [16:49:35] [02puppet] 07Reception123 opened pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [16:49:35] [url] Page not found · GitHub · GitHub | github.com [16:49:48] Zppix: well, it's only since this change. I don't remember it being a problem before [16:50:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.44, 1.91, 1.75 [16:50:24] Before though you werent limiting it to maxjobs to 1. So technically its working as intended [16:50:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.19, 3.68, 3.67 [16:51:06] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e65c80d2980c...04038cdfa01c [16:51:07] [url] Comparing e65c80d2980c...04038cdfa01c · miraheze/puppet · GitHub | github.com [16:51:08] [02miraheze/puppet] 07Reception123 0304038cd - fix [16:51:09] [02puppet] 07Reception123 synchronize pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [16:51:10] [url] Page not found · GitHub · GitHub | github.com [16:51:36] yeah, though it has unintended consequences [16:52:02] paladox: when you've got some spare time, mind having a look at https://github.com/miraheze/puppet/pull/2592 ? I'm sure I did something very wrong on the nginx side, I wasn't quite sure what to do other than what I tried [16:52:02] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [16:52:29] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.15, 2.15, 1.44 [16:52:37] I wonder if you somehow overide the maxjobs param temporarily to get through the refreshlinks, then reintroduce it, I wonder if it would once again struggle to keep up [16:53:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.03, 11.36, 10.66 [16:53:11] That would confirm it’s the parameter causing issues and not some underlying issue [16:53:32] that would be interesting [16:54:29] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.65, 1.68, 1.35 [16:54:35] Otherwise, i would just say its jobrunner being jobrunner, half of my time as a sysadmin was spent going bald because of the sudden 10k pending jobs every other week lol [16:55:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.64, 10.79, 10.55 [16:55:51] yeah, though I'm quite sure before the param things went fine when doing imports recently [16:55:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.72, 1.98, 1.94 [16:56:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.45, 1.77, 1.77 [16:57:51] Reception123: before the param, I don’t think any of the job related issues were import related iirc. Jobrunning in mw is poorly designed as it is. I find it hard to believe that WMF never ran into this issue as bad, when they handled jobs similarly to this [16:57:52] PROBLEM - cp21 Disk Space on cp21 is WARNING: DISK WARNING - free space: / 4273 MB (10% inode=96%); [16:57:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.73, 2.62, 2.18 [16:58:01] [02puppet] 07JohnFLewis reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r882882359 [16:58:02] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [16:58:03] [02puppet] 07JohnFLewis reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r882883051 [16:58:03] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [16:58:04] [02puppet] 07JohnFLewis reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r882881388 [16:58:05] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [16:59:52] [02puppet] 07Universal-Omega reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r882881075 [16:59:53] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [16:59:53] [02puppet] 07Universal-Omega reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r882882708 [16:59:55] ... [16:59:55] [02puppet] 07Universal-Omega reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r882881632 [16:59:57] ... [17:00:07] Yes MHBOT … [17:01:02] [02puppet] 07Reception123 reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r882886344 [17:01:03] ... [17:01:28] [02puppet] 07Reception123 reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r882886794 [17:02:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.31, 1.57, 1.68 [17:02:36] JohnLewis: the 'Okay' was actually from the file you gave me. I just didn't realise it accidentally found its way there :P [17:02:41] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [17:03:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.43, 8.48, 9.74 [17:03:02] but yeah, since I don't want to completely remove mwtask111 yet, is it possible to add two in the same row, or did you mean you wanted me to remove mwtask111 from the backends? [17:03:07] [02puppet] 07Universal-Omega reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r882888058 [17:03:08] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [17:03:26] PROBLEM - prometheus101 Current Load on prometheus101 is CRITICAL: CRITICAL - load average: 4.64, 2.46, 1.21 [17:03:35] icinga-miraheze: “yes mediawiki” [17:04:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.76, 9.74, 8.77 [17:05:14] Zppix: lol that always bugged me. But https://github.com/miraheze/puppet/pull/2588 should make those labels not appear anymore... [17:05:15] [url] Rewrite grafana-webhook by Universal-Omega · Pull Request #2588 · miraheze/puppet · GitHub | github.com [17:05:34] CosmicAlpha: about the switch, if I don't do it though won't the script not work? [17:05:42] As that's why I understood we needed varnish [17:05:46] CosmicAlpha: kinda feel like it is just icinga being snarky [17:06:18] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.01, 8.82, 8.56 [17:06:37] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.32, 2.80, 3.23 [17:07:21] PROBLEM - prometheus101 Current Load on prometheus101 is WARNING: WARNING - load average: 0.98, 1.89, 1.29 [17:07:32] Reception123: maybe, but you can't have 2 backends for the same thing. All the backends are done on cache proxies, so trying 2 makes the cache proxies try 2, which won't exactly work. The way it is now, only the second one would take effect anyway I think, and override the other. [17:07:41] alerting : [FIRING:1, RESOLVED:1] (mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [17:08:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.91, 3.64, 3.96 [17:09:21] RECOVERY - prometheus101 Current Load on prometheus101 is OK: OK - load average: 0.58, 1.44, 1.20 [17:10:06] CosmicAlpha: oh ok, I guess I can remove it for now and if it doesn't work I'll do it then [17:10:26] PROBLEM - cp20 Disk Space on cp20 is WARNING: DISK WARNING - free space: / 4266 MB (10% inode=96%); [17:10:28] If it’s not broke, break it. :P [17:11:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.10, 10.11, 9.92 [17:11:13] Reception123: also, letsencrypt::nginx isn't being loaded anywhere it seems. [17:11:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.34, 10.23, 9.73 [17:12:54] CosmicAlpha: oh, where am I supposed to? [17:13:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.07, 11.02, 10.27 [17:13:23] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.79, 9.67, 9.58 [17:13:33] RhinosF1: seen [17:14:22] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=obeymewiki (END - exit=0) [17:14:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.72, 1.67 [17:14:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:14:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.52, 4.00 [17:15:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.62, 10.29, 10.09 [17:15:19] Reception123: to prevent conflict while it's still on mwtask111, I guess in site? Otherwise in init.pp, but that would be loaded on all LetsEncrypt servers, meaning mwtask111, as well, which is already done in MediaWiki. I'm not 100% sure here. [17:16:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.39, 3.71, 3.75 [17:16:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.59, 1.99, 1.77 [17:16:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.78, 3.81, 4.04 [17:17:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.47, 9.86, 9.97 [17:20:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.89, 2.93, 3.86 [17:20:39] ^ JohnLewis what would you suggest? [17:21:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.47, 10.50, 10.24 [17:23:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.18, 10.78, 10.35 [17:24:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.04, 3.47, 3.86 [17:24:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 4.00, 3.92 [17:25:28] !log [reception@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (END - exit=0) [17:25:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:25:49] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.06, 11.68, 10.32 [17:26:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 2.46, 3.20, 3.72 [17:26:13] icinga-miraheze: we know, Reception123 is currently melting the servers, it will be ok [17:27:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.82, 11.46, 10.73 [17:27:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.27, 11.45, 10.39 [17:28:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.79, 1.97, 1.98 [17:29:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.75, 12.10, 11.07 [17:30:08] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.42, 2.27, 3.21 [17:30:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.02, 4.06, 3.93 [17:30:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.35, 2.10, 2.02 [17:30:46] [02ssl] 07Universal-Omega opened pull request 03#529: Add 2 certs - 13https://github.com/miraheze/ssl/pull/529 [17:30:47] [url] Page not found · GitHub · GitHub | github.com [17:31:55] [02ssl] 07Universal-Omega synchronize pull request 03#529: Add 2 certs - 13https://github.com/miraheze/ssl/pull/529 [17:31:56] [url] Page not found · GitHub · GitHub | github.com [17:32:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.80, 3.85 [17:32:17] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/04038cdfa01c...aa988bdea115 [17:32:18] [url] Comparing 04038cdfa01c...aa988bdea115 · miraheze/puppet · GitHub | github.com [17:32:19] [02miraheze/puppet] 07Reception123 03aa988bd - requested changes [17:32:20] [02puppet] 07Reception123 synchronize pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [17:32:21] [url] Page not found · GitHub · GitHub | github.com [17:32:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.81, 1.99, 1.99 [17:32:47] [02puppet] 07Reception123 commented on pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592#issuecomment-1138831754 [17:32:48] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [17:33:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.48, 11.71, 11.18 [17:33:09] [02ssl] 07Universal-Omega synchronize pull request 03#529: Add 2 certs - 13https://github.com/miraheze/ssl/pull/529 [17:33:10] [url] Page not found · GitHub · GitHub | github.com [17:33:53] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/aa988bdea115...54afaaa11e5d [17:33:53] [url] Comparing aa988bdea115...54afaaa11e5d · miraheze/puppet · GitHub | github.com [17:33:54] [02miraheze/puppet] 07Reception123 0354afaaa - add letsencrypt::nginx to site.pp [17:33:56] [02puppet] 07Reception123 synchronize pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [17:33:56] [url] Page not found · GitHub · GitHub | github.com [17:34:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.88, 3.87 [17:36:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.73, 3.51, 3.74 [17:37:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.89, 11.98, 11.37 [17:37:16] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.55, 9.79, 10.11 [17:39:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.30, 11.38, 11.23 [17:41:11] [02ssl] 07Reception123 closed pull request 03#529: Add 2 certs - 13https://github.com/miraheze/ssl/pull/529 [17:41:12] [url] Page not found · GitHub · GitHub | github.com [17:41:13] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+2/-0/±1] 13https://github.com/miraheze/ssl/compare/70143fbbfe78...a8ce4dd035bb [17:41:13] [url] Comparing 70143fbbfe78...a8ce4dd035bb · miraheze/ssl · GitHub | github.com [17:41:14] [02miraheze/ssl] 07Universal-Omega 03a8ce4dd - Add 2 certs (#529) [17:41:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.22, 1.68, 1.91 [17:42:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.40, 3.90 [17:44:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.44, 3.79, 3.98 [17:46:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.53, 2.05, 2.01 [17:48:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.26, 3.86, 3.73 [17:48:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.91, 3.98 [17:50:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.64, 3.65 [17:52:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.15, 3.87, 3.74 [17:52:43] [02ssl] 07Universal-Omega opened pull request 03#530: renew monarchists.wiki cert - 13https://github.com/miraheze/ssl/pull/530 [17:52:44] [url] Page not found · GitHub · GitHub | github.com [17:52:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.49, 3.98, 3.96 [17:53:07] [02ssl] 07Reception123 closed pull request 03#530: renew monarchists.wiki cert - 13https://github.com/miraheze/ssl/pull/530 [17:53:07] ... [17:53:08] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a8ce4dd035bb...8d36d843f4b7 [17:53:09] [url] Comparing a8ce4dd035bb...8d36d843f4b7 · miraheze/ssl · GitHub | github.com [17:53:10] [02miraheze/ssl] 07Universal-Omega 038d36d84 - renew monarchists.wiki cert (#530) [17:53:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.26, 1.45, 1.69 [17:54:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.13, 3.60, 3.65 [17:54:50] [02ssl] 07Universal-Omega opened pull request 03#531: Remove stardewmodding.wiki cert - 13https://github.com/miraheze/ssl/pull/531 [17:54:50] [url] Page not found · GitHub · GitHub | github.com [17:55:04] [02ssl] 07Universal-Omega synchronize pull request 03#531: Remove stardewmodding.wiki cert - 13https://github.com/miraheze/ssl/pull/531 [17:55:05] [url] Page not found · GitHub · GitHub | github.com [17:55:25] [02ssl] 07Universal-Omega synchronize pull request 03#531: Remove stardewmodding.wiki cert - 13https://github.com/miraheze/ssl/pull/531 [17:55:26] ... [17:55:44] [02ssl] 07Universal-Omega synchronize pull request 03#531: Remove stardewmodding.wiki cert - 13https://github.com/miraheze/ssl/pull/531 [17:55:45] [url] Page not found · GitHub · GitHub | github.com [17:56:38] [02dns] 07Universal-Omega opened pull request 03#279: Remove stardewmodding.wiki zone - 13https://github.com/miraheze/dns/pull/279 [17:56:39] [url] Page not found · GitHub · GitHub | github.com [17:57:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.76, 9.55, 10.10 [17:57:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.19, 2.54, 2.09 [17:58:53] [02ssl] 07Reception123 closed pull request 03#531: Remove stardewmodding.wiki cert - 13https://github.com/miraheze/ssl/pull/531 [17:58:54] [url] Page not found · GitHub · GitHub | github.com [17:58:55] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-2/±2] 13https://github.com/miraheze/ssl/compare/8d36d843f4b7...aa82012fe3a6 [17:58:55] [url] Comparing 8d36d843f4b7...aa82012fe3a6 · miraheze/ssl · GitHub | github.com [17:58:56] [02miraheze/ssl] 07Universal-Omega 03aa82012 - Remove stardewmodding.wiki cert (#531) [17:59:29] [02puppet] 07Reception123 closed pull request 03#2591: Remove `puppetmaster` from hieradata - 13https://github.com/miraheze/puppet/pull/2591 [17:59:29] ... [17:59:31] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/157a3b8b8ca8...cb84ba32a642 [17:59:31] [url] Comparing 157a3b8b8ca8...cb84ba32a642 · miraheze/puppet · GitHub | github.com [17:59:32] [02miraheze/puppet] 07Universal-Omega 03cb84ba3 - Remove `puppetmaster` from hieradata (#2591) [18:03:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.13, 4.37, 3.96 [18:05:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.85, 2.00 [18:07:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.61, 2.24, 2.13 [18:11:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.24, 3.86, 3.91 [18:12:31] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.02, 1.71, 1.96 [18:13:12] [02dns] 07Reception123 closed pull request 03#279: Remove stardewmodding.wiki zone - 13https://github.com/miraheze/dns/pull/279 [18:13:12] [url] Page not found · GitHub · GitHub | github.com [18:13:13] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/f44e8cd7c67d...ccf401489781 [18:13:14] [url] Comparing f44e8cd7c67d...ccf401489781 · miraheze/dns · GitHub | github.com [18:13:15] [02miraheze/dns] 07Universal-Omega 03ccf4014 - Remove stardewmodding.wiki zone (#279) [18:14:28] [02puppet] 07JohnFLewis reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r882945411 [18:14:28] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [18:14:51] [02puppet] 07JohnFLewis reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r882945737 [18:14:52] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [18:16:11] [02puppet] 07Reception123 reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r882946756 [18:16:11] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [18:16:34] [02puppet] 07Reception123 synchronize pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [18:16:35] [url] Page not found · GitHub · GitHub | github.com [18:16:36] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/54afaaa11e5d...52c444acaf15 [18:16:37] [url] Comparing 54afaaa11e5d...52c444acaf15 · miraheze/puppet · GitHub | github.com [18:16:37] [02miraheze/puppet] 07Reception123 0352c444a - tab->space [18:16:55] [02puppet] 07Reception123 reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r882947404 [18:16:55] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [18:17:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.02, 3.77, 3.82 [18:17:17] [02miraheze/puppet] 07Reception123 pushed 0379 commits to 03Reception123-patch-3 [+8/-26/±226] 13https://github.com/miraheze/puppet/compare/52c444acaf15...4c5496edf571 [18:17:18] [url] Comparing 52c444acaf15...4c5496edf571 · miraheze/puppet · GitHub | github.com [18:17:19] [02miraheze/puppet] 07Reception123 034c5496e - Merge branch 'master' into Reception123-patch-3 [18:17:20] [02puppet] 07Reception123 synchronize pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [18:17:21] [url] Page not found · GitHub · GitHub | github.com [18:17:55] [02puppet] 07JohnFLewis reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r882948198 [18:17:56] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [18:19:20] [02puppet] 07Reception123 synchronize pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [18:19:21] [url] Page not found · GitHub · GitHub | github.com [18:19:22] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4c5496edf571...5a571a27625e [18:19:22] [url] Comparing 4c5496edf571...5a571a27625e · miraheze/puppet · GitHub | github.com [18:19:23] [02miraheze/puppet] 07Reception123 035a571a2 - Update common.yaml [18:20:31] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.97, 2.83, 2.26 [18:21:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.59, 3.73 [18:21:12] Reception123: puppet111 is still assigned 8203 though [18:22:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.92, 10.38, 9.89 [18:22:26] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.70, 3.71, 3.19 [18:23:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.82, 3.80 [18:23:33] JohnLewis: my bad, why do the simple things always get me [18:23:41] [02puppet] 07Reception123 synchronize pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [18:23:41] [url] Page not found · GitHub · GitHub | github.com [18:23:42] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5a571a27625e...716f317bf4d7 [18:23:43] [url] Comparing 5a571a27625e...716f317bf4d7 · miraheze/puppet · GitHub | github.com [18:23:44] otherwise, is the rest ok? [18:23:44] [02miraheze/puppet] 07Reception123 03716f317 - last fix [18:23:58] I didn't expect the nginx stuff to be right really, so it's surprising basically everything other than that was wrong but that seemingly wasn't [18:24:37] :P [18:24:58] only the comment by UO for nginx as we don't serve landing on puppet111 [18:25:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.56, 3.70 [18:26:31] JohnLewis: actually, will the script not work though if the req.backend isn't set for puppet111? if so I guess I should set that to puppet111 and then if something happens I can set it back to mwtask111 until it's figured out [18:27:41] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [18:28:07] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/assignImportedEdits.php --wiki=obeymewiki --import-prefix=wikia:obey-me --from=LuckyFluffy (END - exit=0) [18:28:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:30:35] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.66, 3.51, 3.95 [18:30:53] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.65, 10.18, 10.07 [18:31:02] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.20, 3.35, 3.26 [18:31:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.48, 3.93, 3.80 [18:34:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.40, 1.69, 1.94 [18:34:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.09, 11.81, 10.75 [18:35:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.85, 3.79 [18:36:48] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.40, 11.72, 10.85 [18:38:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.04, 3.47, 3.70 [18:38:32] RECOVERY - monarchists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'monarchists.wiki' will expire on Wed 24 Aug 2022 16:50:46 GMT +0000. [18:39:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 3.82, 3.77 [18:39:42] Reception123: yeah, if the script is ran on puppet111, web requests need to go there [18:40:38] ah, then I'll have to add it and worst case if something goes wrong I can re-add mwtask [18:40:47] is there still anything in need of correction? [18:41:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.62, 3.69 [18:41:34] Reception123: My comment about landing. [18:42:33] CosmicAlpha: ah yes my bad, forgot about that one [18:43:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.48, 10.19, 9.59 [18:43:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.12, 4.58, 4.01 [18:44:31] CosmicAlpha: though for /srv/mediawiki I would just create an empty dir right? I don't imagine there would be a need to clone it too for no reason right? [18:44:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.33, 1.83, 1.82 [18:45:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.83, 3.80 [18:45:30] Reception123: not for no reason, if web root is pointing to landing, then it needs /srv/mediawiki/landing, but puppet111 is not a MediaWiki server, so to me /srv/mediawiki on there does not make to much sense either. [18:46:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.99, 10.17, 9.24 [18:46:26] CosmicAlpha: ah yeah, what I meant is that we wouldn't actually clone and mediawiki stuff [18:46:31] I'd just mkdir mediawiki and that's it [18:46:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.49, 1.70, 1.78 [18:47:14] I don't know if that'd work, at the very least if pointing to landing, then it needs /srv/mediawiki/landing I think. [18:47:40] CosmicAlpha: I guess it's easier if I just do this then [18:47:42] https://www.irccloud.com/pastebin/fp8SY687/ [18:48:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.59, 3.82 [18:48:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.79, 10.59, 9.74 [18:48:18] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.07, 9.51, 9.13 [18:48:34] If you also add something to ensure /srv/mediawiki exists, and not just a manual mkdir, so it does not fail puppet for new servers, etc... [18:48:39] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.06, 11.26, 10.92 [18:49:17] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.45, 3.61 [18:50:04] CosmicAlpha: I know with a file it would be file { then ensure present but does that work with a dir? [18:50:12] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.84, 9.89, 9.58 [18:50:15] ensure directory. [18:50:24] but still using file [18:50:42] I see, thanks! [18:51:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.36, 3.55 [18:52:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.26, 4.10, 3.91 [18:52:23] Reception123: you could also use ensure_directory('present') I think. but just ensure => directory would probably be better in this case. [18:52:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.54, 11.44, 11.07 [18:52:38] * ensure => ensure_directory('present') [18:52:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.89, 1.34, 1.60 [18:53:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.32, 11.35, 10.38 [18:53:09] would still need file, etc... so ensure => directory is probably simpler. [18:53:22] yeah... [18:53:26] so would this be fine? [18:53:28] https://www.irccloud.com/pastebin/O5XUtO0O/ [18:55:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.25, 3.56, 3.56 [18:56:00] theoretically, but not that much spacing before => probably, and also timeout may not be needed here. [18:56:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.35, 3.91, 3.88 [18:57:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.36, 10.93, 10.47 [18:57:15] Thanks, will fix and add then [18:58:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.30, 4.43, 4.08 [18:58:16] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/716f317bf4d7...fe187e7665e1 [18:58:17] [url] Comparing 716f317bf4d7...fe187e7665e1 · miraheze/puppet · GitHub | github.com [18:58:18] [02miraheze/puppet] 07Reception123 03fe187e7 - require /srv/landing + /srv/mediawiki exist [18:58:19] [02puppet] 07Reception123 synchronize pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [18:58:20] [url] Page not found · GitHub · GitHub | github.com [18:58:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.61, 1.85, 1.75 [18:59:48] Reception123: left one more review [19:00:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.12, 1.99, 1.82 [19:02:06] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [19:02:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.94, 1.87, 1.79 [19:02:58] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.18, 9.41, 10.01 [19:04:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.67, 3.76, 3.91 [19:04:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.49, 11.46, 11.21 [19:04:47] Why are we pulling landing on puppet111? It shouldn’t be serving general traffic, only LE reqs [19:05:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.59, 3.62, 3.81 [19:06:06] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [19:08:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.56, 1.56, 1.68 [19:09:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.26, 4.13, 3.94 [19:10:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.90, 11.70, 11.43 [19:12:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.04, 4.27, 4.05 [19:15:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.01, 10.12, 9.55 [19:19:32] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.46, 9.00, 9.29 [19:24:12] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.79, 11.23, 11.14 [19:26:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.15, 10.55, 10.89 [19:26:15] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.93, 1.95, 1.77 [19:28:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.19, 1.79, 1.41 [19:28:26] JohnLewis: I think I did mention it wasn't a MediaWiki server so doing this MediaWiki stuff did not make much sense in this case, but I guess we wanted to go the route of using landing as the web root instead of changing that. [19:29:21] well what would the alternative be? [19:29:33] But it’s only accessible via the one url, so downloading landing is pointless to just be there [19:30:08] /var/www, /var/www/html, /var/www/challenges? Any of those probably [19:30:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.48, 1.34 [19:31:26] ah well we do already have alias /var/www/challenges/; [19:32:01] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.41, 1.62, 1.68 [19:34:15] Reception123: make it the webroot, or one of the others. [19:36:02] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [19:36:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.68, 11.28, 10.92 [19:38:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.64, 11.23, 10.94 [19:42:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [19:43:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.43, 3.90 [19:43:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.86, 11.50, 11.11 [19:46:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.30, 3.88 [19:48:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.81, 1.77, 1.99 [19:49:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.74, 11.91, 11.43 [19:50:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.33, 10.05, 9.32 [19:51:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.71, 3.38, 3.65 [19:51:45] CosmicAlpha: oh, where would that go? [19:52:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.79, 9.09, 9.04 [19:52:21] Reception123: replace the root that points to landing right now. (in nginx for letsencrypt) [19:53:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.37, 2.84, 3.41 [19:53:25] CosmicAlpha: ah so just replace that with root /var/www/challenges/ ? [19:53:42] I think so. [19:54:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.74, 2.06, 2.00 [19:55:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.19, 2.98, 3.39 [19:59:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.49, 8.64, 9.97 [20:00:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.38, 4.23, 3.82 [20:02:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.79, 3.71 [20:03:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.77, 10.02, 10.22 [20:05:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.30, 10.15, 9.62 [20:05:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.27, 11.47, 10.72 [20:06:09] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.31, 3.46, 3.07 [20:07:56] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [20:07:57] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:08:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.43, 2.99, 3.35 [20:08:03] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.70, 3.52, 3.14 [20:08:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.15, 1.76, 2.00 [20:09:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.17, 11.26, 10.86 [20:10:00] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.66, 3.99, 3.36 [20:12:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.92, 3.80, 3.36 [20:12:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.37, 3.27 [20:15:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.45, 10.81, 10.63 [20:16:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.56, 3.50 [20:17:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.91, 11.00, 10.74 [20:17:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.74, 3.08, 3.20 [20:18:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.71, 3.36, 3.30 [20:19:12] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.90, 9.93, 9.91 [20:19:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.48, 11.38, 10.90 [20:20:03] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:21:10] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:21:10] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:21:40] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.55, 3.74, 3.50 [20:22:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.65, 3.24, 3.40 [20:22:22] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:44] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.22, 1.37, 1.64 [20:23:13] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14656 bytes in 7.482 second response time [20:23:13] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14656 bytes in 7.198 second response time [20:23:44] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 6.668 second response time [20:24:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.83, 3.30, 3.35 [20:24:10] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14642 bytes in 2.860 second response time [20:24:22] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14650 bytes in 0.470 second response time [20:24:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.86, 9.64, 9.03 [20:26:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.16, 1.78, 1.73 [20:26:51] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.49, 9.44, 9.03 [20:27:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.61, 11.97, 11.46 [20:28:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.35, 1.90, 1.53 [20:28:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.77, 1.80, 1.75 [20:29:06] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [20:29:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 3.57, 3.40 [20:30:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.44, 3.29, 3.31 [20:30:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.07, 1.91, 1.79 [20:30:54] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.09, 10.08, 9.78 [20:31:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.87, 12.13, 11.63 [20:31:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.86, 2.93, 3.18 [20:32:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.96, 11.75, 10.23 [20:32:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.80, 1.81, 1.58 [20:32:52] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.07, 9.77, 9.71 [20:34:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.90, 10.91, 10.13 [20:34:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.62, 1.37, 1.44 [20:36:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.45, 3.08, 3.26 [20:36:39] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15103 MB (11% inode=99%); [20:37:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.68, 11.94, 11.78 [20:38:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.58, 1.98, 1.91 [20:40:01] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.26, 9.89, 9.89 [20:40:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.76, 2.02, 1.94 [20:41:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.83, 12.50, 12.02 [20:42:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.43, 3.16 [20:42:34] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.66, 11.02, 10.27 [20:42:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.47, 1.82, 1.88 [20:44:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.28, 3.26, 3.12 [20:44:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.83, 10.63, 10.23 [20:47:57] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.55, 10.39, 10.15 [20:48:00] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.71, 3.63, 3.32 [20:49:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.99, 10.43, 9.60 [20:49:55] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.75, 9.60, 9.88 [20:50:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.41, 3.55, 3.33 [20:50:24] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.06, 11.43, 10.70 [20:51:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.31, 10.09, 9.58 [20:52:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.53, 11.45, 10.80 [20:52:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.10, 2.03, 1.95 [20:53:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.75, 10.70, 10.30 [20:54:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.94, 3.30, 3.29 [20:55:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.89, 10.67, 9.96 [20:55:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.85, 10.04, 10.11 [20:56:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.34, 3.74, 3.32 [20:56:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.89, 1.91, 1.91 [21:00:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.87, 3.22, 3.26 [21:00:11] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.71, 9.68, 10.19 [21:01:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.69, 10.03, 9.96 [21:03:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.39, 10.43, 10.13 [21:04:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.15, 1.79, 1.81 [21:05:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.55, 9.56, 9.84 [21:06:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.53, 1.63, 1.75 [21:06:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.66, 4.04, 3.56 [21:07:48] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [21:08:19] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: DISK CRITICAL - free space: / 2340 MB (5% inode=96%); [21:08:26] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.83, 1.70, 1.47 [21:08:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.79, 3.58, 3.26 [21:08:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.90, 3.58 [21:09:23] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/64d1e320ea34 [21:09:25] [02miraheze/mediawiki] 07Universal-Omega 0364d1e32 - CdnCacheUpdate: remove http hack [21:09:26] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mediawiki [21:09:26] [url] Page not found · GitHub · GitHub | github.com [21:09:28] [02mediawiki] 07Universal-Omega opened pull request 03#7222: CdnCacheUpdate: remove http hack - 13https://github.com/miraheze/mediawiki/pull/7222 [21:09:28] [url] Page not found · GitHub · GitHub | github.com [21:09:50] [02mediawiki] 07Universal-Omega edited pull request 03#7222: CdnCacheUpdate: remove http hack - 13https://github.com/miraheze/mediawiki/pull/7222 [21:09:51] ... [21:10:23] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.85, 1.55 [21:10:44] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.15, 1.49, 1.67 [21:11:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.75, 11.15, 11.83 [21:11:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.57, 10.17, 9.17 [21:12:10] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.07, 10.41, 10.03 [21:12:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.48, 1.45 [21:13:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.70, 9.45, 9.03 [21:14:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.05, 10.26, 10.02 [21:16:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.01, 9.67, 9.83 [21:18:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.99, 3.72, 3.62 [21:20:18] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 3.88, 3.69 [21:21:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.93, 10.99, 11.18 [21:22:04] [02mediawiki] 07RhinosF1 commented on pull request 03#7222: CdnCacheUpdate: remove http hack - 13https://github.com/miraheze/mediawiki/pull/7222#issuecomment-1139071355 [21:22:05] [url] CdnCacheUpdate: remove http hack by Universal-Omega · Pull Request #7222 · miraheze/mediawiki · GitHub | github.com [21:22:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.83, 3.43, 3.54 [21:25:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.34, 11.15, 11.21 [21:26:04] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.26, 2.95, 3.33 [21:26:37] [02puppet] 07paladox reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r883096300 [21:26:39] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [21:28:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.22, 2.99, 3.32 [21:29:02] [02puppet] 07paladox reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r883097664 [21:29:03] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [21:29:28] [02puppet] 07paladox reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r883097889 [21:29:28] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [21:32:23] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.61, 10.99, 10.13 [21:34:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [21:34:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.02, 10.63, 10.10 [21:35:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.38, 3.98 [21:36:31] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: DISK CRITICAL - free space: / 2342 MB (5% inode=96%); [21:37:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.45, 9.24, 10.19 [21:37:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.14, 4.19, 4.20 [21:38:19] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.16, 9.97, 9.96 [21:40:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.86, 3.33, 3.23 [21:41:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.89, 10.32, 9.58 [21:42:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.52, 3.31 [21:43:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.25, 1.51, 1.92 [21:43:39] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.29, 10.02, 9.57 [21:44:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.16, 3.40, 3.29 [21:47:35] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.04, 1.65, 1.87 [21:49:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.50, 3.39, 3.91 [21:50:45] [02mediawiki] 07Universal-Omega commented on pull request 03#7222: CdnCacheUpdate: remove http hack - 13https://github.com/miraheze/mediawiki/pull/7222#issuecomment-1139092069 [21:50:46] [url] CdnCacheUpdate: remove http hack by Universal-Omega · Pull Request #7222 · miraheze/mediawiki · GitHub | github.com [21:51:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.86, 4.23, 4.17 [21:52:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.18, 10.03, 9.84 [21:53:32] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.38, 1.75, 1.87 [21:53:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.85, 3.74, 4.00 [21:54:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.36, 1.96, 1.47 [21:54:33] [02puppet] 07Universal-Omega reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r883111866 [21:54:35] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [21:55:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.92, 3.97, 4.07 [21:56:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.71, 1.87, 1.49 [21:58:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.62, 1.64, 1.44 [21:59:29] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.08, 1.38, 1.67 [22:00:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.88, 3.59, 3.31 [22:02:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.47, 3.87, 3.44 [22:03:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.46, 3.44, 3.15 [22:05:27] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.70, 3.20, 3.09 [22:05:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.61, 10.23, 9.72 [22:06:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.43, 3.62, 3.49 [22:06:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.19, 10.01, 10.11 [22:06:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 2.00, 1.70 [22:07:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.43, 3.47, 3.91 [22:08:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.81, 3.23, 3.35 [22:08:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.67, 1.61 [22:09:56] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.48, 9.64, 9.58 [22:09:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.09, 3.96, 4.03 [22:10:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.03, 1.72, 1.29 [22:12:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.80, 4.01, 3.63 [22:12:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.83, 1.85, 1.40 [22:13:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.19, 10.41, 10.19 [22:13:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.53, 3.24 [22:14:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.54, 3.50 [22:14:44] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.26, 1.68, 1.39 [22:15:23] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.88, 3.32, 3.19 [22:16:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.21, 3.18, 3.38 [22:16:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.49, 2.30, 1.88 [22:17:19] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 2.00, 1.75, 1.68 [22:18:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.37, 1.83, 1.75 [22:19:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.54, 1.77, 1.45 [22:19:18] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.45, 1.99, 1.78 [22:19:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.17, 10.55, 9.34 [22:20:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.84, 3.36, 3.07 [22:20:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.47, 3.25, 3.42 [22:20:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.14, 1.86, 1.76 [22:21:09] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.28, 1.59, 1.42 [22:22:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.17, 3.24, 3.05 [22:22:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.80, 3.88, 3.63 [22:22:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.13, 1.52, 1.64 [22:23:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.26, 10.28, 9.57 [22:24:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.84, 3.66 [22:24:39] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14596 MB (10% inode=99%); [22:24:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.19, 9.81, 10.10 [22:25:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.43, 4.10, 3.47 [22:25:42] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.70, 9.78, 9.47 [22:28:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.06, 4.42, 3.92 [22:28:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.38, 10.91, 10.51 [22:29:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.79, 3.47 [22:30:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.88, 3.49, 3.64 [22:31:08] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.99, 1.76, 1.56 [22:32:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.44, 3.24, 3.04 [22:33:08] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.89, 1.47, 1.48 [22:34:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.34, 2.91, 2.94 [22:35:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.99, 3.14, 3.26 [22:35:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.50, 3.97 [22:37:09] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.68, 1.98, 1.98 [22:38:06] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [22:39:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.72, 4.48, 4.21 [22:41:07] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.15, 2.05, 2.01 [22:43:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.54, 3.35 [22:43:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.57, 9.55, 8.47 [22:45:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.13, 3.31, 3.28 [22:45:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.90, 8.93, 8.38 [22:46:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.53, 3.09, 3.35 [22:52:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.93, 1.53, 1.31 [22:52:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.22, 2.45, 1.57 [22:54:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.65, 3.49, 3.37 [22:54:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.49, 3.88, 3.46 [22:54:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.43, 1.51, 1.33 [22:54:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.56, 9.34, 10.06 [22:54:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.84, 1.90, 1.47 [22:56:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.44, 3.02, 3.21 [22:56:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.65, 3.43 [22:58:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [22:58:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.69, 1.88, 1.54 [22:58:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.99, 1.47, 1.40 [23:00:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.69, 3.18, 3.29 [23:00:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.41, 1.61, 1.48 [23:06:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.45, 3.30 [23:06:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.33, 4.05, 3.67 [23:08:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.95, 3.51, 3.31 [23:08:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.51, 3.50 [23:10:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.82, 3.31, 3.25 [23:10:53] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.75, 1.87, 2.00 [23:12:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.98, 3.27, 3.25 [23:14:06] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.52, 2.87, 3.25 [23:16:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.22, 1.93, 1.98 [23:20:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.57, 3.39 [23:21:06] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.30, 2.14, 1.55 [23:22:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.79, 3.53, 3.27 [23:24:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.47, 4.14, 3.51 [23:25:06] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.31, 1.71, 1.51 [23:26:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.86, 3.88, 3.58 [23:26:45] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.18, 1.71, 1.93 [23:27:06] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.50, 1.25, 1.36 [23:28:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.66, 4.94, 3.98 [23:32:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.46, 3.88, 3.78 [23:34:42] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.10, 1.37, 1.68 [23:37:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.80, 2.99, 3.39 [23:38:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.67, 3.77 [23:43:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 3.56, 3.96 [23:47:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.89, 4.92, 4.38 [23:48:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.72, 4.15, 3.79 [23:48:34] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.45, 1.89, 1.68 [23:48:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 3.64, 3.38 [23:50:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.78, 3.70, 3.68 [23:52:32] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.54, 1.92, 1.76 [23:52:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.88, 3.50, 3.39 [23:54:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.10, 1.95, 1.79 [23:54:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.50, 4.53, 3.76 [23:56:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.13, 3.65, 3.65 [23:56:30] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.87, 1.78