[00:00:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.56, 3.64 [00:02:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.11, 10.16, 9.37 [00:02:45] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 11.03, 3.73, 2.08 [00:02:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.44, 3.58, 3.67 [00:04:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.35, 4.09, 3.82 [00:04:27] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.41, 1.59, 1.68 [00:04:31] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 10146 MB (25% inode=96%); [00:04:38] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.13, 9.78, 9.32 [00:04:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.33, 4.42, 3.95 [00:05:40] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 10043 MB (25% inode=96%); [00:08:26] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.74, 1.72 [00:10:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.38, 1.95, 1.80 [00:10:32] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.87, 1.59, 1.69 [00:22:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.98, 10.88, 10.09 [00:22:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.39, 3.35, 3.92 [00:24:23] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.95, 10.07, 9.88 [00:24:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.50, 4.17, 4.15 [00:26:39] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15033 MB (11% inode=99%); [00:28:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.31, 10.54, 10.11 [00:30:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 3.72, 3.96 [00:32:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.17, 3.47, 3.91 [00:32:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.93, 1.99 [00:32:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.58, 10.18, 10.07 [00:38:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.64, 4.52, 4.13 [00:38:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.60, 4.03, 3.94 [00:39:10] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.13, 1.90, 1.62 [00:40:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.95, 1.98 [00:40:41] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [00:41:07] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.95, 1.89, 1.64 [00:42:10] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.98, 1.93, 1.97 [00:42:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.96, 3.98 [00:43:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.57, 1.55 [00:44:09] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.18, 2.02, 2.01 [00:46:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.35, 3.62, 3.91 [00:54:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.31, 3.93, 3.88 [00:54:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.16, 3.63, 3.67 [00:56:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.34, 3.77, 3.84 [00:56:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.10, 3.09, 3.47 [00:58:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.95, 2.86, 3.35 [01:01:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.62, 3.16, 3.89 [01:03:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.20, 3.93, 4.08 [01:05:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.84, 3.73, 3.98 [01:05:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.46, 1.82, 1.97 [01:08:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.49, 2.85, 3.30 [01:13:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.12, 3.80, 3.81 [01:15:53] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.25, 1.35, 1.66 [01:17:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.02, 3.82, 3.86 [01:19:51] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.10, 1.75, 1.75 [01:21:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.82, 10.13, 8.54 [01:23:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.68, 9.75, 8.59 [01:23:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.22, 10.26, 9.54 [01:23:49] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.60, 1.79, 1.78 [01:25:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.26, 9.90, 9.50 [01:27:47] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.16, 1.48, 1.65 [01:27:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.32, 4.52, 3.93 [01:29:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.25, 3.02 [01:30:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.50, 3.85, 3.35 [01:32:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.66, 3.35 [01:32:39] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14598 MB (10% inode=99%); [01:33:30] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.36, 3.28, 3.10 [01:36:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.82, 2.98, 3.16 [01:39:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.31, 4.49, 3.61 [01:40:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.99, 4.08, 3.60 [01:40:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.95, 1.72, 1.66 [01:42:42] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.60, 1.66, 1.64 [01:43:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.91, 3.55 [01:45:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.98, 3.24, 3.36 [01:45:41] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [01:46:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.38, 3.66, 3.62 [01:50:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.55, 2.09, 1.82 [01:52:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.28, 2.99, 3.36 [01:53:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.74, 3.98 [01:55:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.49, 4.14, 4.08 [01:58:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.42, 4.33, 3.78 [02:00:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.70, 3.67, 3.61 [02:03:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.88, 3.40, 3.85 [02:04:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.77, 2.79, 3.27 [02:04:30] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.60, 1.93, 1.91 [02:06:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.80, 9.83, 9.13 [02:08:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.89, 9.35, 9.03 [02:09:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.71, 3.72, 3.76 [02:10:27] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.02, 1.85, 1.87 [02:12:26] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.90, 1.90, 1.89 [02:13:56] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [02:13:56] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:13:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.66, 3.71 [02:14:26] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.96, 1.92 [02:21:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.93, 3.02, 3.39 [02:36:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.85, 3.30, 3.06 [02:38:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.96, 3.61, 3.18 [02:38:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.88, 3.67, 3.50 [02:40:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.78, 3.55 [02:42:12] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.98, 1.96, 2.00 [02:42:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.62, 3.55, 3.27 [02:42:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.59, 3.71, 3.55 [02:44:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 4.17, 3.75 [02:46:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.91, 3.43, 3.23 [02:46:10] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.39, 2.06, 2.03 [02:46:23] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.60, 3.33, 3.27 [02:48:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.45, 3.09, 3.13 [02:54:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.99, 3.95, 3.81 [02:56:06] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.36, 1.77, 1.96 [03:02:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 4.35, 4.00 [03:02:43] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.48, 3.22 [03:04:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.21, 1.40, 1.70 [03:04:38] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.77, 3.20, 3.15 [03:06:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.49, 3.74, 3.87 [03:17:54] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.39, 1.75, 1.61 [03:19:53] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.99, 1.84, 1.66 [03:19:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.43, 4.39, 3.91 [03:20:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.52, 3.69, 3.07 [03:21:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.60, 3.14 [03:21:52] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.47, 2.29, 1.84 [03:22:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 4.00, 3.81, 3.27 [03:24:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.50, 3.13 [03:24:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.34, 3.24, 3.13 [03:25:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.08, 3.78, 3.83 [03:26:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.18, 3.23, 3.09 [03:27:49] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.49, 1.87, 1.82 [03:31:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.73, 3.41, 3.54 [03:33:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.36, 3.51 [03:35:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.32, 3.53, 3.54 [03:39:43] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.11, 1.77, 1.76 [03:41:42] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.99, 1.86, 1.80 [03:41:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.47, 3.53 [03:43:42] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.92, 1.83 [03:43:43] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [03:43:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.61, 3.72, 3.21 [03:43:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.69, 4.31, 3.81 [03:45:05] !log ban req.http.Host == static.miraheze.org && req.url ~ "^/legendsdatacronwiki/c/cd/DatacronLogo2021.svg" on cp* [03:45:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:47:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.67, 4.86, 3.76 [03:50:12] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.86, 10.49, 9.57 [03:57:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.94, 3.86 [04:01:35] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.94, 1.99 [04:02:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.85, 9.73, 9.89 [04:02:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.53, 3.84 [04:04:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.06, 3.89, 3.94 [04:07:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.65, 3.75, 3.65 [04:07:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.52, 10.43, 10.10 [04:09:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.24, 1.98, 2.00 [04:09:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.77, 9.47, 9.80 [04:11:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.99, 1.95, 1.99 [04:11:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.03, 3.33, 3.55 [04:12:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.26, 3.50, 3.90 [04:13:38] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [04:13:38] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:16:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.55, 3.81 [04:17:28] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.46, 2.12, 2.04 [04:18:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.49, 3.75 [04:19:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.29, 4.67, 3.95 [04:20:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 9.05, 5.28, 4.35 [04:31:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.65, 1.84, 1.99 [04:43:23] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.16, 1.34, 1.64 [04:43:33] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [04:51:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.73, 1.68 [04:51:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.94, 3.18, 3.85 [04:53:23] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.37, 1.61, 1.64 [04:53:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.75, 3.95, 4.02 [05:03:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.63, 3.61, 3.97 [05:06:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.19, 2.94, 3.78 [05:13:29] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [05:15:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.56, 2.87, 3.34 [05:16:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.61, 3.68, 3.69 [05:18:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.88, 3.39, 3.58 [05:19:14] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.75, 2.16, 1.79 [05:23:12] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.70, 1.99, 1.82 [05:24:21] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/ccf401489781...8d002025e5fe [05:24:21] [url] Comparing ccf401489781...8d002025e5fe · miraheze/dns · GitHub | github.com [05:24:22] [02miraheze/dns] 07Universal-Omega 038d00202 - Remove old acme challenges for betaheze (#278) [05:24:24] [02dns] 07Reception123 closed pull request 03#278: Remove old acme challenges for betaheze - 13https://github.com/miraheze/dns/pull/278 [05:24:24] [url] Page not found · GitHub · GitHub | github.com [05:24:47] [02dns] 07Reception123 closed pull request 03#277: Delete forecasting.wiki zone - 13https://github.com/miraheze/dns/pull/277 [05:24:48] [url] Page not found · GitHub · GitHub | github.com [05:24:49] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/8d002025e5fe...129d8c028cc0 [05:24:49] [url] Comparing 8d002025e5fe...129d8c028cc0 · miraheze/dns · GitHub | github.com [05:24:50] [02miraheze/dns] 07Universal-Omega 03129d8c0 - Delete forecasting.wiki zone (#277) [05:25:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.23, 2.10, 1.88 [05:26:21] [02puppet] 07Reception123 synchronize pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [05:26:21] [url] Page not found · GitHub · GitHub | github.com [05:26:23] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fe187e7665e1...e73149535637 [05:26:23] [url] Comparing fe187e7665e1...e73149535637 · miraheze/puppet · GitHub | github.com [05:26:24] [02miraheze/puppet] 07Reception123 03e731495 - Update init.pp [05:26:49] [02puppet] 07Reception123 synchronize pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [05:26:49] [url] Page not found · GitHub · GitHub | github.com [05:26:50] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e73149535637...157f1c00248a [05:26:51] [url] Comparing e73149535637...157f1c00248a · miraheze/puppet · GitHub | github.com [05:26:52] [02miraheze/puppet] 07Reception123 03157f1c0 - Update nginx.conf.erb [05:27:10] [02puppet] 07Reception123 synchronize pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [05:27:10] [url] Page not found · GitHub · GitHub | github.com [05:27:12] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/157f1c00248a...d09c6e85206b [05:27:12] [url] Comparing 157f1c00248a...d09c6e85206b · miraheze/puppet · GitHub | github.com [05:27:13] [02miraheze/puppet] 07Reception123 03d09c6e8 - Update default.vcl [05:27:18] [02puppet] 07Reception123 reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r882886344 [05:27:18] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [05:28:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.29, 3.43, 3.50 [05:30:07] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d09c6e85206b...55e5a87c30a2 [05:30:07] [url] Comparing d09c6e85206b...55e5a87c30a2 · miraheze/puppet · GitHub | github.com [05:30:08] [02miraheze/puppet] 07Reception123 0355e5a87 - Update puppet111.yaml [05:30:10] [02puppet] 07Reception123 synchronize pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [05:30:10] [url] Page not found · GitHub · GitHub | github.com [05:31:08] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.70, 1.94, 1.89 [05:32:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 3.85, 3.66 [05:36:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 3.50, 3.54 [05:38:25] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.49, 3.26 [05:39:04] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.21, 1.85, 1.84 [05:42:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.56, 3.46, 3.58 [05:42:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.89, 3.34, 3.27 [05:43:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.90, 1.93, 1.89 [05:46:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.46, 4.53, 3.97 [05:47:00] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.29, 1.99, 1.92 [05:48:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.30, 4.20, 3.67 [05:50:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.82, 3.59 [05:56:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.11, 3.74, 3.62 [06:00:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.82, 3.82, 3.75 [06:02:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 4.15, 3.88 [06:07:47] [02puppet] 07Reception123 commented on pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592#issuecomment-1139312771 [06:07:48] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [06:09:27] [02puppet] 07Universal-Omega commented on pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592#issuecomment-1139313788 [06:09:28] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [06:12:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.21, 3.94, 3.96 [06:22:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 3.62, 3.94 [06:22:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.12, 3.58, 3.71 [06:24:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.99, 3.77, 3.78 [06:26:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.99, 3.98, 4.02 [06:26:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.78, 4.69, 4.12 [06:32:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.30, 3.50, 3.90 [06:32:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.44, 3.62, 3.93 [06:33:05] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/55e5a87c30a2...08222e89a896 [06:33:06] [url] Comparing 55e5a87c30a2...08222e89a896 · miraheze/puppet · GitHub | github.com [06:33:07] [02miraheze/puppet] 07Reception123 0308222e8 - add ssl-cert to puppet-users [06:33:08] [02puppet] 07Reception123 synchronize pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [06:33:09] [url] Page not found · GitHub · GitHub | github.com [06:34:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.13, 3.79, 3.96 [06:36:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.05, 3.51, 3.84 [06:38:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.05, 3.80, 3.85 [06:40:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.50, 3.73 [06:42:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 3.87, 3.86 [06:42:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.63, 3.74, 3.78 [06:44:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.37, 3.62, 3.79 [06:44:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.62, 3.31, 3.62 [06:46:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.64, 3.84, 3.83 [06:46:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.22, 3.69, 3.71 [06:48:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.71, 3.82, 3.83 [06:48:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.92, 3.83, 3.76 [06:54:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.33, 2.59, 3.28 [06:57:07] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.21, 1.82, 1.29 [06:58:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.79, 4.01, 3.74 [06:58:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 4.01, 3.73 [06:59:03] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.29, 1.65, 1.29 [07:00:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.49, 3.00, 3.40 [07:00:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.06, 3.24, 3.48 [07:02:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.80, 2.79, 3.28 [07:08:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.35, 3.59, 3.44 [07:10:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.64, 3.25, 3.34 [07:13:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.41, 3.48, 3.99 [07:15:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.16, 3.81, 4.04 [07:17:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.38, 3.31, 3.83 [07:19:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.62, 3.88 [07:21:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.60, 3.65, 3.85 [07:23:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.89, 4.10, 3.99 [07:25:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.71, 3.55, 3.80 [07:35:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.46, 3.03, 3.39 [07:41:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.34, 3.41 [07:45:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.08, 2.93, 3.25 [08:11:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.71, 3.73, 3.36 [08:13:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.80, 3.42 [08:15:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.26, 3.40, 3.34 [08:33:53] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.60, 1.81, 1.65 [08:37:45] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.13, 1.63, 1.63 [08:41:37] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.25, 1.92, 1.74 [08:42:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.77, 2.05, 1.64 [08:44:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.34, 1.81, 1.60 [08:46:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.96, 1.47, 1.50 [08:49:19] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.93, 1.94, 1.82 [08:51:15] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.24, 1.89, 1.81 [08:58:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.70, 3.75, 3.29 [08:58:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.46, 2.03, 1.67 [09:00:05] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.71, 3.34, 3.19 [09:03:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.04, 3.62, 3.38 [09:04:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.71, 1.61, 1.64 [09:04:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.23, 1.86, 1.98 [09:06:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.49, 2.10, 2.05 [09:07:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.91, 3.54 [09:08:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.70, 1.74, 1.68 [09:08:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.62, 1.87, 1.97 [09:10:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.44, 1.61, 1.64 [09:12:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.64, 3.58, 3.06 [09:12:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.95, 3.65 [09:12:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.58, 2.23, 2.09 [09:13:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.42, 3.05 [09:13:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.89, 4.78, 4.00 [09:15:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.66, 3.70, 3.19 [09:17:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.67, 3.16, 3.04 [09:23:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.47, 3.30, 3.82 [09:27:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.22, 2.57, 3.39 [09:34:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.25, 3.14, 2.88 [09:35:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.80, 1.83, 1.98 [09:36:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.05, 2.98, 2.84 [09:40:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.19, 1.57, 1.94 [09:47:23] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.16, 1.75, 1.82 [09:50:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.45, 1.81, 1.86 [09:52:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.58, 1.62, 1.78 [09:53:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.43, 1.82, 1.86 [09:54:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.53, 1.90, 1.86 [09:59:23] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.39, 1.51, 1.70 [10:05:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.73, 1.70, 1.73 [10:05:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.30, 8.84, 7.18 [10:06:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.25, 1.90, 2.00 [10:07:23] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.32, 1.61, 1.69 [10:07:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.94, 8.32, 7.18 [10:13:23] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.77, 1.71 [10:15:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.70, 1.76, 1.72 [10:18:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.56, 2.04, 1.94 [10:19:23] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.69, 2.10, 1.86 [10:21:58] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.87, 1.79, 1.60 [10:23:53] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.52, 2.92, 2.07 [10:25:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.88, 1.87 [10:30:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.28, 2.73 [10:30:43] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.96, 3.08, 2.65 [10:32:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.14, 2.86, 2.64 [10:32:38] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.92, 2.64, 2.55 [10:35:27] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.49, 1.80, 1.96 [10:37:23] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.06, 1.79, 1.77 [10:40:41] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [10:42:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.94, 1.57, 1.93 [10:45:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.88, 1.86 [10:46:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.46, 1.69, 1.87 [10:47:00] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.22, 1.52, 1.69 [10:47:23] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.48, 2.02, 1.90 [10:48:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.44, 1.63, 1.83 [10:52:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.01, 1.80, 1.85 [10:52:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.12, 4.18, 3.46 [10:52:49] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.86, 1.66, 1.68 [10:56:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.08, 4.00, 3.55 [10:56:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.94, 1.97, 1.90 [10:58:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.80, 3.52 [10:58:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.08, 1.56, 1.66 [11:00:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.92, 3.15, 3.32 [11:02:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.02, 1.69, 1.76 [11:04:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.39, 1.74, 1.66 [11:04:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.46, 1.58, 1.72 [11:06:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.70, 1.64, 1.63 [11:08:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.33, 1.85, 1.79 [11:10:44] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.13, 1.57, 1.70 [11:14:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.60, 2.19, 1.91 [11:18:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.38, 1.85, 1.84 [11:20:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.38, 2.37, 2.06 [11:27:16] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.78, 1.75, 1.62 [11:30:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 3.57, 3.23 [11:30:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.19, 1.67, 1.95 [11:31:07] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.30, 1.68, 1.64 [11:32:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.13, 3.34, 3.18 [11:33:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.70, 1.96 [11:41:45] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.94, 1.88, 1.71 [11:42:44] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.39, 1.47, 1.68 [11:43:41] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.09, 1.59, 1.63 [11:44:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.00, 3.65, 3.26 [11:45:23] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.35, 1.40, 1.66 [11:46:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.68, 3.19, 3.14 [11:48:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.90, 1.75, 1.72 [11:49:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.71, 1.96, 1.82 [11:49:29] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.73, 1.83, 1.72 [11:50:44] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.57, 1.68, 1.70 [11:51:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.80, 1.78 [11:51:24] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.54, 2.27, 1.90 [11:51:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.65, 3.96, 3.47 [11:53:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.67, 3.43 [11:54:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.70, 2.15, 1.89 [11:59:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.56, 3.13, 3.35 [12:03:23] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.59, 1.60, 1.69 [12:06:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.74, 10.74, 9.54 [12:07:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.56, 10.45, 9.23 [12:08:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.30, 9.88, 9.38 [12:11:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.39, 11.00, 9.70 [12:12:19] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.69, 1.71 [12:12:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.65, 10.53, 9.76 [12:13:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.22, 10.85, 9.82 [12:14:18] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.43, 1.61, 1.68 [12:14:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.21, 1.72, 1.95 [12:18:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.45, 2.00, 1.82 [12:19:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.35, 3.21 [12:20:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.55, 1.94, 1.96 [12:22:22] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.90, 3.74, 3.34 [12:22:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.47, 11.96, 10.84 [12:22:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.91, 1.55, 1.81 [12:23:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.69, 3.37 [12:24:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.96, 3.38, 3.04 [12:26:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.94, 3.23, 3.03 [12:26:09] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.84, 3.29, 3.24 [12:26:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.01, 2.04, 1.93 [12:27:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.71, 3.82, 3.49 [12:29:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.43, 3.20, 3.30 [12:30:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.60, 1.99, 1.95 [12:33:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.03, 9.26, 9.88 [12:35:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 3.63, 3.45 [12:36:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.38, 1.08, 1.56 [12:37:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.73, 2.94, 3.22 [12:40:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.34, 11.24, 11.62 [12:40:41] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [12:40:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.23, 1.63, 1.71 [12:40:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.22, 4.96, 4.02 [12:42:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.24, 9.69, 8.81 [12:42:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.12, 1.55, 1.67 [12:44:15] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.58, 9.72, 8.93 [12:45:41] alerting : [FIRING:1, RESOLVED:1] (mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [12:46:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.45, 1.68, 1.71 [12:47:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.53, 3.14 [12:48:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.36, 2.99 [12:49:11] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.86, 3.33, 3.12 [12:50:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.28, 3.33, 3.03 [12:50:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 4.00, 4.00 [12:50:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.99, 1.50, 1.64 [12:51:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.52, 10.33, 9.82 [12:53:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.82, 3.58, 3.25 [12:53:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 11.27, 11.24 [12:54:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 4.10, 4.04 [12:54:58] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.39, 3.16, 3.13 [12:55:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.24, 9.88, 9.79 [12:55:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.54, 10.20, 10.83 [12:56:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.78, 3.93 [12:57:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.79, 11.23, 11.14 [12:58:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.40, 2.25, 1.89 [12:59:23] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.46, 10.64, 9.38 [12:59:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.47, 10.58, 10.06 [12:59:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.52, 11.50, 11.26 [13:00:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.96, 1.77, 1.76 [13:01:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.31, 9.90, 9.86 [13:03:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.56, 10.25, 9.57 [13:05:23] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.17, 10.03, 9.58 [13:07:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.29, 2.96 [13:08:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.57, 3.61 [13:08:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.87, 1.44, 1.65 [13:09:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.90, 3.16, 2.95 [13:10:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.14, 3.44 [13:11:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.01, 10.42, 9.85 [13:12:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 3.46, 3.52 [13:14:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.74, 1.77, 1.75 [13:15:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.71, 3.44, 3.20 [13:15:23] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 17.03, 12.06, 10.54 [13:15:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.70, 8.35, 9.91 [13:16:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.22, 3.93, 3.74 [13:16:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.52, 10.56, 9.71 [13:16:15] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.69, 3.46, 3.11 [13:18:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.26, 1.56, 1.68 [13:19:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.09, 3.09, 3.09 [13:19:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.43, 11.64, 10.92 [13:19:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.68, 3.65 [13:20:02] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.35, 3.39, 3.15 [13:20:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.68, 9.90, 9.67 [13:21:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.55, 3.32, 3.53 [13:22:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.25, 1.63, 1.95 [13:22:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.97, 1.85, 1.76 [13:23:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.25, 9.96, 9.88 [13:25:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.74, 9.71, 9.79 [13:25:43] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.44, 1.77, 1.96 [13:26:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.53, 2.26, 1.91 [13:26:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.19, 1.77, 1.92 [13:27:23] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.59, 9.54, 10.15 [13:27:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.34, 3.13, 3.39 [13:28:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.35, 1.58, 1.83 [13:29:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.16, 10.39, 10.07 [13:30:00] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.29, 3.74, 3.36 [13:31:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.58, 10.08, 10.00 [13:33:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.17, 10.11, 10.23 [13:33:39] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.38, 1.38, 1.67 [13:34:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.91, 3.35, 3.28 [13:34:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.49, 1.99, 1.95 [13:35:23] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.51, 9.43, 9.98 [13:38:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.01, 1.85, 1.81 [13:41:34] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.56, 1.61 [13:41:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 4.03, 3.62 [13:42:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.31, 1.44, 1.69 [13:44:26] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.76, 1.86, 1.29 [13:44:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.32, 1.85, 1.86 [13:45:32] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.29, 1.50, 1.59 [13:46:00] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.34, 3.83, 3.47 [13:46:23] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.95, 1.48, 1.21 [13:46:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.23, 2.19, 1.91 [13:47:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.73, 3.68 [13:48:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.39, 2.16, 1.95 [13:49:30] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.79, 2.08, 1.79 [13:50:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.39, 3.68, 3.50 [13:52:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.84, 1.99, 1.92 [13:54:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.97, 10.58, 10.18 [13:54:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.07, 2.47, 2.11 [13:55:27] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.50, 1.85, 1.81 [13:56:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.24, 3.22, 3.33 [13:56:21] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.14, 10.99, 10.37 [13:58:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.72, 11.22, 10.54 [13:59:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.22, 3.94, 3.72 [14:02:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 11.26, 10.67 [14:02:47] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.75, 3.49, 3.39 [14:04:41] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.27, 3.38, 3.36 [14:06:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.99, 1.65, 1.95 [14:07:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.76, 3.73 [14:08:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.36, 11.69, 11.09 [14:11:23] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.22, 2.01, 1.85 [14:11:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 4.04, 3.87 [14:15:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.82, 1.90, 1.84 [14:16:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.36, 1.45, 1.70 [14:17:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.44, 3.23, 3.23 [14:17:32] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1040 bytes in 0.840 second response time [14:17:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.03, 3.76, 3.84 [14:18:39] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) [14:18:54] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.85, 3.09, 3.18 [14:19:23] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.47, 2.10, 1.92 [14:20:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.26, 9.18, 10.19 [14:21:19] paladox, Reception123: db crash [14:21:28] !sre [14:21:51] Hmm that's no good. I've got no access until tonight [14:21:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.70, 4.11, 3.96 [14:22:08] Maybe it’ll fix itself? [14:22:11] Maybe in around 2 hours [14:22:19] Zppix: let's hope, usually does [14:22:31] Reception123: should I get John [14:22:33] !log restart db121 [14:22:33] .op [14:22:34] Attempting to OP... [14:22:39] No paladox is here [14:22:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:23:08] well i restarted mariadb not the machine... i'll fix the log [14:23:18] paladox: is it coming back? [14:23:22] Can we check cause? [14:23:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.86, 3.88 [14:23:57] it's restarting... [14:24:15] ok restarted now [14:24:15] paladox: please check logs for a cause while it does [14:24:18] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.74, 3.16, 1.87 [14:24:39] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 132 Threads: 50 Questions: 7460 Slow queries: 381 Opens: 1470 Open tables: 1464 Queries per second avg: 56.515 [14:24:46] Think that's quite obvious... [14:24:55] Thanks for fixing paladox ! [14:25:00] it'll be OOM [14:25:02] May 27 14:15:20 db121 kernel: [4295171.525323] mariadbd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [14:25:06] like it does all the time [14:25:17] Should download more ram [14:25:17] i checked the disk as soon as i logged in to make sure it wasn't that [14:25:32] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 52093 bytes in 1.324 second response time [14:25:50] Ok [14:25:55] Thanks paladox [14:25:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.53, 4.40, 4.07 [14:25:59] yw [14:26:18] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 1.50, 2.73, 1.89 [14:26:30] I'm walking home [14:26:45] Send a text if anything needed in next 10 minutes [14:27:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.93, 1.99, 1.97 [14:27:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.70, 3.33, 3.01 [14:27:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.57, 9.71, 9.74 [14:28:33] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 3.91, 3.32 [14:29:23] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.02, 1.98, 1.97 [14:29:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.67, 3.27, 3.04 [14:29:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.13, 8.96, 9.47 [14:32:09] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.79, 3.28, 3.06 [14:32:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.25, 1.79, 1.99 [14:33:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.85, 1.95, 1.97 [14:33:30] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.35, 4.13, 3.47 [14:34:03] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.71, 3.08, 3.01 [14:34:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.87, 3.80, 3.48 [14:34:44] Home [14:34:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.02, 1.87, 1.99 [14:35:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.52, 3.51, 3.32 [14:35:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.15, 9.85, 9.67 [14:36:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.71, 1.73, 1.92 [14:37:23] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.41, 2.03, 1.99 [14:37:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.85, 9.18, 9.46 [14:39:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.36, 3.81, 3.47 [14:40:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.00, 3.24, 3.33 [14:40:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.09, 1.88, 1.94 [14:41:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.99, 1.99 [14:41:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.44, 3.38 [14:42:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.35, 1.69, 1.86 [14:43:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.77, 3.79, 3.50 [14:44:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.65, 4.07, 3.61 [14:44:40] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.97, 3.47, 3.25 [14:45:23] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.16, 2.01, 2.00 [14:45:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.80, 3.60, 3.48 [14:45:41] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [14:46:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.63, 3.42, 3.43 [14:47:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.82, 3.57 [14:48:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.86, 2.14, 1.94 [14:49:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.14, 3.28, 3.41 [14:50:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.23, 1.74, 1.82 [14:52:15] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.25, 3.82, 3.50 [14:52:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.93, 2.83, 2.26 [14:53:25] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.33, 3.00, 3.29 [14:54:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.31, 3.02, 3.31 [14:54:09] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.11, 3.54, 3.44 [14:56:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.65, 2.76, 1.91 [14:58:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.99, 3.17, 3.30 [14:59:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.42, 1.75, 1.96 [14:59:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.44, 3.40 [15:00:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.26, 1.99, 1.79 [15:01:23] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.29, 3.25, 3.33 [15:01:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.52, 3.30, 3.92 [15:02:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.08, 1.97, 1.80 [15:04:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.44, 1.71, 1.72 [15:07:23] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.27, 1.34, 1.66 [15:08:41] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.08, 3.45, 3.37 [15:08:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.69, 1.57, 1.65 [15:09:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.07, 3.77, 3.86 [15:11:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.77, 3.85 [15:12:05] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [15:12:28] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.46, 3.87, 3.54 [15:15:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.09, 4.11, 3.93 [15:17:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.37, 3.51, 3.17 [15:17:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.67, 4.11, 3.48 [15:18:09] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.41, 3.77, 3.65 [15:19:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.11, 3.29, 3.13 [15:19:23] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.65, 1.62 [15:19:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.93, 3.64, 3.38 [15:20:03] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 5.22, 4.15, 3.79 [15:20:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.23, 1.64, 1.91 [15:21:21] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.46, 3.27, 3.28 [15:21:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.62, 1.65, 1.62 [15:21:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.81, 3.88 [15:23:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.67, 4.44, 4.09 [15:24:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.33, 3.75, 3.71 [15:24:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.60, 1.96, 1.97 [15:25:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.94, 3.95 [15:26:00] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.84, 3.92, 3.77 [15:26:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.82, 1.93, 1.68 [15:27:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 3.93, 3.93 [15:28:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.54, 3.60, 3.66 [15:28:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.69, 1.82, 1.67 [15:28:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.79, 1.97, 1.98 [15:29:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.83, 3.84, 3.90 [15:30:00] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.17, 3.72, 3.69 [15:30:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.02, 3.50, 3.22 [15:32:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.77, 3.35, 3.56 [15:32:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.49, 3.04, 3.08 [15:33:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.83, 3.86 [15:34:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.30, 2.99, 3.40 [15:34:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.60, 1.64, 1.63 [15:38:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 10.44, 5.35, 3.16 [15:38:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.54, 2.42, 1.93 [15:40:04] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [15:40:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.47, 4.41, 3.58 [15:40:35] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.14, 4.63, 3.75 [15:42:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.85, 3.85, 3.55 [15:42:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.95, 1.88, 1.85 [15:44:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.78, 3.50 [15:46:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.89, 3.56 [15:46:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.91, 1.42, 1.67 [15:48:09] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.82, 3.83, 3.58 [15:48:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.66, 3.17, 3.35 [15:50:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.77, 2.24, 1.84 [15:50:22] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 5.07, 4.22, 3.63 [15:50:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.69, 10.13, 9.50 [15:52:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.76, 3.14, 3.35 [15:54:09] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.51, 3.90, 3.64 [15:56:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.69, 1.75, 1.70 [15:56:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.21, 10.74, 9.92 [15:58:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.73, 3.51 [15:58:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.94, 1.42, 1.58 [15:58:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.60, 4.56, 3.86 [15:58:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.59, 11.00, 10.13 [16:00:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.91, 3.36, 3.40 [16:00:06] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.45, 1.88, 1.91 [16:04:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.27, 4.58, 3.83 [16:08:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.96, 1.92 [16:08:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.86, 3.73 [16:10:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.06, 3.72, 3.90 [16:10:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.17, 11.08, 10.51 [16:12:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.16, 3.73, 3.68 [16:12:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.12, 10.83, 10.50 [16:13:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.96, 1.95 [16:14:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.37, 3.26, 3.52 [16:14:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.88, 9.59, 10.11 [16:16:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.51, 3.20, 3.40 [16:16:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.29, 3.52, 3.57 [16:16:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.29, 4.15, 3.94 [16:17:57] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.67, 2.18, 2.03 [16:18:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.40, 3.52 [16:18:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 4.00, 3.88, 3.86 [16:20:00] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.95, 3.96, 3.65 [16:20:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.35, 4.00, 3.91 [16:22:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.63, 3.84, 3.64 [16:22:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.12, 3.70, 3.81 [16:22:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.26, 1.76, 1.17 [16:22:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 6.76, 2.85, 1.88 [16:23:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.85, 2.69, 3.20 [16:24:00] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.09, 3.80, 3.64 [16:24:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.39, 1.11 [16:26:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.82, 3.77, 3.65 [16:28:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.64, 10.93, 10.30 [16:29:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.29, 3.90 [16:30:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.79, 9.61, 9.88 [16:33:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.06, 3.98, 4.02 [16:34:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.66, 4.17, 3.87 [16:35:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.60, 3.38 [16:37:48] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.87, 1.98 [16:38:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.84, 1.64, 1.89 [16:39:47] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.21, 1.92, 1.98 [16:39:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.02, 3.38, 3.35 [16:39:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.67, 3.68, 3.97 [16:40:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.36, 3.17, 3.38 [16:41:46] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.61, 1.83, 1.95 [16:43:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.25, 4.22, 4.09 [16:44:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.47, 3.43, 3.45 [16:44:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.88, 1.24, 1.62 [16:44:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.49, 1.56, 1.97 [16:45:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.78, 3.50 [16:47:43] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.18, 1.76, 1.85 [16:47:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.99, 3.64, 3.49 [16:48:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.66, 3.29, 3.40 [16:52:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.55, 3.59, 3.50 [16:52:44] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.57, 1.37, 1.69 [16:53:40] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.86, 1.89 [16:54:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.05, 2.94, 3.27 [16:56:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.02, 1.87, 1.83 [16:57:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.15, 3.88, 3.64 [16:58:00] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.18, 3.63, 3.48 [16:58:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.25, 1.67, 1.77 [16:59:37] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.36, 1.40, 1.66 [16:59:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.80, 3.60, 3.58 [17:00:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.91, 3.92, 3.62 [17:00:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.13, 3.59, 3.97 [17:00:44] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.12, 1.49, 1.69 [17:02:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 3.97, 4.06 [17:04:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.55, 3.57, 3.91 [17:04:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.48, 1.64, 1.71 [17:05:47] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.98, 3.17, 3.38 [17:06:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.99, 10.17, 9.62 [17:06:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.85, 3.21, 3.40 [17:06:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.45, 1.82, 1.76 [17:07:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.74, 9.80, 9.57 [17:08:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.66, 3.95, 3.96 [17:08:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.97, 1.80, 1.76 [17:11:30] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.94, 1.62, 1.58 [17:13:29] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.39, 1.50, 1.54 [17:14:00] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.04, 3.72, 3.52 [17:14:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.94, 3.57, 3.42 [17:14:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.05, 1.97, 1.82 [17:16:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.70, 3.32, 3.40 [17:16:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.85, 1.88, 1.80 [17:18:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.33, 3.68, 3.91 [17:18:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.79, 3.06, 3.30 [17:18:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.38, 2.35, 1.98 [17:19:26] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.34, 1.88, 1.67 [17:20:00] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.28, 3.78, 3.57 [17:21:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 10.42, 9.69 [17:22:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.38, 3.74, 3.59 [17:22:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.43, 4.27, 4.03 [17:22:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.87, 1.71, 1.43 [17:23:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.93, 1.75 [17:23:28] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:30] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=webkinzguidewiki (START) [17:23:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.32, 9.88, 9.58 [17:24:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:24:20] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:24:32] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:24:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.78, 5.66, 4.26 [17:25:04] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:23] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.31, 2.05, 1.81 [17:25:30] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14656 bytes in 4.976 second response time [17:26:18] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14656 bytes in 0.336 second response time [17:26:27] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 1.018 second response time [17:26:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.06, 2.16, 1.66 [17:26:59] [02puppet] 07Universal-Omega synchronize pull request 03#2588: Rewrite grafana-webhook - 13https://github.com/miraheze/puppet/pull/2588 [17:26:59] [url] Page not found · GitHub · GitHub | github.com [17:27:03] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14656 bytes in 1.241 second response time [17:27:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.60, 1.85, 1.77 [17:30:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.34, 1.90, 1.68 [17:32:00] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.66, 3.22, 3.38 [17:32:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.30, 10.10, 9.72 [17:32:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.26, 1.67, 1.62 [17:33:23] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.19, 1.99, 1.85 [17:39:25] [02puppet] 07Universal-Omega edited pull request 03#2588: Rewrite grafana-webhook - 13https://github.com/miraheze/puppet/pull/2588 [17:39:25] [url] Page not found · GitHub · GitHub | github.com [17:45:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.61, 1.91, 1.94 [17:46:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.15, 4.37, 2.88 [17:47:23] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.16, 1.96, 1.95 [17:54:29] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.81, 3.14, 2.32 [17:55:14] [02puppet] 07Universal-Omega synchronize pull request 03#2588: Rewrite grafana-webhook - 13https://github.com/miraheze/puppet/pull/2588 [17:55:14] [url] Page not found · GitHub · GitHub | github.com [17:57:10] [02puppet] 07Universal-Omega edited pull request 03#2588: Rewrite grafana-webhook - 13https://github.com/miraheze/puppet/pull/2588 [17:57:10] [url] Page not found · GitHub · GitHub | github.com [17:58:29] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 3.04, 3.14, 2.52 [18:00:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.62, 1.38, 1.90 [18:01:12] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [18:01:12] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:02:18] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.67, 8.81, 9.87 [18:03:15] [02ssl] 07Universal-Omega opened pull request 03#532: Add wiki.horizonsend.net cert - 13https://github.com/miraheze/ssl/pull/532 [18:03:16] [url] Page not found · GitHub · GitHub | github.com [18:04:00] [02ssl] 07Universal-Omega synchronize pull request 03#532: Add wiki.horizonsend.net cert - 13https://github.com/miraheze/ssl/pull/532 [18:04:00] ... [18:06:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.40, 0.98, 1.58 [18:07:02] [02puppet] 07Reception123 synchronize pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [18:07:02] [url] Page not found · GitHub · GitHub | github.com [18:07:03] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/08222e89a896...94569274eaad [18:07:04] [url] Comparing 08222e89a896...94569274eaad · miraheze/puppet · GitHub | github.com [18:07:05] [02miraheze/puppet] 07Reception123 039456927 - newline [18:08:59] [02ssl] 07Reception123 closed pull request 03#532: Add wiki.horizonsend.net cert - 13https://github.com/miraheze/ssl/pull/532 [18:09:00] ... [18:09:01] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/aa82012fe3a6...1c3058fd7f4d [18:09:01] [url] Comparing aa82012fe3a6...1c3058fd7f4d · miraheze/ssl · GitHub | github.com [18:09:02] [02miraheze/ssl] 07Universal-Omega 031c3058f - Add wiki.horizonsend.net cert (#532) [18:10:41] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [18:12:53] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=webkinzguidewiki (END - exit=0) [18:13:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:15:41] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [18:16:12] [02puppet] 07Universal-Omega reviewed pull request 03#2592 commit - 13https://github.com/miraheze/puppet/pull/2592#discussion_r883876046 [18:16:14] [url] add varnish/nginx to puppet111 (adding SSL) by Reception123 · Pull Request #2592 · miraheze/puppet · GitHub | github.com [18:17:13] [02puppet] 07Reception123 synchronize pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [18:17:14] [url] Page not found · GitHub · GitHub | github.com [18:17:15] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/94569274eaad...83a55ca6cf18 [18:17:15] [url] Comparing 94569274eaad...83a55ca6cf18 · miraheze/puppet · GitHub | github.com [18:17:16] [02miraheze/puppet] 07Reception123 0383a55ca - Update nginx.pp [18:26:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.77, 2.33, 1.79 [18:31:17] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.85, 1.73, 1.71 [18:31:54] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:33:14] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.73, 1.38, 1.58 [18:37:32] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.45, 1.53, 1.97 [18:43:16] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.82, 3.38, 3.12 [18:45:12] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.97, 3.23, 3.09 [18:46:36] [02puppet] 07Reception123 closed pull request 03#2592: add varnish/nginx to puppet111 (adding SSL) - 13https://github.com/miraheze/puppet/pull/2592 [18:46:36] [url] Page not found · GitHub · GitHub | github.com [18:46:37] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+2/-0/±5] 13https://github.com/miraheze/puppet/compare/cb84ba32a642...5fc09dd83150 [18:46:38] [url] Comparing cb84ba32a642...5fc09dd83150 · miraheze/puppet · GitHub | github.com [18:46:38] [02miraheze/puppet] 07Reception123 035fc09dd - add varnish/nginx to puppet111 (adding SSL) (#2592) [18:46:40] [02miraheze/puppet] 07Reception123 deleted branch 03Reception123-patch-3 [18:46:42] [02puppet] 07Reception123 deleted branch 03Reception123-patch-3 - 13https://github.com/miraheze/puppet [18:46:42] [url] Page not found · GitHub · GitHub | github.com [18:46:59] PROBLEM - cp21 Disk Space on cp21 is WARNING: DISK WARNING - free space: / 4277 MB (10% inode=96%); [18:47:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.83, 10.33, 9.63 [18:47:20] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.30, 1.69, 1.79 [18:48:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.41, 9.79, 9.51 [18:49:18] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.81, 1.73, 1.79 [18:49:33] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:54] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:51:04] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.32, 3.97, 3.44 [18:51:34] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22337 bytes in 6.000 second response time [18:51:52] [02puppet] 07Universal-Omega opened pull request 03#2593: letsencrypt: ensure /var/www exists - 13https://github.com/miraheze/puppet/pull/2593 [18:51:52] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14656 bytes in 1.281 second response time [18:51:53] [url] Page not found · GitHub · GitHub | github.com [18:52:14] [02puppet] 07Reception123 closed pull request 03#2593: letsencrypt: ensure /var/www exists - 13https://github.com/miraheze/puppet/pull/2593 [18:52:15] [url] Page not found · GitHub · GitHub | github.com [18:52:16] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5fc09dd83150...2228d4514b26 [18:52:16] [url] Comparing 5fc09dd83150...2228d4514b26 · miraheze/puppet · GitHub | github.com [18:52:17] [02miraheze/puppet] 07Universal-Omega 032228d45 - letsencrypt: ensure /var/www exists (#2593) [18:52:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.58, 10.86, 10.05 [18:53:13] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.57, 2.39, 2.01 [18:54:57] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.17, 3.66, 3.43 [18:55:44] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/var/www/.well-known] [18:56:53] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.44, 4.00, 3.58 [18:57:38] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [18:58:50] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.31, 3.73, 3.53 [19:01:23] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.48, 1.69, 1.41 [19:01:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.88, 9.80, 9.99 [19:01:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.23, 1.73, 1.88 [19:02:21] PROBLEM - cp31 Stunnel HTTP for puppet111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:02:54] PROBLEM - puppet111 HTTPS on puppet111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:02:57] PROBLEM - cp21 Stunnel HTTP for puppet111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:02:58] PROBLEM - cp20 Stunnel HTTP for puppet111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:07] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.22, 2.97, 3.28 [19:03:08] PROBLEM - cp30 Stunnel HTTP for puppet111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:41] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.08, 1.81, 1.88 [19:03:49] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.86, 3.57, 3.05 [19:04:38] paladox: hi, what files would need copying from mwtask111 to puppet111 for renewal to work? [19:04:50] would /etc/letsencrypt be enough? [19:05:17] /etc/letsencrypt and also you'll need to manually clone the ssl repo i think in /srv/ssl? [19:05:21] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.63, 1.80, 1.53 [19:05:31] and make sure the ssh known host key is saved for the user so it has not problems pushing [19:05:49] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.97, 4.02, 3.28 [19:06:14] paladox: why manually clone? [19:06:23] Everything should be in puppet [19:06:25] .deop [19:06:26] Attempting to OP... [19:06:52] because you have to save the ssh known host key. [19:06:56] for github [19:07:38] paladox: oh, where is the known host key? [19:07:57] Reception123: look on task [19:08:04] Or potentially on your pc [19:08:37] Well, there's many potential places it could be [19:08:49] Guess I could try a find [19:09:03] Or https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints [19:09:04] [url] GitHub's SSH key fingerprints - GitHub Docs | docs.github.com [19:09:17] Reception123: it'll be the homedir of the user git uses [19:09:18] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.27, 1.66, 1.54 [19:09:45] /root i think? [19:09:58] i'm looking quickly to make sure i've given the correct info! [19:10:08] Ah, that would make sense because there's definitely no /home for any other user than salt and us [19:11:32] it's cloned as root, although the ssh key is in /var/lib/nagios i *think* [19:11:49] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 2.76, 3.62, 3.40 [19:12:09] Ah I see, thanks! [19:12:49] yeh it's definitely root where the known_host is [19:12:55] you can confirm with ssh-keygen -F github.com (i just did) [19:13:52] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.39, 3.18, 3.26 [19:14:44] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.41, 3.22, 3.24 [19:15:11] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.27, 1.84, 1.96 [19:15:42] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [19:16:44] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.95, 3.14, 3.21 [19:17:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.48, 2.03, 2.01 [19:18:36] PROBLEM - cp20 Disk Space on cp20 is WARNING: DISK WARNING - free space: / 4279 MB (10% inode=96%); [19:18:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.49, 3.18, 3.84 [19:20:44] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.03, 3.51, 3.32 [19:20:56] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.04, 1.67, 1.89 [19:22:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.74, 3.90 [19:22:44] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.03, 3.38, 3.31 [19:23:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.68, 10.52, 10.08 [19:24:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.34, 3.73 [19:24:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.64, 2.21, 2.06 [19:25:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.45, 11.16, 10.36 [19:26:12] [02puppet] 07Universal-Omega opened pull request 03#2594: Introduce `role::letsencrypt` - 13https://github.com/miraheze/puppet/pull/2594 [19:26:12] [url] Page not found · GitHub · GitHub | github.com [19:27:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.86, 11.28, 10.50 [19:28:10] https://www.irccloud.com/pastebin/Xiw9MiAC/ [19:28:18] ^ paladox got this when trying to generate, would CosmicAlpha's PR fix that do you think? [19:28:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.86, 12.15, 10.90 [19:29:06] [02puppet] 07paladox closed pull request 03#2584: jobrunner: don't set server twice in redis.queues - 13https://github.com/miraheze/puppet/pull/2584 [19:29:06] [url] Page not found · GitHub · GitHub | github.com [19:29:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2228d4514b26...99d82794a383 [19:29:08] [url] Comparing 2228d4514b26...99d82794a383 · miraheze/puppet · GitHub | github.com [19:29:09] [02miraheze/puppet] 07Universal-Omega 0399d8279 - jobrunner: don't set server twice in redis.queues (#2584) [19:29:32] Reception123: have you confirmed it's going via puppet111? [19:32:28] Reception123: port 443 isn't open [19:32:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.05, 4.28, 3.95 [19:32:48] oh, how would that be done? [19:33:18] using the ferm rule [19:33:22] paladox: my PR should do that. [19:33:26] open it to the cache proxies only [19:33:28] CosmicAlpha: oh! [19:33:35] if I did it right. [19:34:21] CosmicAlpha: we want to limit those ports to the cache proxies i think [19:34:33] Ok, so icinga does not need them then? [19:34:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.91, 3.86 [19:34:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.67, 11.98, 11.26 [19:35:04] do we monitor the http for puppet111? [19:35:23] Yes. [19:35:35] Or at least it is alerting [19:36:05] ah ok [19:36:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.23, 4.00, 3.89 [19:37:06] [02puppet] 07Universal-Omega synchronize pull request 03#2594: Introduce `role::letsencrypt` - 13https://github.com/miraheze/puppet/pull/2594 [19:37:07] [url] Page not found · GitHub · GitHub | github.com [19:37:12] paladox: & [19:37:16] : ^ [19:37:29] CosmicAlpha: https://github.com/miraheze/puppet/blob/master/modules/role/manifests/mediawiki.pp#L10 [19:37:30] [url] puppet/mediawiki.pp at master · miraheze/puppet · GitHub | github.com [19:37:32] can just copy this [19:37:49] oh you did [19:38:05] [02puppet] 07paladox closed pull request 03#2594: Introduce `role::letsencrypt` - 13https://github.com/miraheze/puppet/pull/2594 [19:38:06] [url] Page not found · GitHub · GitHub | github.com [19:38:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±5] 13https://github.com/miraheze/puppet/compare/99d82794a383...f49d62d0a539 [19:38:07] [url] Comparing 99d82794a383...f49d62d0a539 · miraheze/puppet · GitHub | github.com [19:38:08] [02miraheze/puppet] 07Universal-Omega 03f49d62d - Introduce `role::letsencrypt` (#2594) [19:38:17] just thought of, icinga would be sending renewal requests to puppet111 [19:38:27] thus it'll have needed the icinga rule anyways in the firewall [19:38:31] Reception123: ^ [19:38:34] thanks! [19:38:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.68, 3.89, 3.87 [19:39:10] and also https://github.com/miraheze/puppet/blob/9a23c1b5f05589ab012babc0eab844a231167535/modules/monitoring/files/scripts/ssl-renew.sh#L24 needs changed when you're ready to do it all on puppet Reception123 [19:39:10] [url] puppet/ssl-renew.sh at 9a23c1b5f05589ab012babc0eab844a231167535 · miraheze/puppet · GitHub | github.com [19:39:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.78, 10.20, 8.64 [19:40:05] CosmicAlpha: paladox so the PR is good to merge now? [19:40:12] i've done it [19:40:22] and ran puppet, you can test now [19:40:40] https://www.irccloud.com/pastebin/YWo0gRCM/ [19:40:41] hmm [19:41:18] PROBLEM - cp30 Stunnel HTTP for puppet111 on cp30 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 322 bytes in 0.258 second response time [19:41:28] PROBLEM - puppet111 HTTPS on puppet111 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 322 bytes in 0.016 second response time [19:41:38] ah, didn't notice [19:41:40] thanks! [19:41:47] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.03, 9.27, 8.49 [19:41:49] oh, I guess we've ran into a problem then [19:41:56] ok works now [19:41:58] PROBLEM - cp20 Stunnel HTTP for puppet111 on cp20 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 322 bytes in 0.024 second response time [19:42:03] at least the url is now 404 [19:42:30] PROBLEM - cp31 Stunnel HTTP for puppet111 on cp31 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 322 bytes in 0.253 second response time [19:43:00] paladox: CosmicAlpha it works! thanks! [19:43:13] PROBLEM - cp21 Stunnel HTTP for puppet111 on cp21 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 322 bytes in 0.015 second response time [19:43:24] No problem! But we will want to fix those 403 alerts... [19:43:38] yeah [19:43:48] well, I guess that's that then [19:43:51] all that's left is the transfers [19:44:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.61, 3.70 [19:45:03] oh, no index exists thus 403 on root. /var/www/challenges/ would be empty [19:46:07] that is just a link [19:47:13] paladox, Reception123: ^ [19:47:30] hmm? [19:48:32] paladox: 403 is given if no index exists, so no index exists, so 403 warnings on root. /var/www/challenges/ is empty because it is just a link to /var/www/.well-known/acme-challenge [19:48:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.54, 3.66 [19:48:45] yeh, no index is supposed to be in there [19:49:28] paladox: right, but that is why it is alerting 403 for icinga monitoring. How do we fix that then? [19:50:06] paladox: also something a bit weird, when I copy /etc/letsencrypt to puppet111 the 'live' dir is missing [19:50:24] and when I tried to only scp that, it just says /home/reception/letsencrypt/live: Permission denied [19:50:31] pick /var/www/html i guess? [19:50:53] Reception123: copy it all and then change the perms to you [19:51:20] paladox: I did copy it to my homedir, though I guess maybe I need to change the perms too then [19:51:31] yeh [19:52:35] paladox: hmm, I did but now this happened [19:52:39] https://www.irccloud.com/pastebin/xkyaI5TP/ [19:53:07] it's owned by root [19:53:15] Reception123: as root? [19:53:17] chown -R reception:reception /home/reception/letsencrypt [19:53:30] now try [19:53:53] ah I did it without the -R, let me try again [19:54:01] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.70, 1.79, 1.99 [19:54:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.41, 3.57, 3.55 [19:54:56] [02puppet] 07Universal-Omega opened pull request 03#2595: letsencrypt: use /var/www/html/ for nginx root - 13https://github.com/miraheze/puppet/pull/2595 [19:54:57] [url] Page not found · GitHub · GitHub | github.com [19:56:01] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.34, 1.92, 2.01 [19:56:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.52, 11.41, 10.88 [19:56:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.73, 3.38, 3.47 [19:57:42] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/letsencrypt/cli.ini] [19:58:01] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.47, 1.87, 1.99 [19:58:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.99, 11.55, 10.99 [19:58:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.10, 3.87, 3.64 [20:00:01] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.34, 2.03, 2.03 [20:00:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 11.57, 11.06 [20:00:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.98, 3.93, 3.69 [20:02:01] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.46, 1.85, 1.97 [20:02:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.66, 10.96, 10.90 [20:04:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.73, 11.49, 11.09 [20:04:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.88, 3.09, 3.37 [20:04:55] paladox: hmm, now I don't know what I could've done wrong this time but when I do scp -r it outputs everything but the destination path on puppet111 is empty [20:05:05] I did scp -r /home/reception/letsencrypt reception@puppet111.miraheze.org:/home/reception/letsencrypt [20:05:38] i see directories in /home/reception/letsencrypt [20:06:34] was looking in the wrong dir :facepalm: made a typo [20:07:18] thanks for helping out paladox :) is there a way we can maybe trigger the renew script early or something to see if it works? [20:07:26] I copied /etc/letsencrypt and known_hosts [20:07:33] you can call the url that icinga does [20:07:49] Reception123: it won't work, the IP needs changed to puppet111. [20:08:11] CosmicAlpha: ah you mean https://github.com/miraheze/puppet/blob/9a23c1b5f05589ab012babc0eab844a231167535/modules/monitoring/files/scripts/ssl-renew.sh#L24 ? [20:08:12] [url] puppet/ssl-renew.sh at 9a23c1b5f05589ab012babc0eab844a231167535 · miraheze/puppet · GitHub | github.com [20:08:19] yes [20:08:26] https://github.com/miraheze/puppet/blob/master/modules/monitoring/files/scripts/ssl-renew.sh [20:08:26] [url] puppet/ssl-renew.sh at master · miraheze/puppet · GitHub | github.com [20:08:31] doing now [20:08:32] yeh [20:08:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.63, 11.62, 11.26 [20:08:48] looks like the ip will need changing Reception123 [20:08:54] cuz isn't that mwtask111? [20:09:04] yeah, I just mentioned that paladox. [20:09:10] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f49d62d0a539...b810b11ee3f7 [20:09:10] [url] Comparing f49d62d0a539...b810b11ee3f7 · miraheze/puppet · GitHub | github.com [20:09:11] ohhhh [20:09:11] [02miraheze/puppet] 07Reception123 03b810b11 - switch ssl-renew IP to puppet111's IP [20:09:29] [02puppet] 07paladox closed pull request 03#2595: letsencrypt: use /var/www/html/ for nginx root - 13https://github.com/miraheze/puppet/pull/2595 [20:09:29] [url] Page not found · GitHub · GitHub | github.com [20:09:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b810b11ee3f7...c700d9371e86 [20:09:31] [url] Comparing b810b11ee3f7...c700d9371e86 · miraheze/puppet · GitHub | github.com [20:09:32] [02miraheze/puppet] 07Universal-Omega 03c700d93 - letsencrypt: use /var/www/html/ for nginx root (#2595) [20:10:01] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.38, 1.35, 1.65 [20:10:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 3.94, 3.63 [20:12:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.58, 3.43, 3.48 [20:13:38] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [20:14:01] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.95, 2.08, 1.91 [20:14:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.27, 3.02, 3.32 [20:18:02] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.57, 1.94, 1.90 [20:20:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.63, 9.10, 10.11 [20:25:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.20, 3.94, 3.54 [20:25:29] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 0.36, 1.23, 1.82 [20:26:01] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.13, 2.60, 2.11 [20:26:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.42, 10.80, 10.49 [20:27:28] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 0.19, 0.87, 1.61 [20:28:01] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.07, 2.00, 1.95 [20:29:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.65, 3.51 [20:30:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.74, 11.07, 10.75 [20:33:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.05, 3.19, 3.39 [20:34:20] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.99, 3.41, 3.18 [20:35:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.24, 3.07, 3.86 [20:36:01] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.12, 1.66, 1.80 [20:36:16] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.47, 3.14, 3.11 [20:38:01] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.27, 1.57, 1.76 [20:38:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.34, 3.86 [20:40:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.35, 9.03, 9.93 [20:40:42] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [20:42:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.22, 3.22 [20:43:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.38, 3.34, 3.59 [20:44:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.45, 3.03, 3.15 [20:45:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.82, 3.25, 3.54 [20:47:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.00, 2.85, 3.35 [20:50:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.95, 11.15, 9.98 [20:50:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.78, 4.37, 4.03 [20:50:47] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.02, 3.64, 3.34 [20:51:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.72, 3.49, 3.51 [20:52:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.31, 11.24, 10.17 [20:52:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.31, 3.70, 3.83 [20:52:44] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.76, 3.21, 3.22 [20:53:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.34, 11.34, 10.55 [20:53:54] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.70, 3.36, 2.85 [20:55:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.52, 10.83, 10.45 [20:55:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.20, 3.72, 3.59 [20:55:49] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.56, 3.17, 2.85 [20:56:01] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.67, 2.39, 2.00 [20:57:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.13, 3.46, 3.50 [20:58:13] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.84, 1.69, 1.40 [20:58:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 10.08, 10.20, 10.03 [20:58:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 4.13, 3.99 [20:59:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.23, 2.60, 1.64 [21:00:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.88, 1.44, 1.35 [21:01:41] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.64, 1.86, 1.49 [21:02:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.51, 3.79 [21:03:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.25, 9.37, 10.01 [21:03:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.30, 1.62, 1.44 [21:03:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.66, 4.22, 3.73 [21:04:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.60, 4.66, 4.17 [21:05:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 4.02, 3.47 [21:07:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.93, 3.51 [21:07:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.72, 3.66 [21:09:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.93, 4.09, 3.62 [21:10:41] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [21:11:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.88, 3.75 [21:12:01] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.80, 1.53, 1.84 [21:13:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.87, 3.89, 3.66 [21:13:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.86, 3.57, 3.65 [21:15:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.68, 4.15, 3.84 [21:16:01] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.52, 1.34, 1.68 [21:17:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.15, 3.95, 3.68 [21:17:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.37, 3.89, 3.79 [21:18:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 10.52, 10.13 [21:20:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.56, 10.85, 10.29 [21:20:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.85, 3.65, 3.61 [21:22:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.11, 11.25, 10.50 [21:24:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.06, 10.16, 10.20 [21:26:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.99, 2.76, 3.28 [21:27:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.24, 2.84, 3.32 [21:28:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.74, 3.20, 3.95 [21:30:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.31, 4.62, 4.36 [21:31:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.56, 3.70, 3.56 [21:32:18] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [21:32:18] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:32:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.58, 3.50 [21:36:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.71, 3.56 [21:40:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.88, 3.70, 3.64 [21:42:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.15, 3.83, 3.69 [21:44:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.78, 3.71 [21:45:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.42, 3.62, 3.82 [21:46:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.62, 3.64 [21:47:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.52, 4.15, 4.00 [21:48:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.91, 3.87, 3.74 [21:51:12] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.52, 3.11, 2.82 [21:53:08] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.29, 3.17, 2.87 [21:55:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.62, 3.51, 3.82 [21:57:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 3.73, 3.85 [21:58:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 2.93, 3.34 [22:01:11] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [22:01:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.69, 3.76, 3.83 [22:09:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.69, 3.56, 3.66 [22:11:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.80, 3.24, 3.53 [22:12:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.23, 3.44, 3.93 [22:14:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 4.06, 4.11 [22:14:44] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.65, 3.41, 3.14 [22:15:03] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.45, 2.03, 1.46 [22:15:22] [02puppet] 07Universal-Omega opened pull request 03#2596: role::graylog: remove buster - 13https://github.com/miraheze/puppet/pull/2596 [22:15:23] [url] Page not found · GitHub · GitHub | github.com [22:15:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.40, 3.93, 3.69 [22:16:44] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.07, 3.58, 3.23 [22:16:58] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.60, 1.77, 1.43 [22:17:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.33, 3.61, 3.61 [22:18:53] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.05, 1.48, 1.36 [22:19:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 3.61, 3.44 [22:20:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.72, 3.99 [22:21:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.72, 3.34, 3.36 [22:22:12] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.94, 3.76, 2.34 [22:22:44] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.00, 3.25, 3.26 [22:25:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.61, 2.74, 3.25 [22:28:03] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.71, 1.80, 1.89 [22:28:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.65, 2.69, 3.34 [22:29:39] [02puppet] 07ugochimobi opened pull request 03#2597: add WikiMiniAtlas to frame-src - 13https://github.com/miraheze/puppet/pull/2597 [22:29:39] [url] Page not found · GitHub · GitHub | github.com [22:31:56] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.97, 1.29, 1.65 [22:36:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.69, 3.63, 3.41 [22:40:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.90, 3.55, 3.41 [22:42:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.51, 3.15, 3.28 [22:43:12] [02puppet] 07Universal-Omega opened pull request 03#2598: varnish: rename `$csp_whitelist` to `$csp` - 13https://github.com/miraheze/puppet/pull/2598 [22:43:13] [url] Page not found · GitHub · GitHub | github.com [22:43:48] [02puppet] 07Universal-Omega synchronize pull request 03#2598: varnish: rename `$csp_whitelist` to `$csp` - 13https://github.com/miraheze/puppet/pull/2598 [22:43:49] [url] Page not found · GitHub · GitHub | github.com [22:45:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.98, 3.60, 3.08 [22:46:35] [02puppet] 07Universal-Omega edited pull request 03#2598: varnish: rename `$csp_whitelist` to `$csp` - 13https://github.com/miraheze/puppet/pull/2598 [22:46:36] [url] Page not found · GitHub · GitHub | github.com [22:47:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.23, 3.02, 2.92 [22:47:30] CosmicAlpha: I don't know but I think tokens are just getting expired every seconds on gratisdatawiki, is that supposed to be so? [22:48:02] ugochimobi: what do you mean? What's happening? [22:52:03] CosmicAlpha: gratispaideia is refusing to transclude data from gratisdata, and on console it says Access to XMLHttpRequest at 'https://gratisdata.miraheze.org/w/api.php?action=wbgetentities&format=json&origin=https%3A%2F%2Fgratispaideia.miraheze.org¢ralauthtoken=_____&ids=Q3080&props=descriptions&languages=en&formatversion=2' from origin 'https://gratispaideia.miraheze.org' has been blocked by CORS policy: No [22:52:03] 'Access-Control-Allow-Origin' header is present on the requested resource. [22:52:08] [url] Main Page - Gratispaideia, the gratis, educational and collaborative encyclopedia | gratispaideia.miraheze.org [22:53:42] when I visit the complete URL above it returns `The centralauthtoken is not valid` [22:54:00] [02puppet] 07Universal-Omega opened pull request 03#2599: Remove LetsEncrypt from mwtask111 - 13https://github.com/miraheze/puppet/pull/2599 [22:54:00] ... [22:55:12] [02puppet] 07Universal-Omega synchronize pull request 03#2599: Remove LetsEncrypt from mwtask111 - 13https://github.com/miraheze/puppet/pull/2599 [22:55:12] ... [22:55:44] CosmicAlpha: any idea? [22:58:32] [02puppet] 07Universal-Omega synchronize pull request 03#2599: Remove LetsEncrypt from mwtask111 - 13https://github.com/miraheze/puppet/pull/2599 [22:58:33] [url] Page not found · GitHub · GitHub | github.com [23:04:01] [02puppet] 07Universal-Omega synchronize pull request 03#2599: Remove LetsEncrypt from mwtask111 - 13https://github.com/miraheze/puppet/pull/2599 [23:04:01] [url] Page not found · GitHub · GitHub | github.com [23:07:14] [02puppet] 07Universal-Omega synchronize pull request 03#2599: Remove LetsEncrypt from mwtask111 - 13https://github.com/miraheze/puppet/pull/2599 [23:07:15] [url] Page not found · GitHub · GitHub | github.com [23:10:23] [02puppet] 07Universal-Omega synchronize pull request 03#2599: Remove LetsEncrypt from mwtask111 - 13https://github.com/miraheze/puppet/pull/2599 [23:10:23] [url] Page not found · GitHub · GitHub | github.com [23:20:16] [02puppet] 07Universal-Omega edited pull request 03#2599: Remove LetsEncrypt from mwtask111 - 13https://github.com/miraheze/puppet/pull/2599 [23:20:17] [url] Page not found · GitHub · GitHub | github.com [23:20:27] ugochimobi: not certain. [23:20:42] [02puppet] 07Universal-Omega edited pull request 03#2599: Remove LetsEncrypt from mwtask111 - 13https://github.com/miraheze/puppet/pull/2599 [23:20:43] ... [23:27:33] CosmicAlpha: it's probably something related to CORS [23:27:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.33, 2.99 [23:28:20] ugochimobi: we don't support cors for anything but loginwiki and metawiki I believe. [23:30:14] At least not in that way with the authentication necessary. [23:31:26] CosmicAlpha: welp, console says that and also XMLHttpRequest [23:32:05] Yes, I know, it used to happen with GlobalWatchlist, until I modified the setup to support at the very least loginwiki and metawiki. [23:33:12] Do a PR to add your site you need to load on only thr site you need loaded from to `$wgCrossSiteAJAXdomains` (and it may only theoretically work for you) [23:33:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.67, 3.25, 3.12 [23:33:58] I'm off for the day now. [23:34:40] CosmicAlpha: wdym only theoritically work for me? [23:34:49] list me as a single user? [23:35:02] or the entire wiki users? [23:35:11] s/list/like [23:35:11] ugochimobi meant to say: like me as a single user? [23:35:48] ugochimobi: it would work for your wiki, if it works at all, I'm not positive if it'll work doing that. [23:36:51] hmm [23:39:42] [02mw-config] 07ugochimobi opened pull request 03#4713: add gratisdata.miraheze.org as wgCrossSiteAJAXdomains on gratispaideiawiki - 13https://github.com/miraheze/mw-config/pull/4713 [23:39:42] [url] Page not found · GitHub · GitHub | github.com [23:40:01] CosmicAlpha: I did as you said https://github.com/miraheze/mw-config/pull/4713 [23:40:01] [url] add gratisdata.miraheze.org as wgCrossSiteAJAXdomains on gratispaideiawiki by ugochimobi · Pull Request #4713 · miraheze/mw-config · GitHub | github.com [23:40:46] miraheze/mw-config - ugochimobi the build passed. [23:41:22] Thanks. I'm away the rest lf the day though now. [23:44:27] [02mw-config] 07Universal-Omega closed pull request 03#4713: add gratisdata.miraheze.org as wgCrossSiteAJAXdomains on gratispaideiawiki - 13https://github.com/miraheze/mw-config/pull/4713 [23:44:28] [url] Page not found · GitHub · GitHub | github.com [23:44:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6f9c5da6d9e0...a673f916e46f [23:44:29] [url] Comparing 6f9c5da6d9e0...a673f916e46f · miraheze/mw-config · GitHub | github.com [23:44:30] [02miraheze/mw-config] 07ugochimobi 03a673f91 - add gratisdata.miraheze.org as wgCrossSiteAJAXdomains on gratispaideiawiki (#4713) [23:45:30] miraheze/mw-config - Universal-Omega the build passed. [23:46:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.36, 3.78, 3.40 [23:46:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.81, 3.39 [23:48:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.99, 3.85, 3.44 [23:48:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.98, 4.36, 3.62 [23:49:19] !log [@test101] starting deploy of {'config': True} to all [23:49:19] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [23:49:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:49:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:50:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.85, 3.68, 3.42 [23:50:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.91, 3.54 [23:51:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.03, 2.37, 1.49 [23:52:52] [02puppet] 07paladox closed pull request 03#2596: role::graylog: remove buster - 13https://github.com/miraheze/puppet/pull/2596 [23:52:52] [url] Page not found · GitHub · GitHub | github.com [23:52:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c700d9371e86...4e4ee07dde4c [23:52:54] [url] Comparing c700d9371e86...4e4ee07dde4c · miraheze/puppet · GitHub | github.com [23:52:55] [02miraheze/puppet] 07Universal-Omega 034e4ee07 - role::graylog: remove buster (#2596) [23:53:15] !log [@mwtask111] starting deploy of {'config': True} to all [23:53:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.23, 1.81, 1.38 [23:53:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:34] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 19s [23:53:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:40] [02puppet] 07paladox closed pull request 03#2598: varnish: rename `$csp_whitelist` to `$csp` - 13https://github.com/miraheze/puppet/pull/2598 [23:53:40] [url] Page not found · GitHub · GitHub | github.com [23:53:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/4e4ee07dde4c...687f4a9e0fdd [23:53:42] [url] Comparing 4e4ee07dde4c...687f4a9e0fdd · miraheze/puppet · GitHub | github.com [23:53:43] [02miraheze/puppet] 07Universal-Omega 03687f4a9 - varnish: rename `$csp_whitelist` to `$csp` (#2598) [23:54:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 4.06, 3.64 [23:54:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.87, 4.01, 3.66 [23:55:24] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.52, 1.34, 1.26 [23:56:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.74, 3.57 [23:56:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.17, 3.28, 3.43 [23:58:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.77, 3.14, 3.36