[00:02:20] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: DISK CRITICAL - free space: / 2220 MB (5% inode=96%); [00:04:00] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 10238 MB (26% inode=96%); [00:04:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.73, 4.25, 3.77 [00:04:20] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 10102 MB (25% inode=96%); [00:04:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.97, 4.56, 3.80 [00:04:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.55, 4.31, 3.39 [00:11:24] [02mw-config] 07ugochimobi opened pull request 03#4714: add gratispaideia.miraheze.org as wgCrossSiteAJAXdomains for gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/4714 [00:11:24] [url] Page not found · GitHub · GitHub | github.com [00:12:32] miraheze/mw-config - ugochimobi the build passed. [00:16:35] [02mw-config] 07Universal-Omega closed pull request 03#4714: add gratispaideia.miraheze.org as wgCrossSiteAJAXdomains for gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/4714 [00:16:36] [url] Page not found · GitHub · GitHub | github.com [00:16:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a673f916e46f...9662e8a249c3 [00:16:37] [url] Comparing a673f916e46f...9662e8a249c3 · miraheze/mw-config · GitHub | github.com [00:16:38] [02miraheze/mw-config] 07ugochimobi 039662e8a - add gratispaideia.miraheze.org as wgCrossSiteAJAXdomains for gratisdatawiki (#4714) [00:17:43] miraheze/mw-config - Universal-Omega the build passed. [00:18:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.64, 3.72, 3.72 [00:19:01] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.81, 1.30 [00:20:55] !log [@test101] starting deploy of {'config': True} to all [00:20:56] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [00:21:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:22:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 3.71, 3.69 [00:22:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.89, 1.45 [00:23:19] !log [@mwtask111] starting deploy of {'config': True} to all [00:23:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:29] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 9s [00:23:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.47, 3.60 [00:28:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.35, 1.64, 1.49 [00:30:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.20, 2.92, 3.32 [00:32:56] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.99, 1.88, 1.63 [00:34:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.38, 3.30, 3.92 [00:34:55] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.98, 1.70 [00:36:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.02, 3.88, 4.05 [00:36:54] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.93, 1.72 [00:40:55] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.41, 1.65, 1.66 [00:43:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.74, 3.52, 4.00 [00:46:52] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.96, 1.79, 1.70 [00:48:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.71, 3.96 [00:48:52] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.36, 2.04, 1.81 [00:49:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.79, 4.31, 4.13 [00:51:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.46, 3.54, 3.86 [00:54:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.46, 3.75 [00:55:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.61, 4.17, 4.05 [00:56:00] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.50, 1.17 [00:56:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.09, 3.08, 3.58 [00:57:59] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.34, 1.36, 1.15 [00:58:48] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.79, 1.97, 1.94 [00:59:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.41, 3.30 [01:00:23] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [01:00:23] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:00:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.29, 2.91, 3.38 [01:00:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.11, 3.26, 3.25 [01:02:47] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.10, 1.97, 1.94 [01:04:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4714-patch-83 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/7de4ca05be94 [01:04:20] [02miraheze/mw-config] 07Universal-Omega 037de4ca0 - Revert "add gratispaideia.miraheze.org as wgCrossSiteAJAXdomains for gratisdatawiki (#4714)" [01:04:21] [02mw-config] 07Universal-Omega created branch 03revert-4714-patch-83 - 13https://github.com/miraheze/mw-config [01:04:22] [url] Page not found · GitHub · GitHub | github.com [01:04:23] [02mw-config] 07Universal-Omega opened pull request 03#4715: Revert "add gratispaideia.miraheze.org as wgCrossSiteAJAXdomains for gratisdatawiki" - 13https://github.com/miraheze/mw-config/pull/4715 [01:04:23] ... [01:04:26] [02mw-config] 07Universal-Omega closed pull request 03#4715: Revert "add gratispaideia.miraheze.org as wgCrossSiteAJAXdomains for gratisdatawiki" - 13https://github.com/miraheze/mw-config/pull/4715 [01:04:26] [url] Page not found · GitHub · GitHub | github.com [01:04:28] [02mw-config] 07Universal-Omega deleted branch 03revert-4714-patch-83 - 13https://github.com/miraheze/mw-config [01:04:28] ... [01:04:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9662e8a249c3...d515d795345f [01:04:30] [url] Comparing 9662e8a249c3...d515d795345f · miraheze/mw-config · GitHub | github.com [01:04:31] [02miraheze/mw-config] 07Universal-Omega 03d515d79 - Revert "add gratispaideia.miraheze.org as wgCrossSiteAJAXdomains for gratisdatawiki" (#4715) [01:04:32] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4714-patch-83 [01:04:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4713-patch-82 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/8c689b484004 [01:04:40] [url] Page not found · GitHub · GitHub | github.com [01:04:41] [02miraheze/mw-config] 07Universal-Omega 038c689b4 - Revert "add gratisdata.miraheze.org as wgCrossSiteAJAXdomains on gratispaideiawiki (#4713)" [01:04:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.74, 3.60 [01:04:43] [02mw-config] 07Universal-Omega created branch 03revert-4713-patch-82 - 13https://github.com/miraheze/mw-config [01:04:43] [url] Page not found · GitHub · GitHub | github.com [01:04:44] [02mw-config] 07Universal-Omega opened pull request 03#4716: Revert "add gratisdata.miraheze.org as wgCrossSiteAJAXdomains on gratispaideiawiki" - 13https://github.com/miraheze/mw-config/pull/4716 [01:04:45] [url] Page not found · GitHub · GitHub | github.com [01:04:46] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.72, 1.90, 1.92 [01:04:47] [02mw-config] 07Universal-Omega closed pull request 03#4716: Revert "add gratisdata.miraheze.org as wgCrossSiteAJAXdomains on gratispaideiawiki" - 13https://github.com/miraheze/mw-config/pull/4716 [01:04:47] ... [01:04:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.55, 3.38 [01:04:48] [02mw-config] 07Universal-Omega deleted branch 03revert-4713-patch-82 - 13https://github.com/miraheze/mw-config [01:04:48] [url] Page not found · GitHub · GitHub | github.com [01:04:50] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4713-patch-82 [01:04:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d515d795345f...c48775dcf573 [01:04:52] [url] Comparing d515d795345f...c48775dcf573 · miraheze/mw-config · GitHub | github.com [01:04:53] [02miraheze/mw-config] 07Universal-Omega 03c48775d - Revert "add gratisdata.miraheze.org as wgCrossSiteAJAXdomains on gratispaideiawiki" (#4716) [01:05:28] miraheze/mw-config - Universal-Omega the build passed. [01:05:34] miraheze/mw-config - Universal-Omega the build passed. [01:05:47] miraheze/mw-config - Universal-Omega the build passed. [01:06:00] miraheze/mw-config - Universal-Omega the build passed. [01:10:41] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [01:10:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.78, 3.20, 3.31 [01:11:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.83, 3.71, 4.00 [01:14:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.92, 3.81 [01:14:43] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.20, 1.89, 1.88 [01:15:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.15, 3.65, 3.89 [01:16:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.58, 4.55, 4.06 [01:19:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.10, 3.76, 3.92 [01:21:38] !log [@test101] starting deploy of {'config': True} to all [01:21:39] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [01:21:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.37, 4.72, 4.24 [01:21:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.61, 3.85, 3.97 [01:24:06] !log [@mwtask111] starting deploy of {'config': True} to all [01:24:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:15] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 8s [01:24:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.23, 1.79, 1.92 [01:26:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.74, 3.94 [01:29:16] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [01:29:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.69, 1.93, 1.51 [01:32:42] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.08, 1.36, 1.67 [01:33:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.94, 1.47, 1.42 [01:35:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.97, 2.87, 3.37 [01:43:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.35, 3.34 [01:47:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.17, 3.04, 3.24 [01:52:34] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.02, 1.77, 1.53 [01:52:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.54, 3.60 [01:53:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.24, 3.71, 3.49 [01:54:34] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.26, 1.58, 1.49 [01:55:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.17, 4.59, 3.85 [01:56:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.19, 3.20, 3.46 [01:58:09] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [01:58:09] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:00:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.36, 4.33, 3.78 [02:08:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.87, 3.78 [02:11:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.07, 3.97, 3.98 [02:13:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.56, 4.66, 4.21 [02:14:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.73, 4.14, 3.87 [02:18:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.65, 2.07, 1.69 [02:20:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.91, 3.93 [02:21:41] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.24, 1.41, 1.09 [02:21:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.53, 3.87 [02:23:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.20, 1.31, 1.09 [02:24:22] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.48, 1.96, 1.81 [02:26:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.92, 4.25, 4.00 [02:27:02] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [02:27:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.60, 10.03, 9.05 [02:27:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.60, 4.04, 3.95 [02:29:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.14, 3.05 [02:29:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.53, 9.18, 8.87 [02:29:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.69, 3.84 [02:30:20] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.83, 1.77 [02:31:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.98, 4.47, 4.12 [02:32:19] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.77, 1.76 [02:32:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.06, 3.86, 3.33 [02:34:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.46, 3.89, 3.41 [02:38:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.71, 3.42 [02:44:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.37, 3.32, 3.39 [02:46:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.71, 3.97 [02:47:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.38, 3.31, 3.81 [02:48:14] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.96, 2.12, 1.88 [02:55:55] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [02:56:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.34, 1.81, 1.88 [02:56:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.88, 4.33, 4.07 [02:59:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.76, 3.07, 3.40 [03:02:09] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.26, 1.46, 1.70 [03:02:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.09, 3.62, 3.84 [03:08:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.48, 3.67 [03:11:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.77, 3.54, 3.45 [03:13:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.09, 3.25, 3.37 [03:18:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.31, 1.77, 1.48 [03:18:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 3.85, 3.85 [03:22:07] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.61, 1.28, 1.35 [03:24:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.16, 2.64, 3.31 [03:29:34] PROBLEM - wiki.macc.nyc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.macc.nyc' expires in 15 day(s) (Mon 13 Jun 2022 03:02:37 GMT +0000). [03:39:28] PROBLEM - miraheze.gq - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.gq' expires in 15 day(s) (Mon 13 Jun 2022 03:12:10 GMT +0000). [03:44:09] PROBLEM - wiki.counterculturelabs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.counterculturelabs.org' expires in 15 day(s) (Mon 13 Jun 2022 03:20:20 GMT +0000). [03:45:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.57, 3.29 [03:47:08] PROBLEM - pastport.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pastport.org' expires in 15 day(s) (Mon 13 Jun 2022 03:28:44 GMT +0000). [03:49:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.57, 4.48, 3.76 [03:49:52] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.91, 1.78, 1.55 [03:51:08] PROBLEM - wiki.erisly.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.erisly.com' expires in 15 day(s) (Mon 13 Jun 2022 03:34:11 GMT +0000). [03:52:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.01, 3.77, 3.62 [03:55:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.66, 3.34 [03:56:42] PROBLEM - podpedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'podpedia.org' expires in 15 day(s) (Mon 13 Jun 2022 03:34:58 GMT +0000). [03:56:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.50, 3.17, 3.40 [03:57:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.51, 3.25, 3.23 [03:57:49] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.19, 2.17, 1.77 [04:00:29] PROBLEM - lcn.zfc.id.lv - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lcn.zfc.id.lv' expires in 15 day(s) (Mon 13 Jun 2022 03:35:39 GMT +0000). [04:06:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.04, 3.14 [04:08:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.29, 3.11, 3.15 [04:12:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 3.59, 3.30 [04:14:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.39, 3.26 [04:15:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 4.00, 3.97, 3.49 [04:16:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.69, 3.22, 3.22 [04:17:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.08, 3.19, 3.25 [04:22:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.99, 3.55 [04:24:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.56, 3.44 [04:26:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.17, 3.10, 3.29 [04:52:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.65, 3.39 [04:54:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.92, 3.28, 3.28 [04:55:23] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [05:00:49] [02puppet] 07Reception123 closed pull request 03#2599: Remove LetsEncrypt from mwtask111 - 13https://github.com/miraheze/puppet/pull/2599 [05:00:50] [url] Page not found · GitHub · GitHub | github.com [05:00:51] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/puppet/compare/687f4a9e0fdd...150b894bbe75 [05:00:51] [url] Comparing 687f4a9e0fdd...150b894bbe75 · miraheze/puppet · GitHub | github.com [05:00:52] [02miraheze/puppet] 07Universal-Omega 03150b894 - Remove LetsEncrypt from mwtask111 (#2599) [05:06:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 4.21, 3.39 [05:06:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 4.40, 3.70 [05:06:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.66, 2.97 [05:08:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.54, 3.94, 3.39 [05:10:41] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [05:10:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.89, 3.67 [05:10:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.91, 3.26 [05:12:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.67, 4.04, 3.54 [05:14:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.63, 3.43, 3.37 [05:14:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.93, 3.73 [05:14:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.50, 3.19, 3.12 [05:16:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.74, 3.69 [05:22:55] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/96acb6a62e3f [05:22:57] [02miraheze/puppet] 07Reception123 0396acb6a - automate copying private keys for SSL certs (T7582) [05:22:58] [02puppet] 07Reception123 created branch 03Reception123-patch-3 - 13https://github.com/miraheze/puppet [05:22:59] [url] Page not found · GitHub · GitHub | github.com [05:23:00] [02puppet] 07Reception123 opened pull request 03#2600: automate copying private keys for SSL certs (T7582) - 13https://github.com/miraheze/puppet/pull/2600 [05:23:00] ... [05:23:06] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [05:23:51] [02puppet] 07Reception123 synchronize pull request 03#2600: automate copying private keys for SSL certs (T7582) - 13https://github.com/miraheze/puppet/pull/2600 [05:23:52] [url] Page not found · GitHub · GitHub | github.com [05:23:53] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/96acb6a62e3f...6eec87e21120 [05:23:53] [url] Comparing 96acb6a62e3f...6eec87e21120 · miraheze/puppet · GitHub | github.com [05:23:54] [02miraheze/puppet] 07Reception123 036eec87e - Update ssl-certificate.py [05:24:04] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 4.14, 3.70 [05:24:16] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [05:25:02] [02ssl] 07Universal-Omega opened pull request 03#533: Certs: move beta to top - 13https://github.com/miraheze/ssl/pull/533 [05:25:03] [url] Page not found · GitHub · GitHub | github.com [05:25:24] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [05:26:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.83, 3.64 [05:26:05] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/1c3058fd7f4d...ea617f098e88 [05:26:06] [url] Comparing 1c3058fd7f4d...ea617f098e88 · miraheze/ssl · GitHub | github.com [05:26:07] [02miraheze/ssl] 07Reception123 03ea617f0 - add wiki.otir.nl cert [05:27:08] PROBLEM - mwtask111 MirahezeRenewSsl on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:27:16] [02ssl] 07Universal-Omega synchronize pull request 03#533: Certs: move beta to top - 13https://github.com/miraheze/ssl/pull/533 [05:27:16] [url] Page not found · GitHub · GitHub | github.com [05:28:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.90, 3.05, 3.35 [05:29:25] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [05:29:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 3.61, 3.56 [05:30:31] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [05:31:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.02, 9.85, 9.00 [05:31:52] [02ssl] 07Reception123 closed pull request 03#533: Certs: move beta to top - 13https://github.com/miraheze/ssl/pull/533 [05:31:53] [url] Page not found · GitHub · GitHub | github.com [05:31:54] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ea617f098e88...aa6d0d18918c [05:31:55] [url] Comparing ea617f098e88...aa6d0d18918c · miraheze/ssl · GitHub | github.com [05:31:56] [02miraheze/ssl] 07Universal-Omega 03aa6d0d1 - Certs: move beta to top (#533) [05:32:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.83, 3.60 [05:34:31] [02puppet] 07Universal-Omega opened pull request 03#2601: Fix varnish syntax - 13https://github.com/miraheze/puppet/pull/2601 [05:34:31] [url] Page not found · GitHub · GitHub | github.com [05:35:46] PROBLEM - puppet111 Current Load on puppet111 is WARNING: WARNING - load average: 5.71, 5.15, 3.70 [05:35:50] [02puppet] 07Universal-Omega synchronize pull request 03#2601: Fix varnish syntax - 13https://github.com/miraheze/puppet/pull/2601 [05:35:50] ... [05:37:07] [02puppet] 07Universal-Omega edited pull request 03#2601: varnish: fix syntax - 13https://github.com/miraheze/puppet/pull/2601 [05:37:07] [url] Page not found · GitHub · GitHub | github.com [05:37:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.99, 9.49, 9.21 [05:37:44] RECOVERY - puppet111 Current Load on puppet111 is OK: OK - load average: 1.74, 3.90, 3.42 [05:37:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.69, 3.62 [05:37:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.24, 3.82, 3.77 [05:37:52] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki.otir.nl.key] [05:38:58] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki.otir.nl.key] [05:39:01] PROBLEM - mw112 Puppet on mw112 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki.otir.nl.key] [05:39:20] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki.otir.nl.key] [05:39:28] PROBLEM - mw102 Puppet on mw102 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki.otir.nl.key] [05:39:37] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki.otir.nl.key] [05:39:44] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki.otir.nl.key] [05:41:38] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.57, 3.59, 3.65 [05:41:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.61, 3.00, 3.34 [05:43:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.23, 3.50 [05:43:38] [02puppet] 07Reception123 closed pull request 03#2601: varnish: fix syntax - 13https://github.com/miraheze/puppet/pull/2601 [05:43:38] [url] Page not found · GitHub · GitHub | github.com [05:43:40] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/150b894bbe75...cb6a9508a918 [05:43:40] [url] Comparing 150b894bbe75...cb6a9508a918 · miraheze/puppet · GitHub | github.com [05:43:41] [02miraheze/puppet] 07Universal-Omega 03cb6a950 - varnish: fix syntax (#2601) [05:47:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.80, 3.57 [05:51:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.90, 3.83, 3.66 [05:51:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.08, 3.17, 3.33 [05:53:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.20, 11.15, 9.74 [05:55:23] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.99, 2.35, 1.48 [05:55:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.69, 11.35, 9.99 [05:55:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.87, 4.01, 3.73 [05:55:38] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 3.62, 3.46 [05:57:23] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.55, 3.18 [05:59:15] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki.otir.nl.key] [05:59:19] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.64, 10.18, 9.79 [05:59:25] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki.otir.nl.key] [05:59:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.84, 3.99, 3.66 [06:01:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.72, 1.57, 1.43 [06:01:24] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.52, 4.45, 3.89 [06:03:06] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.84, 3.24, 3.18 [06:05:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.17, 3.36, 3.58 [06:05:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.48, 3.74, 3.89 [06:08:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.12, 1.67, 1.29 [06:09:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.44, 3.97, 3.77 [06:10:50] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.26, 1.51, 1.28 [06:11:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.55, 3.48, 3.61 [06:13:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.60, 3.81, 3.82 [06:14:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.96, 2.85, 3.34 [06:15:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.60, 3.74 [06:16:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.91, 1.94, 1.38 [06:17:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.17, 3.64, 3.73 [06:19:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.48, 3.64 [06:20:09] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.16, 1.62, 1.37 [06:21:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.23, 3.68, 3.69 [06:22:56] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.53, 3.53, 2.78 [06:23:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.83, 3.45, 3.59 [06:23:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.56, 3.93, 3.57 [06:27:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.94, 4.08, 3.79 [06:28:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.61, 3.24 [06:29:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.88, 3.77 [06:29:48] [02puppet] 07JohnFLewis commented on pull request 03#2600: automate copying private keys for SSL certs (T7582) - 13https://github.com/miraheze/puppet/pull/2600#issuecomment-1140186037 [06:29:48] [url] automate copying private keys for SSL certs (T7582) by Reception123 · Pull Request #2600 · miraheze/puppet · GitHub | github.com [06:30:52] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.54, 3.83, 3.27 [06:31:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.72, 4.20, 3.92 [06:32:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.30, 3.64, 3.31 [06:33:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.81, 3.81 [06:34:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.49, 3.14, 3.17 [06:35:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.64, 3.54, 3.87 [06:39:12] [02puppet] 07Reception123 commented on pull request 03#2600: automate copying private keys for SSL certs (T7582) - 13https://github.com/miraheze/puppet/pull/2600#issuecomment-1140187487 [06:39:12] [url] automate copying private keys for SSL certs (T7582) by Reception123 · Pull Request #2600 · miraheze/puppet · GitHub | github.com [06:40:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.88, 1.56, 1.30 [06:42:03] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:42:16] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:42:26] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:42:52] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.92, 3.37, 3.36 [06:42:58] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:43:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 8.21, 4.91, 4.22 [06:43:08] RECOVERY - mw102 Puppet on mw102 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:43:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.90, 4.50, 3.93 [06:43:55] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:44:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.13, 1.42, 1.30 [06:44:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.75, 4.90, 3.76 [06:52:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.69, 1.81, 1.56 [06:52:08] [02puppet] 07JohnFLewis commented on pull request 03#2600: automate copying private keys for SSL certs (T7582) - 13https://github.com/miraheze/puppet/pull/2600#issuecomment-1140188901 [06:52:08] [url] automate copying private keys for SSL certs (T7582) by Reception123 · Pull Request #2600 · miraheze/puppet · GitHub | github.com [06:53:09] RECOVERY - mw112 Puppet on mw112 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:53:45] [02puppet] 07Reception123 commented on pull request 03#2600: automate copying private keys for SSL certs (T7582) - 13https://github.com/miraheze/puppet/pull/2600#issuecomment-1140189115 [06:53:45] [url] automate copying private keys for SSL certs (T7582) by Reception123 · Pull Request #2600 · miraheze/puppet · GitHub | github.com [06:54:24] [02puppet] 07Reception123 synchronize pull request 03#2600: automate copying private keys for SSL certs (T7582) - 13https://github.com/miraheze/puppet/pull/2600 [06:54:25] [url] Page not found · GitHub · GitHub | github.com [06:54:26] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6eec87e21120...bed073836b48 [06:54:26] [url] Comparing 6eec87e21120...bed073836b48 · miraheze/puppet · GitHub | github.com [06:54:27] [02miraheze/puppet] 07Reception123 03bed0738 - switch to homedir [06:54:52] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.39, 3.83, 3.53 [06:56:14] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:57:24] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.23, 2.08, 1.71 [06:59:48] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:00:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.04, 1.66, 1.60 [07:00:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.05, 10.45, 9.54 [07:00:52] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.40, 3.84, 3.69 [07:02:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.60, 9.89, 9.44 [07:03:28] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:05:20] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:05:37] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [07:06:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.35, 10.33, 9.77 [07:08:52] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.22, 3.50, 3.51 [07:10:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.08, 10.02, 9.76 [07:10:52] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 4.00, 3.61, 3.55 [07:20:52] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.30, 3.00, 3.33 [07:27:24] !log upgraded phabricator [07:27:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:27:37] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.63, 1.71, 1.47 [07:29:31] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.61, 1.64, 1.47 [07:36:34] [02puppet] 07Universal-Omega opened pull request 03#2602: Move change-password to LetsEncrypt - 13https://github.com/miraheze/puppet/pull/2602 [07:36:35] [url] Page not found · GitHub · GitHub | github.com [07:37:14] [02puppet] 07Universal-Omega synchronize pull request 03#2602: Move change-password to LetsEncrypt - 13https://github.com/miraheze/puppet/pull/2602 [07:37:15] [url] Page not found · GitHub · GitHub | github.com [07:39:13] [02puppet] 07Universal-Omega opened pull request 03#2603: mediawiki: remove acme-challenge - 13https://github.com/miraheze/puppet/pull/2603 [07:39:13] [url] Page not found · GitHub · GitHub | github.com [07:40:37] [02puppet] 07Universal-Omega edited pull request 03#2603: mediawiki: remove acme-challenge - 13https://github.com/miraheze/puppet/pull/2603 [07:40:38] [url] Page not found · GitHub · GitHub | github.com [07:42:50] [02puppet] 07Universal-Omega synchronize pull request 03#2603: mediawiki: remove acme-challenge - 13https://github.com/miraheze/puppet/pull/2603 [07:42:50] [url] Page not found · GitHub · GitHub | github.com [07:46:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.47, 3.22, 3.85 [07:52:11] [02puppet] 07Reception123 closed pull request 03#2600: automate copying private keys for SSL certs (T7582) - 13https://github.com/miraheze/puppet/pull/2600 [07:52:12] [url] Page not found · GitHub · GitHub | github.com [07:52:13] [02miraheze/puppet] 07Reception123 deleted branch 03Reception123-patch-3 [07:52:14] [02puppet] 07Reception123 deleted branch 03Reception123-patch-3 - 13https://github.com/miraheze/puppet [07:52:14] ... [07:52:16] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cb6a9508a918...b61884fef540 [07:52:16] [url] Comparing cb6a9508a918...b61884fef540 · miraheze/puppet · GitHub | github.com [07:52:17] [02miraheze/puppet] 07Reception123 03b61884f - automate copying private keys for SSL certs (T7582) (#2600) [07:52:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.83, 3.91 [07:53:11] [02puppet] 07Reception123 closed pull request 03#2602: Move change-password to LetsEncrypt - 13https://github.com/miraheze/puppet/pull/2602 [07:53:11] ... [07:53:12] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/b61884fef540...a94f37091ea6 [07:53:13] [url] Comparing b61884fef540...a94f37091ea6 · miraheze/puppet · GitHub | github.com [07:53:14] [02miraheze/puppet] 07Universal-Omega 03a94f370 - Move change-password to LetsEncrypt (#2602) [07:53:32] [02puppet] 07Reception123 closed pull request 03#2603: mediawiki: remove acme-challenge - 13https://github.com/miraheze/puppet/pull/2603 [07:53:33] [url] Page not found · GitHub · GitHub | github.com [07:53:34] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/a94f37091ea6...cccc39d61c17 [07:53:34] [url] Comparing a94f37091ea6...cccc39d61c17 · miraheze/puppet · GitHub | github.com [07:53:35] [02miraheze/puppet] 07Universal-Omega 03cccc39d - mediawiki: remove acme-challenge (#2603) [07:54:52] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.72, 3.12, 2.55 [07:56:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.52, 3.89, 3.93 [07:56:58] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.27, 1.94, 1.66 [07:58:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.65, 4.30, 4.08 [07:58:51] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.31, 1.75, 1.63 [07:58:52] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 3.21, 3.21, 2.72 [08:00:42]