[00:03:06] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:06] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.60, 20.75, 18.83 [00:03:52] RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 8175 MB (21% inode=96%); [00:03:57] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 15.50, 9.64, 8.53 [00:04:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.96, 9.09, 8.30 [00:06:04] PROBLEM - fr.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fr.gyaanipedia.com could not be found [00:06:12] PROBLEM - wiki.ripto.gq - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ripto.gq could not be found [00:06:23] PROBLEM - reviews.thejoshmeister.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for reviews.thejoshmeister.com could not be found [00:06:23] PROBLEM - wiki.sdbx.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sdbx.dev could not be found [00:06:36] PROBLEM - wiki.newnormal.fans - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.newnormal.fans could not be found [00:06:44] PROBLEM - wiki.patriam.cc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.patriam.cc could not be found [00:06:47] PROBLEM - wiki.mcpirevival.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mcpirevival.tk could not be found [00:06:55] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.19, 22.84, 20.11 [00:08:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.34, 19.97, 19.39 [00:12:45] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.96, 21.26, 20.16 [00:13:03] RECOVERY - reviews.thejoshmeister.com - reverse DNS on sslhost is OK: SSL OK - reviews.thejoshmeister.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:05] RECOVERY - fr.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - fr.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:05] RECOVERY - wiki.sdbx.dev - reverse DNS on sslhost is OK: SSL OK - wiki.sdbx.dev reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:06] RECOVERY - wiki.ripto.gq - reverse DNS on sslhost is OK: SSL OK - wiki.ripto.gq reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:24] RECOVERY - wiki.patriam.cc - reverse DNS on sslhost is OK: SSL OK - wiki.patriam.cc reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:28] RECOVERY - wiki.newnormal.fans - reverse DNS on sslhost is OK: SSL OK - wiki.newnormal.fans reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:30] RECOVERY - wiki.mcpirevival.tk - reverse DNS on sslhost is OK: SSL OK - wiki.mcpirevival.tk reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:14:44] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 36.60, 25.84, 21.87 [00:16:18] PROBLEM - runzeppelin.ru - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for runzeppelin.ru could not be found [00:16:20] PROBLEM - wiki.lefrenchmelee.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.lefrenchmelee.fr could not be found [00:16:21] PROBLEM - www.zenbuddhism.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.zenbuddhism.info could not be found [00:16:22] PROBLEM - podpedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for podpedia.org could not be found [00:16:22] PROBLEM - files.pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for files.pornwiki.org could not be found [00:16:23] PROBLEM - hololive.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hololive.wiki could not be found [00:16:29] PROBLEM - nenawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nenawiki.org could not be found [00:16:32] PROBLEM - antiguabarbudacalypso.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for antiguabarbudacalypso.com could not be found [00:16:36] PROBLEM - bn.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bn.gyaanipedia.com could not be found [00:20:14] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBvDv [00:20:15] [02miraheze/ManageWiki] 07Universal-Omega 03f0c88c1 - Add more information to dialog [00:20:17] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vpSns [00:20:18] [02ManageWiki] 07Universal-Omega opened pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:20:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.96, 22.81, 21.83 [00:21:13] miraheze/ManageWiki - Universal-Omega the build passed. [00:23:09] RECOVERY - runzeppelin.ru - reverse DNS on sslhost is OK: SSL OK - runzeppelin.ru reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:23:13] RECOVERY - wiki.lefrenchmelee.fr - reverse DNS on sslhost is OK: SSL OK - wiki.lefrenchmelee.fr reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:23:16] RECOVERY - antiguabarbudacalypso.com - reverse DNS on sslhost is OK: SSL OK - antiguabarbudacalypso.com reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:23:17] RECOVERY - www.zenbuddhism.info - reverse DNS on sslhost is OK: SSL OK - www.zenbuddhism.info reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:23:18] RECOVERY - podpedia.org - reverse DNS on sslhost is OK: SSL OK - podpedia.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:23:20] RECOVERY - hololive.wiki - reverse DNS on sslhost is OK: SSL OK - hololive.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:23:20] RECOVERY - files.pornwiki.org - reverse DNS on sslhost is OK: SSL OK - files.pornwiki.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:23:21] RECOVERY - bn.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - bn.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:23:30] RECOVERY - nenawiki.org - reverse DNS on sslhost is OK: SSL OK - nenawiki.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:25:21] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBvds [00:25:22] [02miraheze/ManageWiki] 07Universal-Omega 035b4237e - Update ext.managewiki.oouiform.reviewChanges.js [00:25:24] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:26:19] miraheze/ManageWiki - Universal-Omega the build passed. [00:26:22] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.10, 18.15, 20.15 [00:26:48] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:26:49] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBvb4 [00:26:51] [02miraheze/ManageWiki] 07Universal-Omega 03a9d26c8 - Update ext.managewiki.oouiform.reviewChanges.js [00:27:49] miraheze/ManageWiki - Universal-Omega the build passed. [00:27:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.74, 7.12, 7.84 [00:28:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.27, 6.86, 7.88 [00:28:36] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:28:38] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBvAx [00:28:39] [02miraheze/ManageWiki] 07Universal-Omega 0315d931a - Update ext.managewiki.oouiform.reviewChanges.js [00:29:29] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:29:30] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBvpO [00:29:32] [02miraheze/ManageWiki] 07Universal-Omega 033254f18 - Update ext.managewiki.oouiform.reviewChanges.js [00:29:43] miraheze/ManageWiki - Universal-Omega the build passed. [00:29:56] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.89, 8.26, 8.17 [00:30:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.66, 7.35, 7.92 [00:30:23] miraheze/ManageWiki - Universal-Omega the build passed. [00:31:05] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [00:31:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.42, 7.58, 7.94 [00:32:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.11, 7.29, 7.83 [00:33:56] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.63, 8.99, 8.42 [00:34:06] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBfUv [00:34:07] [02miraheze/ManageWiki] 07Universal-Omega 033599e21 - Update ext.managewiki.oouiform.reviewChanges.js [00:34:09] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:34:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.03, 8.63, 8.25 [00:35:04] miraheze/ManageWiki - Universal-Omega the build passed. [00:36:03] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBfIB [00:36:04] [02miraheze/ManageWiki] 07Universal-Omega 03a0161fd - Update ext.managewiki.oouiform.reviewChanges.js [00:36:06] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:36:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.66, 7.57, 7.92 [00:37:03] miraheze/ManageWiki - Universal-Omega the build passed. [00:38:01] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBfqi [00:38:02] [02miraheze/ManageWiki] 07Universal-Omega 038ef14aa - Update ext.managewiki.oouiform.reviewChanges.js [00:38:04] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:39:00] miraheze/ManageWiki - Universal-Omega the build passed. [00:39:33] [02ManageWiki] 07Universal-Omega closed pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:39:35] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vpSns [00:39:37] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [00:39:58] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.66, 6.64, 7.65 [00:40:57] [02miraheze/ManageWiki] 07Universal-Omega pushed 038 commits to 03Universal-Omega-patch-3 [+0/-0/±8] 13https://git.io/JBfsl [00:40:59] [02miraheze/ManageWiki] 07Universal-Omega 03f0c88c1 - Add more information to dialog [00:41:00] [02miraheze/ManageWiki] 07Universal-Omega 035b4237e - Update ext.managewiki.oouiform.reviewChanges.js [00:41:02] [02miraheze/ManageWiki] 07Universal-Omega 03a9d26c8 - Update ext.managewiki.oouiform.reviewChanges.js [00:41:03] [02miraheze/ManageWiki] ... and 5 more commits. [00:41:05] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vpSns [00:41:06] [02ManageWiki] 07Universal-Omega reopened pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:41:37] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBfG0 [00:41:39] [02miraheze/ManageWiki] 07Universal-Omega 03673c3d5 - Update ext.managewiki.oouiform.reviewChanges.js [00:41:40] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:42:38] miraheze/ManageWiki - Universal-Omega the build passed. [00:43:17] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:43:19] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBfc0 [00:43:20] [02miraheze/ManageWiki] 07Universal-Omega 03ba29a57 - Update ext.managewiki.oouiform.reviewChanges.js [00:44:15] miraheze/ManageWiki - Universal-Omega the build passed. [00:46:10] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.90, 5.71, 6.74 [00:46:27] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBfBl [00:46:29] [02miraheze/ManageWiki] 07Universal-Omega 03ff5b5da - Update ext.managewiki.oouiform.reviewChanges.js [00:46:30] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:47:22] miraheze/ManageWiki - Universal-Omega the build passed. [00:48:17] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBfEa [00:48:18] [02miraheze/ManageWiki] 07Universal-Omega 0357b6ce7 - Update ext.managewiki.oouiform.reviewChanges.js [00:48:20] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:49:13] miraheze/ManageWiki - Universal-Omega the build passed. [00:49:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.58, 7.44, 7.53 [00:50:20] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:50:22] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBf2l [00:50:23] [02miraheze/ManageWiki] 07Universal-Omega 035d1f9cb - Update ext.managewiki.oouiform.reviewChanges.js [00:51:19] miraheze/ManageWiki - Universal-Omega the build passed. [00:52:00] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBfw3 [00:52:01] [02miraheze/ManageWiki] 07Universal-Omega 0394c1fb7 - Update ext.managewiki.oouiform.reviewChanges.js [00:52:03] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:52:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.94, 6.87, 6.91 [00:52:57] miraheze/ManageWiki - Universal-Omega the build passed. [00:53:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.96, 7.74, 7.67 [00:54:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.23, 6.35, 6.72 [00:54:23] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBf6B [00:54:25] [02miraheze/ManageWiki] 07Universal-Omega 03e6d7779 - Update ext.managewiki.oouiform.reviewChanges.js [00:54:26] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:55:22] miraheze/ManageWiki - Universal-Omega the build passed. [00:58:37] PROBLEM - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.qadrishattari.xyz' expires in 10 day(s) (Sat 07 Aug 2021 10:28:46 GMT +0000). [00:58:52] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBf9H [00:58:54] [02miraheze/ManageWiki] 07Universal-Omega 03e20cbdf - Update ext.managewiki.oouiform.reviewChanges.js [00:58:55] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [00:59:51] miraheze/ManageWiki - Universal-Omega the build passed. [01:00:06] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBf72 [01:00:07] [02miraheze/ManageWiki] 07Universal-Omega 038b86717 - Update ext.managewiki.oouiform.reviewChanges.js [01:00:09] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [01:00:59] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBfdB [01:01:00] [02miraheze/ManageWiki] 07Universal-Omega 038b91ded - Update ext.managewiki.oouiform.reviewChanges.js [01:01:02] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [01:01:04] miraheze/ManageWiki - Universal-Omega the build passed. [01:01:59] miraheze/ManageWiki - Universal-Omega the build passed. [01:03:59] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.88, 8.21, 7.47 [01:05:56] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBJf4 [01:05:58] [02miraheze/ManageWiki] 07Universal-Omega 037f96b75 - Update ext.managewiki.oouiform.reviewChanges.js [01:05:59] [02ManageWiki] 07Universal-Omega synchronize pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [01:06:48] miraheze/ManageWiki - Universal-Omega the build passed. [01:06:55] [02ManageWiki] 07Universal-Omega closed pull request 03#293: Add more information to dialog - 13https://git.io/JBvDk [01:06:57] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vpSns [01:06:59] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [01:07:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.25, 7.02, 7.22 [01:13:57] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.57, 6.14, 6.73 [01:22:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.28, 6.90, 6.40 [01:24:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.78, 6.51, 6.33 [01:25:07] PROBLEM - wiki.thefactoryhka.com.pa - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.thefactoryhka.com.pa All nameservers failed to answer the query. [01:32:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.76, 6.93, 6.62 [01:34:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.99, 7.39, 6.64 [01:34:33] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.52, 8.27, 7.15 [01:36:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.45, 7.24, 6.67 [01:36:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.90, 7.75, 7.09 [01:38:10] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.50, 6.75, 6.57 [01:39:43] PROBLEM - wiki.thefactoryhka.com.pa - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.que [01:39:43] stname, 'A')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.003994464874268 seconds [01:42:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.11, 2.84 [01:42:15] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.52, 6.65, 6.77 [01:45:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.35, 2.45, 2.65 [01:46:46] PROBLEM - wiki.thefactoryhka.com.pa - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.thefactoryhka.com.pa All nameservers failed to answer the query. [02:00:41] RECOVERY - wiki.thefactoryhka.com.pa - reverse DNS on sslhost is OK: SSL OK - wiki.thefactoryhka.com.pa reverse DNS resolves to cp13.miraheze.org - CNAME OK [02:05:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.86, 6.47, 6.13 [02:06:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.42, 6.87, 6.11 [02:07:57] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.51, 5.80, 5.93 [02:08:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.91, 6.54, 6.08 [03:05:17] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.73, 7.72, 6.43 [03:06:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.88, 6.85, 6.27 [03:07:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.52, 6.98, 6.33 [03:08:29] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.78, 6.15, 6.08 [03:09:12] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.20, 6.41, 6.20 [03:23:39] RECOVERY - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Wed 27 Jul 2022 23:59:59 GMT +0000. [03:32:37] PROBLEM - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.qadrishattari.xyz' expires in 10 day(s) (Sat 07 Aug 2021 10:28:46 GMT +0000). [04:28:09] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.49, 7.23, 5.85 [04:28:30] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.51, 7.96, 6.20 [04:30:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.78, 7.48, 6.12 [04:33:57] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.96, 9.20, 7.09 [04:34:12] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.06, 6.81, 5.75 [04:36:12] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.42, 6.61, 5.80 [04:50:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.78, 20.42, 18.35 [04:52:36] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.01, 18.72, 17.97 [05:08:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.28, 7.05, 7.87 [05:11:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.81, 7.00, 7.99 [05:22:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.73, 5.61, 6.61 [05:23:57] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.82, 5.47, 6.68 [05:25:07] !log [reception@jobrunner4] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=mojwlasnyswiatwiki /home/reception/plmojwlasnyswiat_pages_current.xml --username-prefix wikia:moj-wlasny-swiat [05:25:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:28:41] !log DELETED and DROPPED ucroniaswiki and hispano76wiki [05:28:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:30:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBqUX [05:30:13] [02miraheze/services] 07MirahezeSSLBot 030f0ae1d - BOT: Updating services config for wikis [05:40:40] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=ucroniaswiki [05:40:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:40:46] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=hispano76wiki [05:40:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:41:49] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/populateMainPage.php --wiki hispano76wiki [05:41:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:41:54] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/populateMainPage.php --wiki ucroniaswiki [05:41:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:47:33] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBqBb [05:47:35] [02miraheze/CreateWiki] 07Universal-Omega 031a3ac46 - Reset wiki cache when creating a wiki. [05:47:36] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vpJTL [05:47:38] [02CreateWiki] 07Universal-Omega opened pull request 03#230: Reset wiki cache when creating a wiki. - 13https://git.io/JBqRU [05:47:58] [02CreateWiki] 07Universal-Omega edited pull request 03#230: Reset wiki cache when creating a wiki. - 13https://git.io/JBqRU [05:48:37] miraheze/CreateWiki - Universal-Omega the build passed. [06:00:43] !log MariaDB [mhglobal]> DELETE FROM localuser WHERE lu_wiki = 'cnkwikiwiki'; [06:00:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:01:57] !log MariaDB [mhglobal]> DELETE FROM localuser WHERE lu_wiki = 'kubixwiki'; [06:02:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:02:38] !log MariaDB [mhglobal]> DELETE FROM localuser WHERE lu_wiki = 'zhinawikiwiki'; [06:02:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:13:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBqNJ [06:13:26] [02miraheze/mw-config] 07Universal-Omega 036d521a1 - Fix key to repoSiteName instead of repoName for FAMEData [06:13:28] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [06:13:29] [02mw-config] 07Universal-Omega opened pull request 03#4014: Fix key to repoSiteName instead of repoName for FAMEData - 13https://git.io/JBqNL [06:14:30] miraheze/mw-config - Universal-Omega the build passed. [06:15:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBqxl [06:15:15] [02miraheze/mw-config] 07Universal-Omega 03d4c3838 - Update Wikibase.php [06:15:17] [02mw-config] 07Universal-Omega synchronize pull request 03#4014: Fix key to repoSiteName instead of repoName for FAMEData - 13https://git.io/JBqNL [06:15:29] [02mw-config] 07Universal-Omega edited pull request 03#4014: Set repoSiteName for FAMEData - 13https://git.io/JBqNL [06:16:01] [02mw-config] 07Universal-Omega edited pull request 03#4014: T7565: Set repoSiteName for FAMEData - 13https://git.io/JBqNL [06:16:19] miraheze/mw-config - Universal-Omega the build passed. [06:20:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.39, 6.44, 5.65 [06:22:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBmJx [06:22:07] [02miraheze/mw-config] 07Universal-Omega 03eded13a - Update Wikibase.php [06:22:08] [02mw-config] 07Universal-Omega synchronize pull request 03#4014: T7565: Set repoSiteName for FAMEData - 13https://git.io/JBqNL [06:22:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBmUq [06:22:23] [02miraheze/mw-config] 07Universal-Omega 035c69eee - Update Wikibase.php [06:22:24] [02mw-config] 07Universal-Omega synchronize pull request 03#4014: T7565: Set repoSiteName for FAMEData - 13https://git.io/JBqNL [06:22:35] [02mw-config] 07Universal-Omega edited pull request 03#4014: T7565: Set repoSiteName for FAMEPedia - 13https://git.io/JBqNL [06:22:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.83, 6.25, 5.68 [06:23:07] miraheze/mw-config - Universal-Omega the build passed. [06:23:18] miraheze/mw-config - Universal-Omega the build passed. [06:27:04] [02mw-config] 07Reception123 closed pull request 03#4014: T7565: Set repoSiteName for FAMEPedia - 13https://git.io/JBqNL [06:27:05] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBmL5 [06:27:07] [02miraheze/mw-config] 07Universal-Omega 033734164 - T7565: Set repoSiteName for FAMEPedia (#4014) [06:27:08] [02miraheze/mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-3 [06:27:10] [02mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [06:27:53] miraheze/mw-config - Reception123 the build passed. [06:29:59] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=asterixwiki /home/reception/frasterix_pages_full.xml --username-prefix wikia:asterix [06:30:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:46:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.58, 20.58, 17.89 [06:48:37] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.12, 19.22, 17.71 [07:03:21] PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.26 ms [07:05:25] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 91.75 ms [09:01:31] For information on the log out of all users, See https://meta.miraheze.org/wiki/Community_noticeboard#SRE_Statement_-_User_Logouts_(T7701) [09:01:40] [url] Community noticeboard - Miraheze Meta | meta.miraheze.org [09:05:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JB3UW [09:05:14] [02miraheze/services] 07MirahezeSSLBot 03462cd5c - BOT: Updating services config for wikis [10:34:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.06, 7.03, 5.86 [10:36:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.73, 7.38, 6.19 [10:36:49] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.22, 6.39, 5.77 [10:38:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.04, 6.21, 5.90 [11:04:48] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.89, 6.61, 5.75 [11:06:44] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.44, 5.76, 5.54 [11:46:16] [02dns] 07RhinosF1 opened pull request 03#210: miraheze.org: add my google sites key - 13https://git.io/JBG6G [11:46:24] Reception123: ^ [12:04:54] !log GDPR script [12:04:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:05:07] [02dns] 07Reception123 closed pull request 03#210: miraheze.org: add my google sites key - 13https://git.io/JBG6G [12:05:09] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBGdg [12:05:10] [02miraheze/dns] 07RhinosF1 03f3e3d14 - miraheze.org: add my google sites key (#210) [12:05:57] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.06, 6.67, 5.47 [12:06:06] Reception123: we need to chat about puppet user [12:06:59] ah yes [12:07:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.03, 6.64, 5.60 [12:11:56] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.53, 7.19, 6.04 [12:13:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.65, 6.91, 6.08 [12:15:58] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.34, 6.25, 5.93 [12:20:39] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 211.77 ms [12:20:59] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [12:21:00] PROBLEM - ns2 Auth DNS on ns2 is CRITICAL: Connection to DNS 51.195.236.218 was refused [12:21:11] PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 215.16 ms [12:22:38] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 90.33 ms [12:23:14] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 93.16 ms [12:27:29] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [12:27:46] Reception123: ^ [12:28:17] Change the ' before google to " [12:35:52] But a Google site key can actually be added via ManageWiki right? [12:45:47] Not for miraheze.org [12:45:52] Only for a specific subdomain [12:46:04] I have magic powers to see any wiki without a custom domain [12:48:57] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:51:19] Why puppet ran [12:51:27] Err if syntax broke then that bad [12:51:29] Reception123: [12:55:28] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:00:40] RhinosF1: let me check [13:02:16] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBZrv [13:02:17] [02miraheze/dns] 07Reception123 03c8af892 - syntax [13:06:58] RhinosF1: Okay [13:19:00] RECOVERY - ns2 Auth DNS on ns2 is OK: DNS OK: 0.008 seconds response time. miraheze.org returns 2001:41d0:801:2000::58af,51.38.69.175 [13:40:15] Reception123: works [13:49:07] great [14:02:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.98, 19.86, 17.57 [14:04:48] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.93, 19.97, 17.92 [14:04:59] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.43, 7.29, 6.19 [14:05:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.43, 7.00, 6.11 [14:06:57] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.00, 7.00, 6.21 [14:07:57] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.33, 6.52, 6.05 [14:08:54] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.37, 6.50, 6.13 [14:21:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.49, 7.52, 6.73 [14:21:57] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.07, 7.79, 6.59 [14:23:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.54, 8.24, 7.08 [14:25:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.29, 7.68, 6.83 [14:33:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.12, 8.50, 7.50 [14:37:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.13, 7.19, 7.24 [14:41:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.87, 6.95, 7.69 [14:43:57] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.27, 5.81, 6.60 [14:49:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.97, 6.75, 6.68 [14:50:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.08, 6.06, 6.80 [14:51:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.56, 6.30, 6.53 [14:54:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.76, 6.67, 6.88 [14:56:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.89, 6.32, 6.72 [15:05:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.90, 7.25, 6.89 [15:11:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.44, 6.78, 6.80 [15:15:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.27, 6.76, 6.76 [15:17:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.44, 6.43, 6.62 [15:20:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.51, 7.02, 6.55 [15:24:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.54, 6.61, 6.52 [16:00:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.28, 6.53, 6.05 [16:02:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.51, 6.16, 5.98 [16:26:01] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 152.79 ms [16:27:14] PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 157.18 ms [16:31:22] PROBLEM - ping6 on dbbackup1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 207.94 ms [16:33:29] PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 162.60 ms [16:35:32] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 99.42 ms [16:35:58] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 89.78 ms [16:36:18] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.17, 6.83, 6.22 [16:37:03] Voidwalker: is ping6 a concern [16:38:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.69, 6.07, 6.01 [16:39:18] Probably not, swift recovery makes it seem like a temporary issue in the DC [16:39:54] both being ramnode makes this more likely as well [16:44:30] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.88, 20.81, 17.83 [16:46:25] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.40, 18.18, 17.23 [16:50:00] Voidwalker: ramnode v6 has been getting flappy [16:54:28] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.41, 1.92, 1.47 [16:56:27] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.84, 1.53, 1.38 [16:58:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.54, 7.11, 6.27 [17:00:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.97, 7.26, 6.44 [17:00:46] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.02, 7.12, 6.57 [17:04:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.90, 7.73, 6.79 [17:06:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.23, 7.19, 6.72 [17:06:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.60, 8.37, 7.27 [17:08:08] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.21, 6.78, 6.62 [17:08:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.70, 7.44, 7.06 [17:12:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.72, 6.50, 6.76 [17:20:29] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBCsi [17:20:30] [02miraheze/ManageWiki] 07Universal-Omega 03c7ff01c - Update ext.managewiki.oouiform.confirmClose.js [17:20:32] [02ManageWiki] 07Universal-Omega synchronize pull request 03#290: Add review changes dialog - 13https://git.io/J4Xld [17:21:34] miraheze/ManageWiki - Universal-Omega the build passed. [17:22:00] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBCGJ [17:22:01] [02miraheze/ManageWiki] 07Universal-Omega 036c44263 - Update ext.managewiki.oouiform.reviewChanges.js [17:22:03] [02ManageWiki] 07Universal-Omega synchronize pull request 03#290: Add review changes dialog - 13https://git.io/J4Xld [17:22:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.86, 7.69, 6.78 [17:22:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.94, 6.80, 6.71 [17:22:59] miraheze/ManageWiki - Universal-Omega the build passed. [17:24:40] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBCZe [17:24:41] [02miraheze/ManageWiki] 07Universal-Omega 037ead591 - Update ext.managewiki.oouiform.reviewChanges.js [17:24:43] [02ManageWiki] 07Universal-Omega synchronize pull request 03#290: Add review changes dialog - 13https://git.io/J4Xld [17:24:55] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.54, 7.32, 6.88 [17:25:07] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBCZt [17:25:08] [02miraheze/ManageWiki] 07Universal-Omega 035669ad2 - Update en.json [17:25:10] [02ManageWiki] 07Universal-Omega synchronize pull request 03#290: Add review changes dialog - 13https://git.io/J4Xld [17:25:34] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBCZY [17:25:34] miraheze/ManageWiki - Universal-Omega the build passed. [17:25:35] [02miraheze/ManageWiki] 07Universal-Omega 0387ef39a - Update qqq.json [17:25:37] [02ManageWiki] 07Universal-Omega synchronize pull request 03#290: Add review changes dialog - 13https://git.io/J4Xld [17:26:00] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBCZR [17:26:01] [02miraheze/ManageWiki] 07Universal-Omega 034636d48 - Update extension.json [17:26:03] [02ManageWiki] 07Universal-Omega synchronize pull request 03#290: Add review changes dialog - 13https://git.io/J4Xld [17:26:05] miraheze/ManageWiki - Universal-Omega the build passed. [17:26:36] miraheze/ManageWiki - Universal-Omega the build passed. [17:26:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.62, 7.53, 7.02 [17:26:58] miraheze/ManageWiki - Universal-Omega the build passed. [17:27:37] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBCZb [17:27:38] [02miraheze/ManageWiki] 07Universal-Omega 0300dcece - Update ext.managewiki.oouiform.reviewChanges.js [17:27:40] [02ManageWiki] 07Universal-Omega synchronize pull request 03#290: Add review changes dialog - 13https://git.io/J4Xld [17:28:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.00, 7.52, 7.04 [17:28:37] miraheze/ManageWiki - Universal-Omega the build passed. [17:34:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.80, 8.65, 7.51 [17:34:33] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.33, 9.35, 7.82 [17:34:41] [02ManageWiki] 07Universal-Omega commented on pull request 03#290: Add review changes dialog - 13https://git.io/JBCcS [17:34:52] [02ManageWiki] 07Universal-Omega closed pull request 03#290: Add review changes dialog - 13https://git.io/J4Xld [17:34:54] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+2/-1/±5] 13https://git.io/JBCcA [17:34:55] [02miraheze/ManageWiki] 07Universal-Omega 03597c71b - Add review changes dialog (#290) [17:34:57] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vpSns [17:34:58] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [17:35:56] miraheze/ManageWiki - Universal-Omega the build passed. [17:38:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.66, 7.93, 7.50 [17:38:39] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JBCWI [17:38:41] [02miraheze/mediawiki] 07Universal-Omega 03307ab07 - Update ManageWiki [17:40:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.81, 7.85, 7.70 [17:44:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.23, 8.46, 7.76 [17:44:14] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.64, 8.63, 8.05 [17:48:55] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.68, 7.14, 6.34 [17:49:32] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.47, 7.81, 6.45 [17:50:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.98, 21.39, 18.74 [17:51:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.84, 7.86, 6.64 [17:52:44] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.59, 6.73, 6.36 [17:53:13] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:54:14] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:56:14] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:56:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.71, 24.17, 20.75 [17:56:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.35, 7.12, 6.61 [17:57:08] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:00:21] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.11, 6.63, 6.52 [18:03:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.65, 6.52, 6.59 [18:06:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.38.69.175/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [18:06:35] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.56, 6.55, 4.20 [18:06:57] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [18:07:01] uh, what [18:07:04] PROBLEM - worldsanskrit.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for worldsanskrit.net could not be found [18:07:04] PROBLEM - wiki.wysteria.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wysteria.ga could not be found [18:07:13] * RhinosF1 here [18:07:23] Voidwalker: we lost 2 cps [18:08:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:08:09] PROBLEM - sopel.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sopel.bots.miraheze.wiki could not be found [18:08:13] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.wikimicrofinanza.it could not be found [18:08:16] PROBLEM - en.famepedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.famepedia.org could not be found [18:08:21] PROBLEM - tot.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tot.wiki could not be found [18:08:40] Voidwalker: cp12/13 [18:08:50] why are ns1/2 different depools [18:08:55] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:09:09] disk usage is weird [18:09:14] mw8 [18:09:18] don't know, by all rights, it doesn't make sense, they were both up for me by the time I checked [18:09:35] what is 128.199.139.216? [18:10:28] cp3 [18:10:31] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.60, 5.60, 4.38 [18:10:59] weird, that IP doesn't resolve for me [18:11:50] nothing in dmesg [18:12:09] Voidwalker: nginx on mw8 & mw11 crashed [18:12:23] ah [18:13:22] Voidwalker: might wanna move to -security [18:13:54] RECOVERY - worldsanskrit.net - reverse DNS on sslhost is OK: SSL OK - worldsanskrit.net reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [18:13:54] RECOVERY - wiki.wysteria.ga - reverse DNS on sslhost is OK: SSL OK - wiki.wysteria.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [18:14:28] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.40, 4.35, 4.17 [18:14:56] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp13.miraheze.org - CNAME OK [18:15:00] RECOVERY - en.famepedia.org - reverse DNS on sslhost is OK: SSL OK - en.famepedia.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [18:15:06] RECOVERY - sopel.bots.miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - sopel.bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [18:15:13] RECOVERY - tot.wiki - reverse DNS on sslhost is OK: SSL OK - tot.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [18:19:40] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki=loginwiki on mw* / jbr* / test3 [18:20:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:21:11] !log restart mw11 php-fpm because no workers [18:21:17] Universal_Omega: can you hold on mw11 [18:21:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:21:46] PROBLEM - olwest.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for olwest.org could not be found [18:21:54] PROBLEM - rct.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rct.wiki could not be found [18:21:55] Universal_Omega: i got no php workers [18:21:58] RhinosF1: it already finished everywhere. [18:22:02] PROBLEM - orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for orain.org could not be found [18:22:05] PROBLEM - dcmultiverse.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dcmultiverse.miraheze.org could not be found [18:22:06] PROBLEM - www.lab612.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.lab612.at could not be found [18:22:11] ^ aware [18:22:11] PROBLEM - archive.a2b2.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for archive.a2b2.org could not be found [18:22:13] PROBLEM - ethene.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ethene.wiki could not be found [18:22:14] !log and mw8 [18:22:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:22:20] PROBLEM - wiki.slimeapedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.slimeapedia.org could not be found [18:22:21] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zymonic.com could not be found [18:22:22] PROBLEM - civwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for civwiki.org could not be found [18:22:23] PROBLEM - munwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for munwiki.info could not be found [18:22:24] PROBLEM - planetnomads.speedbomb.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for planetnomads.speedbomb.de could not be found [18:26:20] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.21, 20.71, 23.58 [18:28:36] RECOVERY - rct.wiki - reverse DNS on sslhost is OK: SSL OK - rct.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [18:28:42] RECOVERY - olwest.org - reverse DNS on sslhost is OK: SSL OK - olwest.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [18:28:53] RECOVERY - archive.a2b2.org - reverse DNS on sslhost is OK: SSL OK - archive.a2b2.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [18:28:54] RECOVERY - orain.org - reverse DNS on sslhost is OK: SSL OK - orain.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [18:28:55] RECOVERY - ethene.wiki - reverse DNS on sslhost is OK: SSL OK - ethene.wiki reverse DNS resolves to cp13.miraheze.org - CNAME OK [18:29:02] RECOVERY - dcmultiverse.miraheze.org - reverse DNS on sslhost is OK: SSL OK - dcmultiverse.miraheze.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [18:29:03] RECOVERY - www.lab612.at - reverse DNS on sslhost is OK: SSL OK - www.lab612.at reverse DNS resolves to cp13.miraheze.org - CNAME OK [18:29:08] RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: SSL OK - wiki.zymonic.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [18:29:09] RECOVERY - wiki.slimeapedia.org - reverse DNS on sslhost is OK: SSL OK - wiki.slimeapedia.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [18:29:14] RECOVERY - civwiki.org - reverse DNS on sslhost is OK: SSL OK - civwiki.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [18:29:16] RECOVERY - munwiki.info - reverse DNS on sslhost is OK: SSL OK - munwiki.info reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [18:29:17] RECOVERY - planetnomads.speedbomb.de - reverse DNS on sslhost is OK: SSL OK - planetnomads.speedbomb.de reverse DNS resolves to cp13.miraheze.org - CNAME OK [18:34:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.15, 7.11, 6.52 [18:36:35] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.23, 7.25, 6.62 [18:38:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.95, 7.34, 6.75 [18:40:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 11.46, 16.61, 19.97 [18:40:23] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.11, 6.41, 6.47 [18:46:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.39, 6.74, 7.99 [18:48:10] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.33, 7.09, 7.95 [18:49:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.49, 7.18, 7.99 [18:54:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.03, 7.22, 7.83 [18:59:59] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.73, 7.99, 7.95 [19:01:58] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.12, 7.68, 7.84 [19:02:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.82, 7.92, 7.70 [19:05:57] PROBLEM - www.fossilfighterswiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.fossilfighterswiki.com could not be found [19:06:11] PROBLEM - wiki.redeemer.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.redeemer.live could not be found [19:06:16] PROBLEM - matttest.mtwiki.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for matttest.mtwiki.cf could not be found [19:06:17] PROBLEM - wildcard.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wildcard.bots.miraheze.wiki could not be found [19:08:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.62, 7.32, 7.64 [19:11:17] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.78, 1.47, 1.15 [19:12:38] RECOVERY - www.fossilfighterswiki.com - reverse DNS on sslhost is OK: SSL OK - www.fossilfighterswiki.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [19:12:57] RECOVERY - wildcard.bots.miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - wildcard.bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [19:13:06] RECOVERY - wiki.redeemer.live - reverse DNS on sslhost is OK: SSL OK - wiki.redeemer.live reverse DNS resolves to cp13.miraheze.org - CNAME OK [19:13:12] RECOVERY - matttest.mtwiki.cf - reverse DNS on sslhost is OK: SSL OK - matttest.mtwiki.cf reverse DNS resolves to cp13.miraheze.org - CNAME OK [19:13:14] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.08, 1.36, 1.15 [19:26:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.03, 6.19, 6.72 [19:27:57] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.11, 5.65, 6.71 [19:30:35] PROBLEM - www.rothwell-leeds.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.rothwell-leeds.co.uk could not be found [19:35:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.86, 7.03, 6.92 [19:36:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.82, 7.24, 6.96 [19:37:24] RECOVERY - www.rothwell-leeds.co.uk - reverse DNS on sslhost is OK: SSL OK - www.rothwell-leeds.co.uk reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [19:37:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.76, 6.53, 6.74 [19:42:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.75, 6.61, 6.79 [19:46:28] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.52, 1.72, 1.30 [19:48:27] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.86, 1.37, 1.22 [19:49:59] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.95, 7.64, 7.10 [19:51:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.56, 7.61, 7.16 [19:52:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.13, 6.90, 6.69 [19:55:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBCxK [19:55:13] [02miraheze/services] 07MirahezeSSLBot 03302dbc7 - BOT: Updating services config for wikis [19:55:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.42, 7.72, 7.30 [19:58:10] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.43, 6.73, 6.69 [20:00:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBChk [20:00:14] [02miraheze/services] 07MirahezeSSLBot 038fe2413 - BOT: Updating services config for wikis [20:01:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.98, 7.92, 7.60 [20:02:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.04, 7.50, 7.02 [20:03:56] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.41, 8.99, 8.01 [20:09:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.77, 7.49, 7.69 [20:10:37] Voidwalker: it would be good to have a future-goals tag [20:10:51] For things we're not doing in this goal period but might think about next [20:11:04] Re the sec patch task as an example [20:13:56] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.28, 7.40, 7.56 [20:14:46] mmm, I might work on it when I have time though, but we'll see [20:15:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.74, 6.78, 7.32 [20:16:16] Voidwalker: ok [20:16:23] We could steal stuff from scap [20:19:59] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.89, 8.27, 7.73 [20:23:58] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.08, 7.69, 7.62 [20:30:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.09, 7.30, 7.83 [20:33:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.93, 8.07, 7.68 [20:34:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.58, 8.20, 7.99 [20:37:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.36, 7.82, 7.70 [20:38:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.53, 7.71, 7.85 [20:42:08] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.02, 7.94, 7.89 [20:42:09] [02miraheze/mediawiki] 07The-Voidwalker pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JBWIX [20:42:11] [02miraheze/mediawiki] 07The-Voidwalker 03caac8be - Update Loops [20:42:49] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://git.io/JBWIQ [20:42:50] [02miraheze/mw-config] 07The-Voidwalker 034e7e49a - enable Loops [20:42:51] [02mw-config] 07The-Voidwalker created branch 03The-Voidwalker-patch-1 - 13https://git.io/vbvb3 [20:43:00] [02mw-config] 07The-Voidwalker opened pull request 03#4015: enable Loops - 13https://git.io/JBWId [20:43:56] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.29, 7.99, 7.75 [20:44:01] miraheze/mw-config - The-Voidwalker the build passed. [20:44:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.57, 7.85, 7.88 [20:45:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.69, 7.28, 7.53 [20:48:59] [02mw-config] 07The-Voidwalker closed pull request 03#4015: enable Loops - 13https://git.io/JBWId [20:49:00] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBWtr [20:49:02] [02miraheze/mw-config] 07The-Voidwalker 0356de913 - enable Loops (#4015) [20:49:04] [02miraheze/mw-config] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 [20:49:06] [02mw-config] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 - 13https://git.io/vbvb3 [20:50:03] miraheze/mw-config - The-Voidwalker the build passed. [20:57:55] [02puppet] 07RhinosF1 opened pull request 03#1825: puppet: rebalance MediaWiki puppet times - 13https://git.io/JBWYw [20:57:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.49, 5.59, 6.56 [20:58:58] [02puppet] 07RhinosF1 synchronize pull request 03#1825: puppet: rebalance MediaWiki puppet times - 13https://git.io/JBWYw [20:59:16] [02puppet] 07RhinosF1 synchronize pull request 03#1825: puppet: rebalance MediaWiki puppet times - 13https://git.io/JBWYw [20:59:40] [02puppet] 07RhinosF1 synchronize pull request 03#1825: puppet: rebalance MediaWiki puppet times - 13https://git.io/JBWYw [21:00:09] [02puppet] 07RhinosF1 synchronize pull request 03#1825: puppet: rebalance MediaWiki puppet times - 13https://git.io/JBWYw [21:00:23] [02puppet] 07RhinosF1 synchronize pull request 03#1825: puppet: rebalance MediaWiki puppet times - 13https://git.io/JBWYw [21:01:49] [02puppet] 07RhinosF1 synchronize pull request 03#1825: puppet: rebalance MediaWiki puppet times - 13https://git.io/JBWYw [21:02:32] [02puppet] 07RhinosF1 synchronize pull request 03#1825: puppet: rebalance MediaWiki puppet times - 13https://git.io/JBWYw [21:03:48] [02puppet] 07RhinosF1 edited pull request 03#1825: puppet: rebalance MediaWiki puppet times - 13https://git.io/JBWYw [21:03:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.21, 7.47, 6.96 [21:04:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.18, 8.60, 7.56 [21:04:19] [02puppet] 07RhinosF1 edited pull request 03#1825: puppet: rebalance MediaWiki puppet times - 13https://git.io/JBWYw [21:05:03] Oh hi JohnLewis [21:05:16] hi [21:07:44] Is there anything we can help with? [21:08:02] no? [21:10:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.05, 20.94, 18.16 [21:10:31] Ok [21:12:15] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:12:20] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.33, 19.13, 17.86 [21:12:35] Voidwalker: I guess mw8 is stuck too? [21:12:46] I can look in a sec if needed [21:13:08] no, my command hasn't updated, like, at all, so I'm probably going to cancel, and do manually [21:13:12] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:13:33] oh, looks like it updated :P [21:13:40] Voidwalker: with a fail? [21:13:58] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.79, 7.74, 7.68 [21:14:08] no, mw8 and mw9 started before my script got to them [21:14:19] Ok [21:14:24] running puppet on mw9 [21:14:29] Ack [21:16:17] looks like it applied the first time, odd [21:16:46] anyway running on mw8 now [21:17:08] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:17:57] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.89, 9.02, 8.18 [21:18:12] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [21:19:50] [02puppet] 07The-Voidwalker closed pull request 03#1825: puppet: rebalance MediaWiki puppet times - 13https://git.io/JBWYw [21:19:52] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/JBWn0 [21:19:53] [02miraheze/puppet] 07RhinosF1 0380dce30 - puppet: rebalance MediaWiki puppet times (#1825) [21:24:00] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.60, 7.56, 7.83 [21:26:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.44, 7.61, 7.95 [21:26:22] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.90, 19.53, 17.95 [21:27:52] hopefully load alerts when puppet runs won't happen at :30 [21:27:57] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.53, 8.08, 7.96 [21:28:10] nothing is there [21:30:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.23, 8.36, 8.11 [21:30:20] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.27, 19.10, 18.16 [21:30:38] You do realise by merging that change you don’t solve any problems but introduce new ones? [21:31:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.33, 7.88, 7.99 [21:32:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.70, 7.66, 7.89 [21:32:11] Servers drift apart in terms of synchronicity, in order to have any benefit of load you’d need minutes of at least 5-10 different between runs. Puppet introducing load also is a given, mitigating by changing it is pointless when it’s a capacity issue instead [21:33:02] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03revert-1825-patch-48 [+0/-0/±6] 13https://git.io/JBWWO [21:33:04] [02miraheze/puppet] 07The-Voidwalker 03f2548b1 - Revert "puppet: rebalance MediaWiki puppet times (#1825)" [21:33:25] We know capacity is virtually at maximum but until we get the new infra deployed, very little we can do [21:33:36] [02puppet] 07The-Voidwalker created branch 03revert-1825-patch-48 - 13https://git.io/vbiAS [21:33:38] yeah, worth a shot, but I think reverting is in order [21:33:42] Then attempting to solve the problem is impossible [21:33:44] and no we don't realise [21:33:56] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.50, 8.10, 8.05 [21:34:21] [02puppet] 07The-Voidwalker opened pull request 03#1826: Revert "puppet: rebalance MediaWiki puppet times " - 13https://git.io/JBWW2 [21:34:42] New infra also isn’t required to solve it because the plan I proposed months ago didn’t require new infra, just more efficient use [21:35:48] Hopefully not, I've been a bit worried about the load on cloud4 sometimes, (though this will be an improvement, certainly) [21:35:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.24, 7.27, 7.74 [21:36:22] implementing your plan but because we don't want to go with one less server pooled at any point we're imaging a new 1 before removing an old one i believe [21:36:27] [02puppet] 07The-Voidwalker closed pull request 03#1826: Revert "puppet: rebalance MediaWiki puppet times " - 13https://git.io/JBWW2 [21:36:27] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/JBWWx [21:36:28] [02miraheze/puppet] 07The-Voidwalker 03518b4ac - Revert "puppet: rebalance MediaWiki puppet times (#1825)" (#1826) [21:37:32] I think part of it though is we'll need new IPv4 addresses to accommodate the two new servers, but that shouldn't be a blocker [21:37:45] You don’t need to depool a server to do my plan? The plan lays out imaging and deployment before depooling and jobrunner* isn’t public facing so depooling has no effect [21:38:09] the other is migrating jobrunner3 to jobchron1, as that is the only step that might cause issues [21:38:20] The main blocker is getting SPF to approve it as EM [21:40:42] I was told Void was EM this morning [21:40:52] technically that's me now, as SPF is stepping back from most roles [21:41:10] So you’re EM and SPF is just an SRE? [21:41:29] I need to go through phab and review assigned tasks soon [21:41:32] yes [21:42:10] communication has been a bit spotty, but I suppose that has a bit to do with timezones and general availability [21:42:20] Docs needs updating then [21:42:26] that too [21:43:04] Also it’s no disrespect to you, but from what I’m seeing, your level of understanding of the infra is concerning if you are indeed the EM [21:43:22] I’d question that decision personally [21:43:39] Reception should rather have stepped in [21:44:20] done sysadmin page [21:44:26] quite understandable, but considering I'm the only one really on Infra, and Reception is DSRE and EM for mediawiki, it is what it is at the moment [21:44:50] There’s potentially appropriate candidates for MW EM [21:45:32] But to have someone with little understanding in such an important position is very poor decision making as the EM for Infra holds a lot of decision and key decision making [21:45:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.94, 5.83, 6.73 [21:46:10] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.46, 7.16, 7.22 [21:48:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.82, 7.06, 7.19 [21:50:13] Being an EM isn't just about experience in Mediawiki though [21:50:26] it's people & management skills too [21:50:40] I know, I defined the roles [21:50:52] but having the experience is a big requirement [21:51:27] With no experience, comes no respect, no decision making, no informed decisions - you have a literal manager who’s role is purely hire and fire not one to lead [21:53:30] Plus the workload of Infra has gone from being tightly managed to be so loose there’s little progress - having an EM is useless without a team so the focus should be on progress not managing which is why the DSRE role should have encapsulated the EM role here freeing up an EM position where it adds values [21:53:58] otherwise you have 3 positions covered by 2 people - where one adds no value and the one that adds value can’t add anything [21:54:17] The situation is far from ideal [21:54:36] but that's not because of the title anyone has been given [21:54:42] It’s embarrassingly far from it [21:55:01] no, the titles mean nothing but the meanings of it is what makes it so [21:55:44] There's little that can be said that we aren't aware of [21:55:49] Plus to also have a MWE leading an investigation into an Infra responsibility was also questionable [21:56:00] unfortunately, we ain't going to change things overnight. [21:56:26] The fact I handled a mail incident at all isn't ideal. [21:56:58] There’s a difference between changing things overnight and accepting unideal practise and excusing it [21:58:08] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.77, 6.18, 6.69 [21:59:21] SRE is four active members right now. Of course there is going to be overlap between roles. If there wasn't some things wouldn't get done. There's only so much we can do at the moment that isn't scrambling to keep thing running in the few hours we all devote to the project. [21:59:37] If we did things in an ideal way, we'd still have a mail server spewing spam today. [21:59:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.09, 6.44, 6.51 [22:00:46] Is that a dig? An ideal way meaning a security leak? [22:01:31] the overlap is what needs to end, the Infra EM role needs value adding to it and the DSRE needs to fulfil it until it’s appropriate to have someone in that role, while the EM role which a team needs filing [22:01:56] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.38, 7.04, 6.72 [22:02:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.78, 6.83, 6.87 [22:03:02] Scrambling also wouldn’t be necessary if things were worked on promptly rather than leaving things until last minute - the capacity planning was done to alleviate this exact issue and the team responsible for leading it delayed it arbitrarily until it becomes an unmanageable problem [22:03:34] The problem here is scrambling is the norm, not something to avoid and prevent [22:03:50] It's not a dig. I don't think anyone prior to the report recently acted wrongly. [22:04:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.44, 7.42, 7.08 [22:04:30] Then why mention that if things were ideal it wouldn’t have been found? [22:04:32] At that point, no one on SRE had the knowledge too [22:05:04] which is why it was scrambled and too is. No one understands LDAP + Mail enough. [22:05:18] Which is an exposed issue that needs solving, running critical infra with no one responsible is dangerous [22:05:42] We're aware [22:06:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.81, 7.50, 7.15 [22:06:15] Keeping saying you’re aware, being aware is nothing, acting is different [22:06:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBWgT [22:06:32] [02miraheze/mw-config] 07Universal-Omega 035d5be9b - Set Wikibase configs for FAMEPedia [22:06:33] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [22:06:44] [02mw-config] 07Universal-Omega opened pull request 03#4016: T7565: Set Wikibase configs for FAMEPedia - 13https://git.io/JBWg3 [22:06:51] this is the problem, knowing there is a problem and solving it are two different things, and the project is infamous for knowing but not doing example, capacity planning [22:07:42] miraheze/mw-config - Universal-Omega the build passed. [22:07:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.72, 7.08, 6.87 [22:08:14] We thought we'd manage. We haven't. [22:08:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBW2v [22:08:35] Which is my point [22:08:36] [02miraheze/mw-config] 07Universal-Omega 03f809e2b - Set Wikibase configs for FAMEPedia (#4016) [22:08:37] [02mw-config] 07Universal-Omega closed pull request 03#4016: T7565: Set Wikibase configs for FAMEPedia - 13https://git.io/JBWg3 [22:08:39] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [22:08:40] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [22:08:45] I have no solution for the fact we have no team members with enough experience. [22:08:49] pre-emoting planning doesn’t wait until you can’t, you do it while you can [22:09:05] We've tried recruiting and it failed miserably [22:09:31] miraheze/mw-config - Universal-Omega the build passed. [22:09:54] I’ve not seen any recruitment attempts? [22:10:14] The Mediawiki developer [22:10:49] https://twitter.com/miraheze/status/1382371225083727872 looks like a recruitment attempt [22:10:50] [Twitter] Miraheze (@miraheze): We're currently looking for a volunteer software developer. Please contact us if you are interested in this role, and if not please feel free to share with anyone who you think may be. Details here: https://meta.miraheze.org/wiki/Miraheze_Vacancies#Role:_Trust_and_Safety_Responder | 6 RTs | 5 ♥s | Posted: 2021-04-14 - 12:32:37EDT [22:11:11] !log [rhinos@jobrunner3] sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/migrateAccount.php --wiki=samskritavyakaranamwiki --username=Swarup --auto (END - exit=0) [22:11:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:12:40] So one tweet counts as a recruitment attempt? Barely an attempt and that was before the loss of experience as well [22:13:24] we found one person and did a horrible job at working with them [22:13:26] Recruitment and retention is at a point where it’s been abused and ignored for too long, failing to give it the attention now fully is setting the project up for failure without a doubt [22:13:32] Owen has just started for a DSRE [22:13:59] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.93, 6.13, 6.56 [22:14:09] Which is going to be an extremely difficult one to fill [22:14:32] again, you're pointing out what we already know [22:14:38] not providing solutions [22:14:55] I have tried, multiple times to put in solutions [22:15:27] which is why I left, there was no attempt to try solutions and give the attention it needed to retain volunteers never mind recruit [22:16:57] where that leaves us is somewhere I don't like [22:18:00] This is basically 1/10th of my todo list at the moment [22:18:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.24, 5.73, 6.53 [22:19:40] Which is why when I knew you joined, you wouldn’t bring around the reform necessary, SRE needs external support and reform, by joining you’ve quickly become so overwhelmed with the work you can’t focus on what you said you would do [22:24:04] We need a huge effort if we're going to be viable in the long term. We also need a team that can bridge the gap until we get there and at least keep us online. [22:24:47] From where? From who? I'd like to say you don't have a point, because frankly you do. But at the same time, you are pointing us to working with more people. People we don't have, and don't know how to find. [22:25:16] I've tried to get external support. [22:25:22] They had a lot to say [22:25:53] PROBLEM - cp12 Disk Space on cp12 is WARNING: DISK WARNING - free space: / 4240 MB (10% inode=96%); [22:36:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.51, 6.80, 6.46 [22:36:16] If you have any suggestions, or are willing to help with these problems, then be my guest. Otherwise, provided there are no further security issues in infra, I will be moving forward on some of these as soon as I can get done with the capacity proposal [22:36:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.88, 7.32, 6.61 [22:36:40] In any case, I'll be away for now [22:38:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.16, 6.57, 6.41 [22:40:25] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.27, 6.58, 6.47 [22:40:48] I'm off too [22:41:40] If there's a plan that's gonna work, we'll listen [22:41:58] Otherwise, we're just being taken off trying to keep things going [22:44:58] You won’t listen, didn’t in the past, nothing at all to provide any assurance you’re interested in listening now. Also the time you’ve had to sort things out in the past, it’s laughable- neigh - a joke to say it’s wasting your time. You’ve labelled the community being consulted in the past as wasting your time [22:46:44] to be honest, even this discussion has further clarified me being here is wasting my time as ‘all the problems are known’ but not being given the effort to be fixed. My plans have been shared numerous times, if you want them, go find them in emails etc because I’m not going to do the work for you plus I can’t even access my emails anymore [22:48:04] As long as this attitude exists, you won’t get new volunteers. This exact attitude is the reason why volunteers are hard to convert [22:50:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.13, 6.50, 6.27 [22:52:00] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.48, 6.21, 6.18 [23:32:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.96, 6.03, 5.57 [23:34:03] [02CreateWiki] 07Universal-Omega closed pull request 03#230: Reset wiki cache when creating a wiki. - 13https://git.io/JBqRU [23:34:05] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vpJTL [23:34:06] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [23:34:40] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.81, 7.21, 6.06 [23:35:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.16, 6.93, 6.16 [23:36:37] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.14, 7.03, 6.16 [23:37:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.06, 6.69, 6.17 [23:42:30] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.18, 6.59, 6.25 [23:54:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.10, 6.75, 6.33 [23:56:10] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.10, 5.96, 6.11