[00:02:53] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.07, 1.75, 1.55 [00:02:56] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:49] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.92, 1.52, 1.49 [00:30:57] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:55:16] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.05, 1.77, 1.30 [03:57:16] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.87, 1.48, 1.25 [06:05:34] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 280.72 ms [06:09:41] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 281.92 ms [08:27:16] PROBLEM - mw8 Disk Space on mw8 is WARNING: DISK WARNING - free space: / 2001 MB (10% inode=74%); [08:29:18] RECOVERY - mw8 Disk Space on mw8 is OK: DISK OK - free space: / 2136 MB (11% inode=74%); [10:15:37] !log [reception@jobrunner3] created local account for MH maintenance on trollpastawiki [10:15:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:15:50] !log [reception@jobrunner3] sudo -u www-data php deleteBatch.php -u Miraheze maintenance script -r [[phab:T7602|Requested]] --wiki trollpastawiki /home/reception/trollpastadel.txt [10:15:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:22:11] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki kittyhousewiki /home/reception/delbackups5/kittyhousewiki.xml --username-prefix=kittyhouse [10:22:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:39:32] PROBLEM - mw11 Disk Space on mw11 is CRITICAL: DISK CRITICAL - free space: / 1086 MB (5% inode=75%); [11:41:32] PROBLEM - mw11 Disk Space on mw11 is WARNING: DISK WARNING - free space: / 1306 MB (6% inode=75%); [12:02:17] PROBLEM - mw8 APT on mw8 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [12:04:17] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 62 packages available for upgrade (18 critical updates). [13:47:59] https://meta.miraheze.org/wiki/Community_noticeboard#Cannot_have_recaptcha_when_creating_account @System Administrators theres a case at the cm [13:48:00] [url] Community noticeboard - Miraheze Meta | meta.miraheze.org [15:39:28] [02GlobalNewFiles] 07supertassu reviewed pull request 03#12 commit - 13https://git.io/JcS2R [16:30:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JcSXG [16:30:14] [02miraheze/services] 07MirahezeSSLBot 03eb7b257 - BOT: Updating services config for wikis [17:32:11] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JcS5l [17:32:13] [02miraheze/GlobalNewFiles] 07Universal-Omega 039f15355 - Update GlobalNewFilesHooks.php [17:32:14] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#12: Convert to Jobs - 13https://git.io/JcOx1 [17:32:37] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JcS5B [17:32:38] [02miraheze/GlobalNewFiles] 07Universal-Omega 039135602 - Update GlobalNewFilesMoveJob.php [17:32:40] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#12: Convert to Jobs - 13https://git.io/JcOx1 [17:33:04] miraheze/GlobalNewFiles - Universal-Omega the build passed. [17:33:10] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JcS52 [17:33:11] [02miraheze/GlobalNewFiles] 07Universal-Omega 03dada9c8 - Update GlobalNewFilesHooks.php [17:33:12] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#12: Convert to Jobs - 13https://git.io/JcOx1 [17:33:23] [02GlobalNewFiles] 07Universal-Omega reviewed pull request 03#12 commit - 13https://git.io/JcS5V [17:33:31] miraheze/GlobalNewFiles - Universal-Omega the build passed. [17:34:06] miraheze/GlobalNewFiles - Universal-Omega the build passed. [17:54:08] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.56, 6.68, 5.39 [17:56:07] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.96, 6.18, 5.35 [18:05:59] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.18, 7.88, 6.31 [18:06:05] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.06, 6.89, 6.04 [18:07:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.97, 7.67, 6.43 [18:08:05] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.64, 6.44, 5.98 [18:11:59] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.86, 6.43, 6.18 [20:04:35] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.77, 4.82, 2.69 [20:04:59] PROBLEM - wiki.phgalaxy.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.phgalaxy.eu.org could not be found [20:05:04] PROBLEM - www.erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.erikapedia.com could not be found [20:05:05] PROBLEM - iesd.mobagenie.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for iesd.mobagenie.my.id could not be found [20:05:08] PROBLEM - famedata.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for famedata.org could not be found [20:05:09] PROBLEM - miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.wiki could not be found [20:05:09] PROBLEM - wiki.consid.vn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.consid.vn could not be found [20:05:12] PROBLEM - wiki.thefactoryhka.com.pa - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thefactoryhka.com.pa could not be found [20:05:12] PROBLEM - bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bots.miraheze.wiki could not be found [20:05:14] PROBLEM - miraheze.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.ga could not be found [20:05:18] PROBLEM - piwik.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for piwik.miraheze.org could not be found [20:05:20] PROBLEM - linkwiki.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for linkwiki.cf could not be found [20:05:24] PROBLEM - vmcodex.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vmcodex.net could not be found [20:05:25] PROBLEM - wiki.raghuveer.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.raghuveer.net could not be found [20:05:59] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.87, 21.57, 15.40 [20:07:35] PROBLEM - wiki.s23.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.s23.org could not be found [20:08:02] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.50, 20.95, 15.95 [20:08:35] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.58, 5.60, 3.57 [20:10:02] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.94, 18.37, 15.58 [20:10:35] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.73, 4.31, 3.35 [20:11:50] RECOVERY - wiki.phgalaxy.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.phgalaxy.eu.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:11:51] RECOVERY - miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:11:52] RECOVERY - famedata.org - reverse DNS on sslhost is OK: SSL OK - famedata.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:11:54] RECOVERY - wiki.consid.vn - reverse DNS on sslhost is OK: SSL OK - wiki.consid.vn reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:12:00] RECOVERY - bots.miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:12:01] RECOVERY - iesd.mobagenie.my.id - reverse DNS on sslhost is OK: SSL OK - iesd.mobagenie.my.id reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:12:02] RECOVERY - wiki.thefactoryhka.com.pa - reverse DNS on sslhost is OK: SSL OK - wiki.thefactoryhka.com.pa reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:12:02] RECOVERY - miraheze.ga - reverse DNS on sslhost is OK: SSL OK - miraheze.ga reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:12:05] RECOVERY - www.erikapedia.com - reverse DNS on sslhost is OK: SSL OK - www.erikapedia.com reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:12:13] RECOVERY - piwik.miraheze.org - reverse DNS on sslhost is OK: SSL OK - piwik.miraheze.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:12:16] RECOVERY - linkwiki.cf - reverse DNS on sslhost is OK: SSL OK - linkwiki.cf reverse DNS resolves to cp13.miraheze.org - CNAME FLAT [20:12:22] RECOVERY - vmcodex.net - reverse DNS on sslhost is OK: SSL OK - vmcodex.net reverse DNS resolves to cp13.miraheze.org - CNAME FLAT [20:12:25] RECOVERY - wiki.raghuveer.net - reverse DNS on sslhost is OK: SSL OK - wiki.raghuveer.net reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:14:18] RECOVERY - wiki.s23.org - reverse DNS on sslhost is OK: SSL OK - wiki.s23.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:15:23] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: testing --Void, last run 12 minutes ago with 0 failures [20:48:28] !log manually cycling graylog index, deleted two oldest indexes [20:48:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:51:15] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.91, 1.93, 1.38 [20:53:15] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.01, 1.62, 1.34 [21:00:40] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:11:31] PROBLEM - mw11 Disk Space on mw11 is CRITICAL: DISK CRITICAL - free space: / 1130 MB (5% inode=75%); [21:13:31] PROBLEM - mw11 Disk Space on mw11 is WARNING: DISK WARNING - free space: / 1160 MB (6% inode=75%); [21:39:50] Voidwalker: should loss of logging not be UBN!? [22:04:03] RECOVERY - graylog2 Disk Space on graylog2 is OK: DISK OK - free space: / 91628 MB (12% inode=99%); [22:54:45] !log deleted additional graylog indexes to regain disk space [22:54:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:55:25] !log manually removed alias from index to restore functionality to graylog [22:55:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:42] graylog back up, gonna leave task open regarding my other points [23:06:54] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.25, 7.12, 5.80 [23:08:53] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.64, 7.06, 5.93 [23:10:51] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.00, 6.56, 5.88