[00:03:00] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.18, 20.01, 18.06 [00:03:18] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown) [00:04:24] RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 8200 MB (21% inode=96%); [00:04:54] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 14.41, 10.04, 7.08 [00:06:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.45, 9.39, 6.94 [00:06:55] PROBLEM - www.erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.erikapedia.com could not be found [00:06:56] PROBLEM - gp.ct777.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for gp.ct777.cf could not be found [00:06:56] PROBLEM - erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for erikapedia.com could not be found [00:09:00] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.30, 24.53, 20.53 [00:10:05] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.79, 7.84, 6.86 [00:10:49] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.45, 7.99, 7.19 [00:11:00] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.47, 21.43, 19.87 [00:13:47] RECOVERY - www.erikapedia.com - reverse DNS on sslhost is OK: SSL OK - www.erikapedia.com reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:13:47] RECOVERY - gp.ct777.cf - reverse DNS on sslhost is OK: SSL OK - gp.ct777.cf reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:48] RECOVERY - erikapedia.com - reverse DNS on sslhost is OK: SSL OK - erikapedia.com reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:13:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.21, 6.51, 6.53 [00:14:50] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.58, 6.33, 6.72 [00:15:02] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.82, 18.70, 19.34 [00:17:50] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-2 [+0/-0/±1] 13https://git.io/JWAwt [00:17:52] [02miraheze/puppet] 07The-Voidwalker 0356d8493 - postfix login require postfixUser objectclass [00:17:53] [02puppet] 07The-Voidwalker created branch 03The-Voidwalker-patch-2 - 13https://git.io/vbiAS [00:24:18] [02puppet] 07The-Voidwalker opened pull request 03#1816: postfix login require postfixUser objectclass - 13https://git.io/JWArU [00:27:21] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-2 [+0/-0/±1] 13https://git.io/JWArz [00:27:23] [02miraheze/puppet] 07The-Voidwalker 036b71527 - enforce login_mismatch on authenticated users [00:27:24] [02puppet] 07The-Voidwalker synchronize pull request 03#1816: postfix login require postfixUser objectclass - 13https://git.io/JWArU [00:31:15] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [00:34:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.66, 7.91, 6.57 [00:35:34] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.05, 7.69, 6.57 [00:36:19] PROBLEM - wiki.openhatch.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.openhatch.org could not be found [00:36:20] PROBLEM - hi.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hi.gyaanipedia.com could not be found [00:36:26] PROBLEM - lg.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lg.gyaanipedia.com could not be found [00:36:26] PROBLEM - vitriol.spoon.army - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vitriol.spoon.army could not be found [00:36:29] PROBLEM - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.bikinibottomdiscord.org could not be found [00:36:30] PROBLEM - ao90.pinho.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ao90.pinho.org could not be found [00:36:35] PROBLEM - wiki.arnab.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.arnab.at could not be found [00:36:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.38, 7.79, 6.70 [00:36:36] PROBLEM - wiki.ssij.se - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ssij.se could not be found [00:36:37] PROBLEM - permanentfuturelab.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for permanentfuturelab.wiki could not be found [00:36:43] PROBLEM - wiki.climatechange.ai - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.climatechange.ai could not be found [00:36:44] PROBLEM - rct.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rct.wiki could not be found [00:36:45] PROBLEM - wiki.phgalaxy.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.phgalaxy.eu.org could not be found [00:36:45] PROBLEM - www.rothwell-leeds.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.rothwell-leeds.co.uk could not be found [00:36:46] PROBLEM - www.zenbuddhism.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.zenbuddhism.info could not be found [00:37:30] PROBLEM - mail2 Puppet on mail2 is WARNING: WARNING: Puppet is currently disabled, message: Testing patch --Void, last run 9 minutes ago with 0 failures [00:37:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.24, 6.98, 6.44 [00:38:34] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JWAKa [00:38:36] [02miraheze/RemovePII] 07Universal-Omega 03fd3eaf9 - Update SpecialRemovePII.php [00:38:37] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [00:39:26] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega - testing RemovePII, last run 5 minutes ago with 0 failures [00:39:33] miraheze/RemovePII - Universal-Omega the build passed. [00:41:33] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.62, 6.44, 6.34 [00:42:33] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.07, 6.80, 6.61 [00:43:10] RECOVERY - wiki.openhatch.org - reverse DNS on sslhost is OK: SSL OK - wiki.openhatch.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:43:10] RECOVERY - hi.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - hi.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:43:12] RECOVERY - ao90.pinho.org - reverse DNS on sslhost is OK: SSL OK - ao90.pinho.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:43:15] RECOVERY - wiki.arnab.at - reverse DNS on sslhost is OK: SSL OK - wiki.arnab.at reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:43:20] RECOVERY - lg.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - lg.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:43:21] RECOVERY - vitriol.spoon.army - reverse DNS on sslhost is OK: SSL OK - vitriol.spoon.army reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:43:22] RECOVERY - wiki.ssij.se - reverse DNS on sslhost is OK: SSL OK - wiki.ssij.se reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:43:24] RECOVERY - permanentfuturelab.wiki - reverse DNS on sslhost is OK: SSL OK - permanentfuturelab.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:43:26] RECOVERY - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is OK: SSL OK - wiki.bikinibottomdiscord.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:43:31] RECOVERY - wiki.climatechange.ai - reverse DNS on sslhost is OK: SSL OK - wiki.climatechange.ai reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:43:33] RECOVERY - rct.wiki - reverse DNS on sslhost is OK: SSL OK - rct.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:43:33] RECOVERY - www.zenbuddhism.info - reverse DNS on sslhost is OK: SSL OK - www.zenbuddhism.info reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:43:38] RECOVERY - www.rothwell-leeds.co.uk - reverse DNS on sslhost is OK: SSL OK - www.rothwell-leeds.co.uk reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:43:46] RECOVERY - wiki.phgalaxy.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.phgalaxy.eu.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [01:04:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.32, 6.33, 5.70 [01:05:12] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.26, 7.51, 6.44 [01:06:26] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.70, 5.93, 5.62 [01:07:11] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.60, 6.70, 6.26 [01:09:28] RECOVERY - mail2 Puppet on mail2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:17:40] !log mail2 banned IP in firewall [01:17:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:13:29] PROBLEM - mail2 Puppet on mail2 is WARNING: WARNING: Puppet is currently disabled, message: Testing patch --Void, last run 16 minutes ago with 0 failures [02:35:57] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JWxUp [02:35:58] [02miraheze/RemovePII] 07Universal-Omega 0326567ea - Update SpecialRemovePII.php [02:36:00] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [02:36:54] miraheze/RemovePII - Universal-Omega the build passed. [02:42:38] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JWxki [02:42:39] [02miraheze/RemovePII] 07Universal-Omega 03f8fbddd - Update SpecialRemovePII.php [02:42:41] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [02:43:29] miraheze/RemovePII - Universal-Omega the build passed. [02:44:30] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JWxkj [02:44:32] [02miraheze/RemovePII] 07Universal-Omega 03261cbc9 - Update SpecialRemovePII.php [02:44:33] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [02:45:33] miraheze/RemovePII - Universal-Omega the build passed. [02:46:03] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JWxIB [02:46:04] [02miraheze/RemovePII] 07Universal-Omega 03a731907 - Update en.json [02:46:06] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [02:46:40] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JWxIg [02:46:41] [02miraheze/RemovePII] 07Universal-Omega 03f2443fb - Update qqq.json [02:46:43] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [02:47:00] miraheze/RemovePII - Universal-Omega the build passed. [02:47:33] miraheze/RemovePII - Universal-Omega the build passed. [02:51:20] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JWxLH [02:51:22] [02miraheze/RemovePII] 07Universal-Omega 0358fc17f - Update SpecialRemovePII.php [02:51:23] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [02:52:12] miraheze/RemovePII - Universal-Omega the build passed. [02:53:28] PROBLEM - mail2 Puppet on mail2 is WARNING: WARNING: Puppet is currently disabled, message: Testing patch --Void, last run 56 minutes ago with 0 failures [02:55:55] [02puppet] 07The-Voidwalker closed pull request 03#1816: postfix login require postfixUser objectclass - 13https://git.io/JWArU [02:55:56] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JWxty [02:55:58] [02miraheze/puppet] 07The-Voidwalker 032de40e3 - postfix login require postfixUser objectclass (#1816) [02:58:22] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JWxqG [02:58:24] [02miraheze/RemovePII] 07Universal-Omega 035543979 - Update SpecialRemovePII.php [02:58:25] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [02:59:14] miraheze/RemovePII - Universal-Omega the build passed. [02:59:28] RECOVERY - mail2 Puppet on mail2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:07:35] [02RemovePII] 07Universal-Omega commented on pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/JWxYB [03:08:09] [02RemovePII] 07Universal-Omega edited a comment on pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/JWxYB [03:08:19] [02RemovePII] 07Universal-Omega edited a comment on pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/JWxYB [04:17:28] PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 126.46 ms [04:19:27] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 87.29 ms [05:43:03] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.47, 1.78, 1.22 [05:45:03] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.28, 1.54, 1.19 [07:35:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.40, 6.85, 5.90 [07:37:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.89, 6.06, 5.72 [10:27:03] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CRITICAL: Full, 5850 files, 29.94MB, 2021-06-27 00:05:00 (3.3 weeks ago) [10:27:03] PROBLEM - bacula2 Disk Space on bacula2 is CRITICAL: DISK CRITICAL - free space: / 8 MB (0% inode=99%); [10:27:03] PROBLEM - bacula2 Bacula Static on bacula2 is CRITICAL: CRITICAL: Full, 10219941 files, 1.220TB, 2021-05-07 04:23:00 (10.6 weeks ago) [10:27:11] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is WARNING: WARNING: Diff, 111159 files, 28.62GB, 2021-06-20 01:15:00 (4.3 weeks ago) [10:27:43] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is WARNING: WARNING: Diff, 111159 files, 28.62GB, 2021-06-20 01:15:00 (4.3 weeks ago) [10:27:43] PROBLEM - bacula2 Disk Space on bacula2 is CRITICAL: DISK CRITICAL - free space: / 8 MB (0% inode=99%); [10:27:43] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CRITICAL: Full, 5850 files, 29.94MB, 2021-06-27 00:05:00 (3.3 weeks ago) [10:27:43] PROBLEM - bacula2 Bacula Static on bacula2 is CRITICAL: CRITICAL: Full, 10219941 files, 1.220TB, 2021-05-07 04:23:00 (10.6 weeks ago) [10:27:46] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is WARNING: WARNING: Diff, 104810 files, 52.01GB, 2021-06-20 02:42:00 (4.3 weeks ago) [10:28:01] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 6697 files, 511.9MB, 2021-06-23 06:21:00 (3.9 weeks ago) [10:31:13] That's expired downtime [10:31:36] D: [10:34:51] Ugochimobi: don't worry about the alerts. They are not new. [10:38:51] Oh, I see [10:39:33] I was just surprised, lol. [10:51:18] I downtimed stuff yesterday when the server crashed [10:51:25] Not sure why it went off again [10:51:29] Because they're old alerts [10:53:26] Okay [10:53:38] You mean the server crashed? [11:25:45] Ugochimobi: yes [11:26:30] RhinosF1: Wow, But it's fixed now I'm sure.. [11:26:42] It is [11:26:48] Not sure why it went funny [11:26:53] It's a provider issue though [11:27:01] And we can cope without bacula for a few hours fine [11:28:01] phew, What's bacula actually? [11:33:21] Backups [11:34:02] Oh, I see in the name [12:06:16] @void ping me when you're around [12:12:51] Void: [12:14:30] Ugochimobi: why do you need to ping him too? [12:15:21] Lol your ping didn't went and same with mine, I am wondering why [12:15:33] RIght now Void hasn't been pinged at all [12:15:43] He's only on discord [12:15:49] So it'll ping on that side surely [12:16:08] I know i'll not ping [12:16:17] *It'll [12:16:47] I just check and it didn't [12:16:59] Oh [12:17:34] @Void [12:18:13] RhinosF1: Check the off-topic [12:18:15] Lol [12:19:16] His username is Orduin on Discord [12:19:19] not Void [12:19:32] Ah [12:20:00] :] [12:23:25] PROBLEM - puppet3 APT on puppet3 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (7 critical updates). [12:23:29] PROBLEM - cp12 APT on cp12 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (7 critical updates). [12:25:12] PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 57 packages available for upgrade (7 critical updates). [12:26:47] PROBLEM - gluster4 APT on gluster4 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (7 critical updates). [12:27:23] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (6 critical updates). [12:28:09] PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 75 packages available for upgrade (7 critical updates). [12:31:07] PROBLEM - cp13 APT on cp13 is CRITICAL: APT CRITICAL: 23 packages available for upgrade (7 critical updates). [12:31:49] PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:33:01] PROBLEM - graylog2 APT on graylog2 is CRITICAL: APT CRITICAL: 11 packages available for upgrade (8 critical updates). [12:34:20] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:34:49] PROBLEM - phab2 APT on phab2 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:35:09] PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:35:25] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:35:26] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:35:48] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:36:00] PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:36:18] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:36:41] PROBLEM - mem1 APT on mem1 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (7 critical updates). [12:37:32] Fun [12:38:17] Lol [12:38:23] PROBLEM - ldap2 APT on ldap2 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (7 critical updates). [12:38:53] 37 packages available for upgrade (7 critical updates) [12:38:55] Wow [12:39:49] Ugochimobi: people release on mass [12:39:58] It was probably 0 yesterday [12:40:03] I think it checks daily [12:40:23] you mean the critical ones [12:40:33] Ye [12:41:38] Oh [12:41:42] and now 7 [12:41:52] Yes [12:42:13] They'll all be linked [12:42:18] Packaging is strange [12:42:30] PROBLEM - services4 APT on services4 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (8 critical updates). [12:42:35] 8 now [12:42:47] Depends on the server [12:42:55] They run different stuff [12:43:18] Yeah [12:43:21] that's true [12:43:28] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (7 critical updates). [12:43:39] It's now back to 7 [12:43:47] X) [12:43:49] PROBLEM - services3 APT on services3 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (8 critical updates). [12:45:16] PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 75 packages available for upgrade (7 critical updates). [12:49:52] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (7 critical updates). [12:50:08] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 111 packages available for upgrade (7 critical updates). [12:50:22] PROBLEM - gluster3 APT on gluster3 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (7 critical updates). [12:50:34] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 20 packages available for upgrade (7 critical updates). [12:50:47] PROBLEM - test4 APT on test4 is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [12:51:36] PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 78 packages available for upgrade (7 critical updates). [12:51:41] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 78 packages available for upgrade (7 critical updates). [12:51:52] PROBLEM - mon2 APT on mon2 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:52:56] PROBLEM - mem2 APT on mem2 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (7 critical updates). [13:32:15] PROBLEM - mw10 APT on mw10 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [13:34:17] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [13:35:54] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.21, 6.55, 5.62 [13:37:55] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.25, 5.88, 5.48 [13:53:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.78, 7.64, 6.33 [13:55:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.48, 6.76, 6.16 [14:35:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.36, 6.58, 5.73 [14:35:16] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.17, 6.96, 5.77 [14:37:10] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.45, 6.12, 5.67 [14:37:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.37, 6.53, 5.74 [15:04:54] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.19, 7.43, 6.24 [15:05:37] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JWjYz [15:05:39] [02miraheze/ManageWiki] 07Universal-Omega 0304db20a - Update en.json [15:05:40] [02ManageWiki] 07Universal-Omega synchronize pull request 03#281: Redesign form submission - 13https://git.io/JW7MN [15:06:28] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JWjY7 [15:06:29] [02miraheze/ManageWiki] 07Universal-Omega 033fa1719 - Update en.json [15:06:31] [02ManageWiki] 07Universal-Omega synchronize pull request 03#281: Redesign form submission - 13https://git.io/JW7MN [15:06:40] miraheze/ManageWiki - Universal-Omega the build passed. [15:06:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.17, 6.63, 6.11 [15:07:29] miraheze/ManageWiki - Universal-Omega the build passed. [17:05:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.02, 6.60, 5.86 [17:07:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.10, 6.00, 5.73 [18:04:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.79, 7.46, 6.06 [18:05:12] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.56, 7.32, 5.88 [18:05:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.83, 7.49, 6.12 [18:07:16] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.12, 7.21, 6.02 [18:09:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.31, 18.74, 15.44 [18:09:33] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.36, 6.05, 5.88 [18:10:25] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.02, 4.78, 3.06 [18:10:59] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.02, 6.62, 6.14 [18:11:01] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.62, 19.50, 16.19 [18:12:25] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.73, 3.48, 2.79 [18:13:11] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.90, 6.60, 6.14 [18:14:07] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.39, 7.25, 6.18 [18:14:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.81, 7.04, 6.42 [18:15:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.41, 6.84, 6.33 [18:16:01] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.30, 6.85, 6.17 [18:17:00] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.28, 6.61, 6.33 [18:17:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.70, 6.56, 6.30 [18:17:56] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.97, 6.71, 6.20 [18:22:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.70, 7.25, 6.62 [18:24:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.67, 6.65, 6.47 [18:27:38] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.19, 6.99, 6.51 [18:29:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.14, 7.44, 6.73 [18:35:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.18, 7.51, 6.99 [18:35:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.01, 6.96, 6.62 [18:37:37] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.13, 6.69, 6.75 [18:37:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.88, 6.62, 6.54 [19:28:10] RECOVERY - mail2 APT on mail2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:28:23] RECOVERY - mw8 APT on mw8 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:30:09] RECOVERY - mw9 APT on mw9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:30:16] RECOVERY - mw10 APT on mw10 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:31:08] RECOVERY - jobrunner3 APT on jobrunner3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:31:26] RECOVERY - mw11 APT on mw11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:31:42] RECOVERY - jobrunner4 APT on jobrunner4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:32:27] !log apt upgrade on mail2, mw*, jobrunner*, and graylog2 [19:32:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:33:10] I'll get the rest when I'm back in an hour or two [19:34:21] RECOVERY - graylog2 APT on graylog2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:00:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.79, 10.42, 7.34 [20:00:47] PROBLEM - www.programming.red - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.programming.red could not be found [20:00:49] PROBLEM - heavyironmodding.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for heavyironmodding.org could not be found [20:00:54] PROBLEM - wiki.mastodon.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mastodon.kr could not be found [20:00:55] PROBLEM - repository.archiopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for repository.archiopedia.org could not be found [20:00:56] PROBLEM - tensegritywiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tensegritywiki.com could not be found [20:00:58] PROBLEM - madzebrascience.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for madzebrascience.wiki could not be found [20:01:54] PROBLEM - vortekspedia.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vortekspedia.tk could not be found [20:01:58] PROBLEM - wiki.digitaldesignhq.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.digitaldesignhq.com could not be found [20:02:00] PROBLEM - wiki.ct777.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ct777.cf could not be found [20:02:01] PROBLEM - ipv6bolivia.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ipv6bolivia.tk could not be found [20:02:01] PROBLEM - zw.fontainebleau-avon.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zw.fontainebleau-avon.fr could not be found [20:02:03] PROBLEM - miraheze.ml - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.ml could not be found [20:02:03] PROBLEM - phabdigests.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for phabdigests.bots.miraheze.wiki could not be found [20:02:03] PROBLEM - miraheze.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.tk could not be found [20:02:04] PROBLEM - wiki.danvs.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.danvs.net could not be found [20:02:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.10, 7.29, 6.55 [20:02:05] PROBLEM - puardian.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for puardian.xyz could not be found [20:02:05] PROBLEM - n64brew.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for n64brew.dev could not be found [20:02:06] PROBLEM - nonciclopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nonciclopedia.org could not be found [20:02:07] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zymonic.com could not be found [20:02:07] PROBLEM - intp.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for intp.miraheze.org could not be found [20:02:08] PROBLEM - threedomwiki.pcast.site - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for threedomwiki.pcast.site could not be found [20:02:11] PROBLEM - www.puzzles.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.puzzles.wiki could not be found [20:02:11] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.37, 7.11, 6.11 [20:02:12] PROBLEM - wiki.velaan.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.velaan.org could not be found [20:02:12] PROBLEM - pt.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pt.graalmilitary.com could not be found [20:02:14] PROBLEM - wiki.jumpbound.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jumpbound.com could not be found [20:02:14] PROBLEM - wiki.worlduniversityandschool.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worlduniversityandschool.org could not be found [20:02:15] PROBLEM - idleon.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for idleon.info could not be found [20:02:16] PROBLEM - you.r-fit.cc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for you.r-fit.cc could not be found [20:02:17] PROBLEM - wiki.astralprojections.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.astralprojections.org could not be found [20:04:02] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.14, 8.13, 6.92 [20:04:11] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.46, 6.57, 6.03 [20:04:53] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.82, 6.88, 6.18 [20:06:00] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.91, 7.37, 6.79 [20:06:26] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.43, 7.54, 7.00 [20:06:48] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.41, 6.15, 6.00 [20:07:27] RECOVERY - www.programming.red - reverse DNS on sslhost is OK: SSL OK - www.programming.red reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:07:28] RECOVERY - heavyironmodding.org - reverse DNS on sslhost is OK: SSL OK - heavyironmodding.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:07:35] RECOVERY - wiki.mastodon.kr - reverse DNS on sslhost is OK: SSL OK - wiki.mastodon.kr reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:07:38] RECOVERY - tensegritywiki.com - reverse DNS on sslhost is OK: SSL OK - tensegritywiki.com reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:07:54] RECOVERY - repository.archiopedia.org - reverse DNS on sslhost is OK: SSL OK - repository.archiopedia.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:08:00] RECOVERY - madzebrascience.wiki - reverse DNS on sslhost is OK: SSL OK - madzebrascience.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:08:01] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.72, 6.41, 6.51 [20:08:34] RECOVERY - vortekspedia.tk - reverse DNS on sslhost is OK: SSL OK - vortekspedia.tk reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:08:42] RECOVERY - wiki.digitaldesignhq.com - reverse DNS on sslhost is OK: SSL OK - wiki.digitaldesignhq.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:08:46] RECOVERY - n64brew.dev - reverse DNS on sslhost is OK: SSL OK - n64brew.dev reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:08:47] RECOVERY - ipv6bolivia.tk - reverse DNS on sslhost is OK: SSL OK - ipv6bolivia.tk reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:08:48] RECOVERY - zw.fontainebleau-avon.fr - reverse DNS on sslhost is OK: SSL OK - zw.fontainebleau-avon.fr reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:08:49] RECOVERY - wiki.ct777.cf - reverse DNS on sslhost is OK: SSL OK - wiki.ct777.cf reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:08:53] RECOVERY - miraheze.ml - reverse DNS on sslhost is OK: SSL OK - miraheze.ml reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:08:54] RECOVERY - intp.miraheze.org - reverse DNS on sslhost is OK: SSL OK - intp.miraheze.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:08:56] RECOVERY - phabdigests.bots.miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - phabdigests.bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:08:58] RECOVERY - miraheze.tk - reverse DNS on sslhost is OK: SSL OK - miraheze.tk reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:09:00] RECOVERY - www.puzzles.wiki - reverse DNS on sslhost is OK: SSL OK - www.puzzles.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:09:03] RECOVERY - nonciclopedia.org - reverse DNS on sslhost is OK: SSL OK - nonciclopedia.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:09:03] RECOVERY - pt.graalmilitary.com - reverse DNS on sslhost is OK: SSL OK - pt.graalmilitary.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:09:04] RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: SSL OK - wiki.zymonic.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:09:05] RECOVERY - wiki.danvs.net - reverse DNS on sslhost is OK: SSL OK - wiki.danvs.net reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:09:08] RECOVERY - wiki.jumpbound.com - reverse DNS on sslhost is OK: SSL OK - wiki.jumpbound.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:09:09] RECOVERY - puardian.xyz - reverse DNS on sslhost is OK: SSL OK - puardian.xyz reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:09:10] RECOVERY - wiki.worlduniversityandschool.org - reverse DNS on sslhost is OK: SSL OK - wiki.worlduniversityandschool.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:09:10] RECOVERY - idleon.info - reverse DNS on sslhost is OK: SSL OK - idleon.info reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:09:11] RECOVERY - threedomwiki.pcast.site - reverse DNS on sslhost is OK: SSL OK - threedomwiki.pcast.site reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:09:12] RECOVERY - you.r-fit.cc - reverse DNS on sslhost is OK: SSL OK - you.r-fit.cc reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:09:12] RECOVERY - wiki.astralprojections.org - reverse DNS on sslhost is OK: SSL OK - wiki.astralprojections.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:09:13] RECOVERY - wiki.velaan.org - reverse DNS on sslhost is OK: SSL OK - wiki.velaan.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:10:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.90, 6.22, 6.59 [21:04:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.19, 6.65, 5.61 [21:04:48] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.45, 7.66, 6.13 [21:05:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.81, 7.91, 6.01 [21:06:47] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.29, 7.18, 6.14 [21:08:08] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.41, 6.70, 5.92 [21:08:47] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.63, 6.57, 6.04 [21:09:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.27, 7.34, 6.26 [21:11:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.37, 6.65, 6.14 [21:35:04] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.86, 6.62, 6.29 [21:35:25] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.01, 8.06, 6.81 [21:35:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.55, 7.14, 6.31 [21:37:05] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.28, 5.75, 6.01 [21:37:32] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.26, 6.69, 6.23 [21:41:21] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.25, 7.77, 7.08 [21:41:33] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.77, 7.95, 6.85 [21:41:59] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.qadrishattari.xyz could not be found [21:43:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.95, 7.72, 6.91 [21:45:17] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.16, 6.64, 6.79 [21:45:33] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.58, 6.52, 6.56 [21:49:54] RECOVERY - mon2 APT on mon2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:50:46] RECOVERY - test4 APT on test4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:53:09] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.51, 7.23, 6.87 [21:53:11] RECOVERY - phab2 APT on phab2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:55:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.75, 7.20, 6.92 [21:55:27] RECOVERY - puppet3 APT on puppet3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:57:12] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:58:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.50, 7.26, 6.69 [21:59:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.35, 8.16, 7.34 [22:00:42] RECOVERY - mem1 APT on mem1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:01:50] !log apt upgrade mon2, test4, phab2, puppet3, mem* [22:01:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:02:05] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.00, 7.46, 6.94 [22:02:56] RECOVERY - mem2 APT on mem2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:03:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.79, 7.58, 7.34 [22:04:00] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.07, 8.77, 7.45 [22:05:04] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.68, 9.04, 7.95 [22:06:06] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.04, 6.96, 6.49 [22:07:10] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:07:21] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:07:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.45, 7.91, 7.43 [22:07:52] RECOVERY - ns2 APT on ns2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:08:01] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.96, 6.27, 6.29 [22:09:00] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.66, 7.43, 7.53 [22:11:22] RECOVERY - cp3 APT on cp3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:11:25] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:15:30] RECOVERY - cp12 APT on cp12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:15:32] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.22, 5.57, 6.54 [22:17:16] PROBLEM - cp12 Disk Space on cp12 is WARNING: DISK WARNING - free space: / 4184 MB (10% inode=96%); [22:18:50] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.32, 5.89, 6.68 [22:19:07] RECOVERY - cp13 APT on cp13 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:22:21] RECOVERY - gluster3 APT on gluster3 is OK: APT OK: 18 packages available for upgrade (0 critical updates). [22:24:48] RECOVERY - gluster4 APT on gluster4 is OK: APT OK: 18 packages available for upgrade (0 critical updates). [22:26:08] !log apt upgrade ns*, cp*, test3, gluster* [22:26:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:26:14] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 41 failures. Last run 3 minutes ago with 41 failures. Failed resources (up to 3 shown) [22:26:32] looking ^ probably just temporary puppet goof [22:27:52] seems so [22:28:14] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:28:23] RECOVERY - ldap2 APT on ldap2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:31:45] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.60, 6.79, 6.74 [22:31:51] RECOVERY - services3 APT on services3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:32:28] RECOVERY - services4 APT on services4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:33:43] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.89, 8.07, 7.19 [22:35:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.05, 7.90, 7.26 [22:36:00] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.39, 7.09, 6.68 [22:36:51] !log apt upgrade ldap2, services* [22:36:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:37:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.71, 6.28, 6.44 [22:41:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.56, 6.22, 6.70 [23:05:06] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.51, 8.50, 6.93 [23:05:11] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.47, 6.85, 6.33 [23:06:10] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.35, 7.33, 6.53 [23:07:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.84, 7.35, 6.70 [23:07:12] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.36, 6.65, 6.33 [23:07:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.19, 7.02, 6.35 [23:08:08] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.28, 7.06, 6.54 [23:09:37] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.79, 6.58, 6.27 [23:10:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.04, 6.36, 6.35 [23:10:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.43, 6.71, 6.58 [23:31:50] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JlePp [23:31:51] [02miraheze/CreateWiki] 07Universal-Omega 0336ff917 - Don't inject RequestWikiAIJob if model file is not set [23:31:53] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vpJTL [23:31:54] [02CreateWiki] 07Universal-Omega opened pull request 03#229: Don't inject RequestWikiAIJob if model file is not set - 13https://git.io/JlePj [23:32:52] miraheze/CreateWiki - Universal-Omega the build passed. [23:33:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JleX3 [23:33:17] [02miraheze/mw-config] 07Universal-Omega 030e1c169 - Set wgCreateWikiPersistentModelFile to false [23:33:18] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [23:33:20] [02mw-config] 07Universal-Omega opened pull request 03#4008: Set wgCreateWikiPersistentModelFile to false - 13https://git.io/JleXs [23:33:46] Voidwalker: ^ [23:34:27] miraheze/mw-config - Universal-Omega the build passed. [23:36:02] I did the CreateWiki PR to prevent the job from even be unnecessarily loaded. There is already a check within the job itself for `file_exists` though. [23:36:41] hmm, I would imagine just setting CreateWikiAIThreshold to false would disable it, but we don't seem to have that configured anywhere [23:37:11] Voidwalker: I thought about that but it seems like it is false by default so likely not. [23:38:40] yeah, looks like it is false, which means it shouldn't be running [23:39:07] Voidwalker: that only is used to make it actually approve the requests automatically, right now it just gives a score. [23:39:15] oh, right [23:40:24] yeah, then looks good [23:40:28] * Voidwalker confused [23:41:50] oh wait [23:42:40] Voidwalker: I'm unable to do it right now, I had to go mobile for a bit. So you can deploy CW PR, update, then mw-config if you want, or I can later. [23:43:07] yeah, I'll take a look, might be a simpler way to do things [23:43:10] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.35, 6.39, 5.51 [23:44:30] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.71, 6.40, 5.59 [23:45:09] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.97, 6.25, 5.57 [23:47:02] nah, probably not. if it works, it works [23:48:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.11, 6.48, 5.82 [23:50:48] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.66, 6.89, 6.12 [23:52:42] oh, I have to disappear for a bit too, so whoever gets back deploys :) [23:53:37] Sounds good. [23:54:46] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.89, 6.67, 6.22 [23:58:41] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.90, 7.90, 6.78