[00:03:37] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:44] RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 8164 MB (21% inode=96%); [00:03:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.63, 21.56, 17.49 [00:04:50] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 17.17, 9.99, 7.44 [00:05:55] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.51, 8.99, 7.14 [00:05:55] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.53, 22.75, 18.39 [00:07:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.90, 22.79, 18.93 [00:08:43] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.55, 7.44, 7.02 [00:09:47] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.70, 3.54, 2.58 [00:09:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.63, 7.46, 6.98 [00:11:47] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 3.02, 3.33, 2.62 [00:14:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.01, 6.45, 6.73 [00:15:50] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.54, 6.25, 6.61 [00:21:55] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.93, 20.39, 19.92 [00:23:46] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.62, 7.12, 6.87 [00:25:45] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.72, 6.20, 6.56 [00:31:33] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:41:48] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.15, 1.89, 1.27 [00:43:48] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.28, 1.70, 1.28 [00:51:05] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.62, 18.52, 17.99 [00:53:00] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.24, 16.81, 17.45 [00:58:12] I'm going to recommend against creating wikis for the moment, redis is very much not stable, and I'm not certain of the fix [01:01:23] that's unfortunate [01:02:04] well, it looks fine at the moment, but I'm seeing redis got killed 85 times in the past 24 hours, which basically means sessions and jobqueue are offline while it restarts [01:02:09] this is not good [01:02:14] and I have no idea why [01:03:46] @Void That's fine, as I'll be in bed by then anyway. [01:04:14] I mean, if you want to now, at this moment, go ahead [01:05:23] Okay, but I'm just doing what you said there. [01:08:11] If anyone creates a wiki and doesn't see a corresponding farmer log entry for the wiki actually being created, please report it as something on that wiki undoubtedly didn't get created properly. [01:11:48] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.80, 1.50, 1.21 [01:12:09] Voidwalker: that's what I've been saying for months. I always thought the farmer log entry issue and redis errors were related, but wasn't 100% sure... so thanks for looking into it! It's been an issue for 3 or 4 months. [01:12:25] Definitely, but what to do about it? [01:13:48] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.21, 1.33, 1.18 [01:14:22] Voidwalker: honestly, I have no idea. I do remember when it started though, right after some work John did on JobRunner (I think). I do think it was somehow related but I have no idea what. [01:14:31] maybe we could stick redis on it's own dedicated server [01:14:58] I suspect the problem is we occasionally get long running jobs that cause memory leaks in php [01:16:45] I'd also suggest not running the foreachwikiindblist maintenance scripts on jobrunner3 [01:18:58] Voidwalker: that would make sense also. We could disable CreateWiki AI and see if that resolves the issue, but that might not be a great approach but may be necessary to fix this issue. (It may have actually started once AI was added I can't remember for sure now, but the AI job is quite a large job I think) I could be completely wrong about AI though. As for foreachwikiindblist that sounds like a great idea also. [01:19:28] yeah, I'm really not too sure about this one [01:19:53] but let me cross check some logs, if it's the CreateWiki AI, we should see the OOM correspond closely with wiki creations [01:21:42] Hmm, from the looks of things redis got killed repeatedly on startup for a few minutes before a php process was killed and that stopped it [01:27:20] identifying this process would go a long way to figuring out what's going on [01:33:38] Hopefully things will be back to normal by tomorrow with everybody present. [03:21:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.78, 3.23, 2.80 [03:23:03] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.28, 2.93, 2.75 [04:11:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 2.90, 2.51 [04:13:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.06, 2.60, 2.45 [04:45:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JWixj [04:45:13] [02miraheze/services] 07MirahezeSSLBot 03014b9f1 - BOT: Updating services config for wikis [05:18:47] PROBLEM - wiki.redeemer.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.redeemer.live could not be found [05:18:47] PROBLEM - vise.dayid.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vise.dayid.org could not be found [05:18:48] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zymonic.com could not be found [05:18:49] PROBLEM - en.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.gyaanipedia.com could not be found [05:18:50] PROBLEM - archive.a2b2.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for archive.a2b2.org could not be found [05:18:51] PROBLEM - wiki.whentheycry.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.whentheycry.org could not be found [05:18:54] PROBLEM - cnt.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cnt.miraheze.org could not be found [05:18:55] PROBLEM - tl.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tl.awiki.org could not be found [05:18:55] PROBLEM - wiki.creaturathegame.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.creaturathegame.com could not be found [05:25:33] RECOVERY - cnt.miraheze.org - reverse DNS on sslhost is OK: SSL OK - cnt.miraheze.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [05:25:35] RECOVERY - tl.awiki.org - reverse DNS on sslhost is OK: SSL OK - tl.awiki.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [05:25:35] RECOVERY - wiki.creaturathegame.com - reverse DNS on sslhost is OK: SSL OK - wiki.creaturathegame.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [05:25:39] RECOVERY - wiki.redeemer.live - reverse DNS on sslhost is OK: SSL OK - wiki.redeemer.live reverse DNS resolves to cp13.miraheze.org - CNAME OK [05:25:40] RECOVERY - vise.dayid.org - reverse DNS on sslhost is OK: SSL OK - vise.dayid.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [05:25:41] RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: SSL OK - wiki.zymonic.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [05:25:43] RECOVERY - en.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - en.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [05:25:46] RECOVERY - archive.a2b2.org - reverse DNS on sslhost is OK: SSL OK - archive.a2b2.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [05:25:48] RECOVERY - wiki.whentheycry.org - reverse DNS on sslhost is OK: SSL OK - wiki.whentheycry.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [05:36:16] [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+3/-0/±3] 13https://git.io/JWPq3 [05:36:18] [02miraheze/ssl] 07Reception123 03fc9ae1c - add wiki.chouverse.cludisciples.cyou cert [05:36:19] [02miraheze/ssl] 07Reception123 03686b779 - add wiki.misslessfighters.club cert [05:36:21] [02miraheze/ssl] 07Reception123 03a944534 - add wiki.homiedb.party cert [05:38:33] [02miraheze/dns] 07Reception123 deleted branch 03revert-208-patch-23 [05:38:35] [02dns] 07Reception123 deleted branch 03revert-208-patch-23 - 13https://git.io/vbQXl [06:15:01] !log reception@jobrunner3:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php createLocalAccount.php --wiki grimmsiblingswiki Limid [06:15:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:18:56] !log reception@jobrunner3:/srv/mediawiki/w/extensions/CreateWiki/maintenance$ sudo -u www-data php populateMainPage.php --wiki grimmsiblingswiki [06:19:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:19:07] !log reception@jobrunner3:/srv/mediawiki/w/extensions/CreateWiki/maintenance$ sudo -u www-data php populateMainPage.php --wiki mahjongwiki [06:19:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:22:35] Reception123: Are you online on Discord 😉 wanna DM you [06:24:14] Yes [06:24:49] Okay💯 [06:41:40] PROBLEM - wiki.climatechange.ai - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.climatechange.ai could not be found [06:48:29] RECOVERY - wiki.climatechange.ai - reverse DNS on sslhost is OK: SSL OK - wiki.climatechange.ai reverse DNS resolves to cp13.miraheze.org - CNAME OK [07:58:12] An invalid user was specified to permission testing to embed this PDF. I am a Sysop and should be able to embed PDFs... [07:58:18] !sre ^ [07:59:35] WTH does it use $wgUser https://github.com/miraheze/PDFEmbed/blob/master/PDFEmbed.hooks.php#L46 ? [07:59:36] [url] PDFEmbed/PDFEmbed.hooks.php at master · miraheze/PDFEmbed · GitHub | github.com [07:59:55] Not a reason to page [08:00:01] That's not a wake me up emergency [08:00:30] Pretty sure $wgUser is on its way out though [08:00:49] If that woke you up, you should enable DND. [08:01:04] And yes, wgUser has been soft-deprecated already. [08:01:22] I was wondering if Redis killed my session (reading the discussion above). [08:02:58] The whole point of !_sre is to say this is an emergency and you should wake up [08:04:19] Redis being down should be an emergency. [08:04:37] But it does not seem that serious indeed, so apologies. [08:19:38] Redis is already a UBN [08:20:03] * RhinosF1 really should write guidelines for paging [08:21:10] I doubt it's redis or you'd have been logged out though and redis on jbr3 isnt sessions [08:21:13] That's rdb [08:22:38] @void: don't confuse redis on rdb with redis on jobrunner3 [08:28:06] Oh, I see. [08:28:50] I managed to fix it by replacing $wgUser with UserIdentity anyway. [08:30:23] I am surprised nobody has complained that the extension does not even work! [08:30:45] It's using Parser::disableCache() which is no longer a thing. [08:37:04] @R4356th: https://meta.miraheze.org/wiki/IRC#Stalkwords [08:37:05] [url] IRC - Miraheze Meta | meta.miraheze.org [10:43:23] Thanks for fixing the issue last night, @Void. [10:44:17] UBN? [10:48:09] Unbreak now [10:48:59] Oh. Sorry if I don't know what these abbreviations are supposed to mean. [12:10:44] Reception123: Online? please check Discord DM. [14:35:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.90, 6.52, 5.69 [14:37:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.65, 6.07, 5.62 [15:05:57] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.08, 8.48, 6.61 [15:07:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.94, 7.23, 6.38 [15:09:50] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.47, 6.31, 6.14 [15:26:20] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.98, 6.35, 6.04 [15:28:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.89, 6.31, 6.07 [15:34:40] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.04, 7.23, 6.43 [15:35:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.52, 6.79, 5.97 [15:35:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.12, 6.59, 5.98 [15:35:57] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.93, 6.85, 6.35 [15:37:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.37, 6.67, 6.34 [15:38:38] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.81, 6.48, 6.35 [15:39:09] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.22, 6.26, 5.97 [15:41:39] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.89, 6.59, 6.22 [15:42:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.13, 8.02, 6.91 [15:43:45] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.98, 7.08, 6.59 [15:44:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.53, 7.40, 6.83 [15:45:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.24, 6.42, 6.41 [15:48:34] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.24, 6.68, 6.65 [16:05:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.11, 6.16, 5.73 [16:07:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.85, 5.49, 5.53 [16:21:15] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.81, 6.93, 6.29 [16:23:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.72, 6.56, 6.24 [16:27:39] PROBLEM - mw8 Disk Space on mw8 is WARNING: DISK WARNING - free space: / 2052 MB (10% inode=74%); [16:35:10] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.84, 7.28, 6.55 [16:35:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.71, 6.87, 5.99 [16:37:10] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.90, 6.52, 6.38 [16:37:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.17, 5.98, 5.76 [16:47:11] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.63, 6.91, 6.10 [16:49:10] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.06, 5.88, 5.82 [17:19:32] [02mw-config] 07Universal-Omega assigned pull request 03#3990: Update extension names to ExtensionRegistry names - 13https://git.io/JctAo [17:20:13] [02mw-config] 07Universal-Omega edited pull request 03#3990: Update extension names to ExtensionRegistry names - 13https://git.io/JctAo [17:20:25] [02ManageWiki] 07Universal-Omega assigned pull request 03#278: ManageWikiExtensions: support description and name messages - 13https://git.io/Jna7W [17:25:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.15, 7.33, 6.21 [17:26:14] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.93, 7.44, 6.12 [17:27:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.78, 7.26, 6.31 [17:28:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.72, 7.07, 6.14 [17:29:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.61, 7.41, 6.34 [17:31:32] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.66, 6.69, 6.21 [17:31:38] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.75, 6.57, 6.24 [17:32:16] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.79, 7.33, 6.43 [17:34:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.35, 6.91, 6.39 [17:35:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.76, 7.47, 6.55 [17:35:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.89, 7.60, 6.68 [17:36:15] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.09, 6.38, 6.26 [17:39:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.26, 6.60, 6.46 [17:42:38] PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 131.42 ms [17:43:29] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.54, 6.29, 6.40 [17:43:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.08, 7.46, 6.84 [17:45:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.09, 7.31, 6.86 [17:47:32] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.89, 6.36, 6.56 [17:48:39] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 89.88 ms [17:51:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.26, 7.37, 7.02 [17:53:32] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.07, 6.64, 6.80 [18:05:08] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.87, 7.14, 6.73 [18:06:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.58, 7.09, 6.76 [18:07:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.65, 6.76, 6.66 [18:08:10] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.70, 6.50, 6.58 [18:20:55] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.90, 6.49, 6.30 [18:22:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.21, 6.02, 6.15 [18:28:16] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.38, 7.48, 6.59 [18:29:40] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.42, 6.75, 6.07 [18:31:39] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.69, 6.78, 6.17 [18:33:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.10, 8.47, 7.00 [18:34:18] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.13, 7.38, 6.90 [18:35:47] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.58, 7.66, 6.80 [18:37:45] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.34, 7.49, 6.84 [18:37:56] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 1.82, 2.09, 1.55 [18:39:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.49, 7.27, 7.05 [18:39:46] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.17, 6.76, 6.64 [18:39:53] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.38, 1.79, 1.51 [18:40:17] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.44, 6.56, 6.71 [18:41:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.42, 8.09, 7.34 [18:41:50] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.91, 1.53, 1.44 [18:43:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.51, 7.19, 7.11 [18:51:33] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.75, 6.36, 6.72 [18:53:20] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 162.48 ms [18:55:20] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 88.71 ms [19:06:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.24, 6.90, 6.55 [19:08:10] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.50, 6.30, 6.37 [19:36:38] [02ManageWiki] 07Universal-Omega edited a comment on pull request 03#278: ManageWikiExtensions: support description and name messages - 13https://git.io/JCK3m [20:10:11] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.93, 5.95, 5.28 [20:12:11] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.22, 6.34, 5.52 [20:17:10] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.15, 6.57, 5.70 [20:19:06] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.58, 7.11, 6.01 [20:19:31] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.86, 6.88, 5.90 [20:22:13] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.59, 7.30, 6.36 [20:24:30] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.93, 7.85, 6.32 [20:24:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.63, 8.14, 6.85 [20:26:25] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.14, 7.41, 6.33 [20:26:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.83, 7.30, 6.70 [20:27:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.70, 7.20, 6.72 [20:28:12] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.83, 7.62, 6.89 [20:29:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.30, 6.66, 6.58 [20:30:42] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.87, 6.73, 6.59 [20:33:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.45, 7.53, 6.94 [20:36:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.96, 7.29, 6.93 [20:37:21] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.57, 7.49, 7.11 [20:38:12] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.75, 7.78, 7.19 [20:38:27] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.35, 6.60, 6.72 [20:40:11] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.11, 7.18, 7.05 [20:41:17] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.38, 6.46, 6.79 [20:41:47] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.72, 6.19, 6.44 [20:44:10] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.02, 6.11, 6.64 [20:50:46] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.13, 3.75, 2.74 [20:51:03] PROBLEM - wiki.helioss.co - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.helioss.co could not be found [20:51:04] PROBLEM - wiki.rebirthofthenight.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.rebirthofthenight.com could not be found [20:51:05] PROBLEM - www.hoolehistoryheritagesociety.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.hoolehistoryheritagesociety.org.uk could not be found [20:51:06] PROBLEM - dariawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dariawiki.org could not be found [20:51:06] PROBLEM - infectowiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for infectowiki.com could not be found [20:51:06] PROBLEM - bots1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bots1.miraheze.wiki could not be found [20:51:07] PROBLEM - civwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for civwiki.org could not be found [20:51:08] PROBLEM - isv.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for isv.orain.org could not be found [20:51:09] PROBLEM - mabi.myomi.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mabi.myomi.net could not be found [20:52:41] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.89, 2.94, 2.56 [20:57:47] RECOVERY - wiki.helioss.co - reverse DNS on sslhost is OK: SSL OK - wiki.helioss.co reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:57:48] RECOVERY - wiki.rebirthofthenight.com - reverse DNS on sslhost is OK: SSL OK - wiki.rebirthofthenight.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:57:52] RECOVERY - bots1.miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - bots1.miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:57:53] RECOVERY - www.hoolehistoryheritagesociety.org.uk - reverse DNS on sslhost is OK: SSL OK - www.hoolehistoryheritagesociety.org.uk reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:57:54] RECOVERY - civwiki.org - reverse DNS on sslhost is OK: SSL OK - civwiki.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:57:54] RECOVERY - dariawiki.org - reverse DNS on sslhost is OK: SSL OK - dariawiki.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:57:54] RECOVERY - infectowiki.com - reverse DNS on sslhost is OK: SSL OK - infectowiki.com reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:57:55] RECOVERY - isv.orain.org - reverse DNS on sslhost is OK: SSL OK - isv.orain.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [20:57:57] RECOVERY - mabi.myomi.net - reverse DNS on sslhost is OK: SSL OK - mabi.myomi.net reverse DNS resolves to cp13.miraheze.org - CNAME OK [21:05:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.96, 6.69, 5.99 [21:07:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.60, 6.25, 5.92 [21:35:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.94, 6.35, 5.93 [21:35:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.15, 6.83, 6.13 [21:37:37] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.79, 5.69, 5.74 [21:37:52] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.08, 6.22, 5.99 [21:43:51] Voidwalker: did you see my response re cp3? [21:45:52] Yeah, still doesn't really provide any useful information. cp3 itself didn't seem to know it was down. [21:46:47] Is there any indication of this being a reoccurring issue? [21:47:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JWDpy [21:47:05] [02miraheze/mw-config] 07Universal-Omega 0306eaa30 - Update LocalExtensions.php [21:47:06] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [21:47:10] [02mw-config] 07Universal-Omega opened pull request 03#4002: Allow GlobalUsage to be used for shared uploads - 13https://git.io/JWDpS [21:48:07] miraheze/mw-config - Universal-Omega the build passed. [21:48:19] Voidwalker: happened twice in one day. I'm assuming a network incident upstream in all honesty. [21:48:37] possibly, cp3 is DO, right? [21:48:50] Voidwalker: correct [21:49:10] Their status page didn't give anything but these were a few minutes log [21:49:42] Twice exact same symptoms like you say. Icinga/DNS says it was down, cp3 shows no sign of it [21:50:10] I saw there was problems on one of their DCs yesterday, but it also seems to be the wrong area [21:58:23] PROBLEM - cp12 Disk Space on cp12 is WARNING: DISK WARNING - free space: / 4234 MB (10% inode=96%); [22:01:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JWDjN [22:01:57] [02miraheze/mw-config] 07Universal-Omega 0386c5c88 - Update ManageWikiSettings.php [22:01:59] [02mw-config] 07Universal-Omega synchronize pull request 03#4002: Allow GlobalUsage to be used for shared uploads - 13https://git.io/JWDpS [22:02:57] miraheze/mw-config - Universal-Omega the build passed. [22:03:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JWyek [22:03:10] [02miraheze/mw-config] 07Universal-Omega 037bb5073 - Update LocalSettings.php [22:03:12] [02mw-config] 07Universal-Omega synchronize pull request 03#4002: Allow GlobalUsage to be used for shared uploads - 13https://git.io/JWDpS [22:04:04] miraheze/mw-config - Universal-Omega the build passed. [22:05:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JWye0 [22:05:09] [02miraheze/mw-config] 07Universal-Omega 0356972dd - Update LocalWiki.php [22:05:10] [02mw-config] 07Universal-Omega synchronize pull request 03#4002: Allow GlobalUsage to be used for shared uploads - 13https://git.io/JWDpS [22:06:07] miraheze/mw-config - Universal-Omega the build passed. [22:06:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JWye6 [22:06:32] [02miraheze/mw-config] 07Universal-Omega 0352a5247 - Update LocalSettings.php [22:06:33] [02mw-config] 07Universal-Omega synchronize pull request 03#4002: Allow GlobalUsage to be used for shared uploads - 13https://git.io/JWDpS [22:07:33] miraheze/mw-config - Universal-Omega the build passed. [22:55:04] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.89, 1.60, 1.17 [22:57:02] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.76, 1.27, 1.10 [23:08:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JWyYh [23:08:43] [02miraheze/mw-config] 07Universal-Omega 036803139 - Update ManageWikiSettings.php [23:08:44] [02mw-config] 07Universal-Omega synchronize pull request 03#4002: Allow GlobalUsage to be used for shared uploads - 13https://git.io/JWDpS [23:09:37] miraheze/mw-config - Universal-Omega the build passed. [23:15:39] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.97, 6.34, 5.74 [23:17:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JWyOh [23:17:12] [02miraheze/mw-config] 07Universal-Omega 0393b375a - Update ManageWikiSettings.php [23:17:14] [02mw-config] 07Universal-Omega synchronize pull request 03#4002: Allow GlobalUsage to be used for shared uploads - 13https://git.io/JWDpS [23:18:10] miraheze/mw-config - Universal-Omega the build passed. [23:19:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.62, 7.87, 6.53 [23:19:37] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.44, 6.29, 5.92 [23:19:56] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.45, 7.51, 6.21 [23:21:00] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.84, 21.40, 17.43 [23:21:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.04, 7.59, 6.60 [23:21:55] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.37, 7.54, 6.39 [23:23:00] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.80, 19.16, 17.12 [23:23:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.06, 6.99, 6.31 [23:23:51] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.55, 9.01, 6.91 [23:25:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.81, 6.61, 6.43 [23:27:37] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.63, 6.79, 6.38 [23:27:51] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.31, 7.76, 6.91 [23:27:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.61, 6.78, 6.39 [23:30:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JWysb [23:30:13] [02miraheze/mw-config] 07Universal-Omega 03091c75c - Update ManageWikiSettings.php [23:30:15] [02mw-config] 07Universal-Omega synchronize pull request 03#4002: Allow GlobalUsage to be used for shared uploads - 13https://git.io/JWDpS [23:31:10] miraheze/mw-config - Universal-Omega the build passed. [23:31:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.69, 7.63, 6.82 [23:36:20] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.74, 7.54, 6.90 [23:37:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.90, 7.88, 7.28 [23:39:53] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.99, 6.50, 6.67 [23:41:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.39, 6.13, 6.68 [23:43:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JWyZE [23:43:55] [02miraheze/mw-config] 07Universal-Omega 032cb31a8 - Update ManageWikiSettings.php [23:43:56] [02mw-config] 07Universal-Omega synchronize pull request 03#4002: Allow GlobalUsage to be used for shared uploads - 13https://git.io/JWDpS [23:44:03] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.87, 6.45, 6.67 [23:44:49] miraheze/mw-config - Universal-Omega the build passed. [23:59:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JWyc7 [23:59:03] [02miraheze/mw-config] 07Universal-Omega 0398667c8 - Update LocalWiki.php [23:59:05] [02mw-config] 07Universal-Omega synchronize pull request 03#4002: Allow GlobalUsage to be used for shared uploads - 13https://git.io/JWDpS [23:59:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JWycN [23:59:16] [02miraheze/mw-config] 07Universal-Omega 03d445d20 - Update ManageWikiSettings.php [23:59:18] [02mw-config] 07Universal-Omega synchronize pull request 03#4002: Allow GlobalUsage to be used for shared uploads - 13https://git.io/JWDpS [23:59:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JWycx [23:59:44] [02miraheze/mw-config] 07Universal-Omega 037303c6f - Update LocalSettings.php [23:59:46] [02mw-config] 07Universal-Omega synchronize pull request 03#4002: Allow GlobalUsage to be used for shared uploads - 13https://git.io/JWDpS