[00:01:23] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.76, 7.39, 7.74
[00:02:15] <icinga-miraheze>	 PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:02:48] <icinga-miraheze>	 RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 8448 MB (21% inode=96%);
[00:04:04] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 16.66, 9.61, 8.13
[00:04:33] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.86, 22.15, 17.60
[00:04:40] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.11, 8.62, 8.06
[00:09:23] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.00, 5.11, 6.56
[00:10:04] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.93, 7.51, 7.81
[00:12:35] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.95, 7.18, 7.72
[00:18:07] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.01, 22.81, 21.66
[00:20:06] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 36.01, 26.81, 23.16
[00:20:28] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.35, 6.73, 7.18
[00:22:06] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.71, 7.05, 7.19
[00:22:25] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.50, 6.39, 7.01
[00:24:00] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.00, 20.87, 21.64
[00:24:04] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.50, 6.80, 7.09
[00:26:05] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.53, 6.02, 6.77
[00:26:21] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.08, 5.79, 6.64
[00:27:54] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.64, 18.30, 20.37
[00:32:17] <icinga-miraheze>	 RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:35:44] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.30, 20.75, 20.48
[00:39:45] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.58, 17.62, 19.33
[00:43:47] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.91, 22.65, 21.04
[00:44:18] <icinga-miraheze>	 PROBLEM - wiki.grottocenter.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.grottocenter.org could not be found
[00:44:18] <icinga-miraheze>	 PROBLEM - patternarchive.online - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for patternarchive.online could not be found
[00:44:19] <icinga-miraheze>	 PROBLEM - mrlove.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mrlove.wiki could not be found
[00:44:58] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.61, 5.07, 3.59
[00:46:57] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.12, 4.69, 3.61
[00:47:44] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.29, 19.47, 20.19
[00:48:53] <icinga-miraheze>	 RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.75, 3.82, 3.41
[00:51:11] <icinga-miraheze>	 RECOVERY - mrlove.wiki - reverse DNS on sslhost is OK: SSL OK - mrlove.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK
[00:51:16] <icinga-miraheze>	 RECOVERY - wiki.grottocenter.org - reverse DNS on sslhost is OK: SSL OK - wiki.grottocenter.org reverse DNS resolves to cp13.miraheze.org - CNAME OK
[00:51:18] <icinga-miraheze>	 RECOVERY - patternarchive.online - reverse DNS on sslhost is OK: SSL OK - patternarchive.online reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK
[01:22:29] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 2.92, 2.56
[01:24:28] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.62, 2.93, 2.62
[01:30:37] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.28, 6.28, 5.50
[01:32:37] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.07, 5.84, 5.44
[01:36:05] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.18, 6.65, 5.83
[01:38:05] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.17, 5.81, 5.63
[03:05:23] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.57, 6.28, 5.40
[03:07:19] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.58, 5.60, 5.25
[03:48:30] <icinga-miraheze>	 PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.06, 1.93, 1.39
[03:49:44] <Voidwalker>	 load's settled down a bit
[03:50:30] <icinga-miraheze>	 RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.10, 1.57, 1.32
[05:05:49] <icinga-miraheze>	 PROBLEM - datacrondatabase.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'datacrondatabase.com' expires in 15 day(s) (Fri 30 Jul 2021 05:01:06 GMT +0000).
[05:10:16] <Not-4d2d>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JClVg
[05:10:18] <Not-4d2d>	 [02miraheze/ssl] 07MirahezeSSLBot 034c10c64 - Bot: Update SSL cert for datacrondatabase.com
[05:33:28] <icinga-miraheze>	 RECOVERY - datacrondatabase.com - LetsEncrypt on sslhost is OK: OK - Certificate 'datacrondatabase.com' will expire on Tue 12 Oct 2021 04:10:05 GMT +0000.
[08:25:07] <RhinosF1>	 majavah: that a very interesting point
[08:25:17] <RhinosF1>	 How many sites do you think we talking?
[08:26:19] <majavah>	 too many to completely ignore them, I'd say
[08:27:34] <RhinosF1>	 majavah: hmm, i have no clue about proxies but then I guess best bet is to pay for the extra
[08:27:38] <majavah>	 i'm not finding any useful numbers online, but from experience a fairly significant amount of sites do not have v6 addresses
[08:28:01] <RhinosF1>	 Removing a jobrunner first will kill us
[08:28:08] <RhinosF1>	 It's why I wanted to do this order
[08:28:14] <RhinosF1>	 Now our load is simply proving it
[08:28:18] <RhinosF1>	 And php-Don
[08:28:20] <RhinosF1>	 From
[08:28:22] <RhinosF1>	 Fpm
[08:29:46] <majavah>	 using private addresses for non-public servers would be one option too, but it would require that all the cache proxies would have connection to the same private network (not sure if you have that setup) and you would need NAT or a proxy to reach the outside world from there
[08:31:06] <RhinosF1>	 Urgh
[08:52:38] <Not-4d2d>	 [02RemovePII] 07supertassu reviewed pull request 03#3 commit - 13https://git.io/JC03W
[08:52:40] <Not-4d2d>	 [02RemovePII] 07supertassu reviewed pull request 03#3 commit - 13https://git.io/JC03l
[08:55:42] <Not-4d2d>	 [02RemovePII] 07Universal-Omega reviewed pull request 03#3 commit - 13https://git.io/JC0cz
[08:56:58] <Not-4d2d>	 [02RemovePII] 07Universal-Omega reviewed pull request 03#3 commit - 13https://git.io/JC0Wc
[12:06:19] <Not-4d2d>	 [02mediawiki] 07R4356th opened pull request 03#2394: T7617: Install ArticlePlaceholder - 13https://git.io/JCgIo
[12:20:47] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.89, 6.37, 5.47
[12:22:34] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.28, 6.69, 5.66
[12:22:46] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.64, 6.17, 5.51
[12:24:30] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.38, 6.22, 5.61
[12:28:04] <Not-4d2d>	 [02mediawiki] 07R4356th opened pull request 03#2395: T7570: Install SkinPerPage - 13https://git.io/JCgzC
[12:29:04] <Not-4d2d>	 [02mediawiki] 07R4356th synchronize pull request 03#2395: T7570: Install SkinPerPage - 13https://git.io/JCgzC
[13:32:43] <Not-4d2d>	 [02mw-config] 07R4356th opened pull request 03#3998: T7570: Install SkinPerPage - 13https://git.io/JC2iU
[13:32:58] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.34, 6.96, 6.28
[13:33:32] <miraheze-github>	 miraheze/mw-config - R4356th the build passed.
[13:36:46] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.36, 6.97, 6.13
[13:36:59] <Not-4d2d>	 [02ManageWiki] 07RhinosF1 commented on pull request 03#278: ManageWikiExtensions: support description and name messages - 13https://git.io/JC2DB
[13:38:43] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.83, 6.45, 6.02
[13:38:56] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.06, 7.49, 6.90
[13:40:54] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.97, 6.77, 6.72
[13:51:55] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.13, 6.58, 5.95
[13:55:54] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.36, 6.47, 6.12
[14:46:04] <icinga-miraheze>	 PROBLEM - chouverse.cludisciples.cyou - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for chouverse.cludisciples.cyou could not be found
[15:31:04] <MH-Discord>	 <MarkusRost> Have there been any news about the cause of the OAuth2 issue? https://phabricator.miraheze.org/T7386
[15:31:05] <MirahezeBot>	 [url] ⚓ T7386 PHP Fatal Error from /srv/mediawiki/w/extensions/OAuth/vendor/league/oauth2-server/src/ResponseTypes/BearerTokenResponse.php: Method MediaWiki\Extensions\OAuth\Entity\AccessTokenEntity::__toS | phabricator.miraheze.org
[16:05:05] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.22, 6.52, 5.57
[16:07:05] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.48, 5.98, 5.48
[16:12:03] <MH-Discord>	 <Ugochimobi> Hello @System Administrators when will this issue of Wiki owners not having their normal founder's right (administrator & bureaucrat) on their wiki, be fixed?
[16:14:31] <RhinosF1>	 When we know how to fix it
[16:14:49] <RhinosF1>	 @MarkusRost: not beyond Reception123 on test3 the other day
[16:17:10] <MH-Discord>	 <Ugochimobi> RhinosF1: Wow! This must be somewhat technical
[16:53:11] <MH-Discord>	 <Universal Omega> It doesn't always happen and it's hard to debug so we really have no idea how to fix it. But I do plan to look in-debth soon.
[16:53:56] <MH-Discord>	 <Universal Omega> Reception123 tried on Test3 with issues. I plan to take a look hopefully today or tomorrow also.
[16:54:30] <Not-4d2d>	 [02ManageWiki] 07Universal-Omega commented on pull request 03#278: ManageWikiExtensions: support description and name messages - 13https://git.io/JCK3m
[16:56:30] <Not-4d2d>	 [02mediawiki] 07Universal-Omega reviewed pull request 03#2395 commit - 13https://git.io/JCKcL
[16:57:35] <MH-Discord>	 <Ugochimobi> Please do😊 we await the resolution
[16:57:38] <RhinosF1>	 @Universal_Omega: is that the accounts issue?
[16:57:44] <RhinosF1>	 Its a resource issue
[17:02:18] <MH-Discord>	 <Universal Omega> RhinosF1: i know but I still plan to look into it to figure a way to hopeful resolve it but I guess may not be able to.
[17:03:50] <RhinosF1>	 Universal_Omega: that's why I want mw12
[17:03:59] <RhinosF1>	 Earlier than planned
[17:04:15] <RhinosF1>	 But beyond somehow finding a way to reduce load, I'm not sure
[17:04:31] <MH-Discord>	 <Universal Omega> RhinosF1: oh that makes sense then. But yeah mw12 would be great!
[17:05:16] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.36, 7.34, 6.11
[17:05:40] <RhinosF1>	 We planned to replace jobrunner[34] with mw1[23] when we did bullseye Universal_Omega but I'm concerned about us being this bad for potentially another month
[17:05:52] <RhinosF1>	 So I think we're best to bring mw12 in now
[17:06:04] <RhinosF1>	 It's extra work but hopefully not too bad
[17:06:05] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.40, 6.89, 6.12
[17:06:15] <Not-4d2d>	 [02mediawiki] 07R4356th synchronize pull request 03#2395: T7570: Install SkinPerPage - 13https://git.io/JCgzC
[17:06:20] <RhinosF1>	 Reimage is fairly quick
[17:07:13] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.54, 6.13, 5.80
[17:07:51] <Not-4d2d>	 [02mediawiki] 07R4356th reviewed pull request 03#2395 commit - 13https://git.io/JC632
[17:08:06] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.44, 6.00, 5.89
[17:43:24] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JCXmU
[17:43:26] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega 038c28818 - Update SpecialMirahezeSurvey.php
[17:43:27] <Not-4d2d>	 [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://git.io/fQRGX
[17:43:48] <Not-4d2d>	 [02MirahezeMagic] 07Universal-Omega opened pull request 03#272: Allow disabling survey with confif - 13https://git.io/JCXYe
[17:44:38] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[17:45:18] <Not-4d2d>	 [02DataDump] 07R4356th opened pull request 03#31: refactor(registration): Convert to new HookHandlers interface - 13https://git.io/JCXcE
[17:46:12] <miraheze-github>	 miraheze/DataDump - R4356th the build passed.
[17:50:51] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JCX7h
[17:50:53] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega 038f83137 - Update extension.json
[17:50:54] <Not-4d2d>	 [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#272: Allow disabling survey with confif - 13https://git.io/JCXYe
[17:51:01] <Not-4d2d>	 [02MirahezeMagic] 07Universal-Omega edited pull request 03#272: Allow disabling survey with config - 13https://git.io/JCXYe
[17:51:48] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[17:55:32] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JC1GZ
[17:55:33] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega 0313dbf4b - Update en.json
[17:55:35] <Not-4d2d>	 [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#272: Allow disabling survey with config - 13https://git.io/JCXYe
[17:56:25] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[17:57:08] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JC14u
[17:57:10] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega 03655ba0c - Update qqq.json
[17:57:11] <Not-4d2d>	 [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#272: Allow disabling survey with config - 13https://git.io/JCXYe
[17:58:12] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[18:01:47] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JC1My
[18:01:48] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega 03bfc49ea - Update en.json
[18:01:50] <Not-4d2d>	 [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#272: Allow disabling survey with config - 13https://git.io/JCXYe
[18:03:07] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[18:05:02] <Not-4d2d>	 [02MirahezeMagic] 07Universal-Omega edited pull request 03#272: Allow disabling survey with config - 13https://git.io/JCXYe
[18:12:31] <Not-4d2d>	 [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JCMlR
[18:12:33] <Not-4d2d>	 [02miraheze/RemovePII] 07Universal-Omega 0380fcabc - Update README.md
[18:12:34] <Not-4d2d>	 [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN
[18:13:30] <miraheze-github>	 miraheze/RemovePII - Universal-Omega the build passed.
[18:30:13] <Not-4d2d>	 [02mw-config] 07Universal-Omega reviewed pull request 03#3998 commit - 13https://git.io/JCDO0
[18:58:02] <Not-4d2d>	 [02mw-config] 07R4356th synchronize pull request 03#3998: T7570: Install SkinPerPage - 13https://git.io/JC2iU
[18:58:12] <Not-4d2d>	 [02mw-config] 07R4356th reviewed pull request 03#3998 commit - 13https://git.io/JCy8j
[18:58:53] <miraheze-github>	 miraheze/mw-config - R4356th the build passed.
[19:01:49] <Not-4d2d>	 [02mw-config] 07Universal-Omega assigned pull request 03#3998: T7570: Install SkinPerPage - 13https://git.io/JC2iU
[19:04:37] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.16, 7.67, 6.33
[19:04:38] <icinga-miraheze>	 PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates).
[19:04:54] <icinga-miraheze>	 PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates).
[19:05:05] <icinga-miraheze>	 PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates).
[19:05:07] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.42, 6.90, 5.98
[19:05:08] <icinga-miraheze>	 PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates).
[19:05:10] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.30, 7.63, 6.31
[19:05:30] <icinga-miraheze>	 PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates).
[19:05:47] <icinga-miraheze>	 PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates).
[19:05:50] <icinga-miraheze>	 PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates).
[19:06:11] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.01, 7.42, 6.27
[19:07:03] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.66, 7.30, 6.22
[19:07:06] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.23, 7.51, 6.42
[19:08:12] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.83, 7.86, 6.60
[19:08:34] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.14, 7.41, 6.59
[19:09:00] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.15, 6.95, 6.22
[19:10:11] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.95, 8.08, 6.81
[19:10:55] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.51, 6.49, 6.15
[19:11:00] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.99, 6.56, 6.28
[19:12:37] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.40, 7.31, 6.69
[19:14:37] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.45, 7.02, 6.67
[19:14:48] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.70, 7.26, 6.57
[19:14:58] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.65, 7.26, 6.65
[19:16:09] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.13, 7.79, 7.19
[19:16:36] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.03, 7.37, 6.84
[19:18:35] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.86, 6.76, 6.67
[19:18:40] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.81, 6.38, 6.40
[19:18:51] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.63, 7.78, 6.95
[19:20:08] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.91, 7.59, 7.15
[19:20:49] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.82, 7.85, 7.08
[19:22:07] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.25, 7.39, 7.13
[19:22:30] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.22, 7.10, 6.85
[19:24:07] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.22, 7.64, 7.25
[19:26:06] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.67, 7.67, 7.31
[19:26:26] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.58, 6.64, 6.71
[19:26:39] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.45, 7.02, 6.75
[19:28:36] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.45, 6.68, 6.79
[19:30:38] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.17, 6.73, 6.71
[19:32:17] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.06, 7.19, 6.95
[19:33:07] <icinga-miraheze>	 PROBLEM - mw9 APT on mw9 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected.
[19:34:15] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.77, 7.99, 7.23
[19:35:08] <icinga-miraheze>	 PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates).
[19:36:05] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.77, 6.35, 6.78
[19:36:12] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 8.00, 7.97, 7.32
[19:36:23] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.63, 7.89, 7.18
[19:38:19] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.46, 7.22, 7.01
[19:42:13] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.72, 6.56, 6.79
[19:46:05] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.90, 6.14, 6.67
[19:50:05] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.98, 7.26, 7.02
[19:52:04] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.59, 8.11, 7.31
[19:52:06] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.83, 7.68, 7.00
[19:54:04] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.17, 7.45, 7.17
[19:54:05] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.91, 7.99, 7.22
[19:58:04] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.78, 6.13, 6.69
[20:02:05] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.31, 6.02, 6.65
[20:08:05] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.80, 5.97, 6.62
[20:21:40] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.97, 7.09, 6.57
[20:22:06] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.58, 6.46, 5.98
[20:24:06] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.49, 6.32, 6.00
[20:25:32] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.76, 6.13, 6.34
[20:34:51] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.03, 6.20, 5.95
[20:36:50] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.12, 5.67, 5.78
[21:06:18] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.96, 6.18, 5.45
[21:08:17] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.03, 5.66, 5.35
[21:33:07] <icinga-miraheze>	 PROBLEM - mw9 APT on mw9 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected.
[21:35:07] <icinga-miraheze>	 PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates).
[23:03:07] <icinga-miraheze>	 PROBLEM - mw9 APT on mw9 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected.
[23:05:09] <icinga-miraheze>	 PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates).
[23:08:29] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JCddK
[23:08:31] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega 039c21f9c - Update SpecialMirahezeSurvey.php
[23:08:32] <Not-4d2d>	 [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#272: Allow disabling survey with config - 13https://git.io/JCXYe
[23:09:14] <icinga-miraheze>	 PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 0 failures
[23:09:26] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[23:10:35] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JCdpx
[23:10:37] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega 03a16ed5e - Update SpecialMirahezeSurvey.php
[23:10:38] <Not-4d2d>	 [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#272: Allow disabling survey with config - 13https://git.io/JCXYe
[23:11:36] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[23:13:36] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:13:43] <icinga-miraheze>	 PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[23:13:50] <icinga-miraheze>	 PROBLEM - ns1 Current Load on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:14:11] <icinga-miraheze>	 PROBLEM - ns1 SSH on ns1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:14:51] <icinga-miraheze>	 PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[23:15:16] <icinga-miraheze>	 PROBLEM - ns1 APT on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:15:16] <icinga-miraheze>	 PROBLEM - ns1 NTP time on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:15:22] <icinga-miraheze>	 PROBLEM - ping4 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[23:16:17] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:16:27] <icinga-miraheze>	 PROBLEM - Host ns1 is DOWN: PING CRITICAL - Packet loss = 100%
[23:16:43] <Not-4d2d>	 [02MirahezeMagic] 07Universal-Omega closed pull request 03#272: Allow disabling survey with config - 13https://git.io/JCXYe
[23:16:45] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JCFsu
[23:16:46] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega 0375631a0 - Allow disabling survey with config (#272)
[23:16:48] <Not-4d2d>	 [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://git.io/fQRGX
[23:16:49] <Not-4d2d>	 [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-5
[23:17:38] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[23:18:02] <icinga-miraheze>	 PROBLEM - mw11 PowerDNS Recursor on mw11 is CRITICAL: Domain miraheze.org was not found by the server
[23:19:03] <Not-4d2d>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JCFWj
[23:19:05] <Not-4d2d>	 [02miraheze/mediawiki] 07Universal-Omega 03ec19aec - Update MirahezeMagic
[23:20:46] <Not-4d2d>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-7 [+0/-0/±1] 13https://git.io/JCFRF
[23:20:48] <Not-4d2d>	 [02miraheze/mw-config] 07Universal-Omega 03383651a - Disable survey
[23:20:49] <Not-4d2d>	 [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-7 - 13https://git.io/vbvb3
[23:20:51] <Not-4d2d>	 [02mw-config] 07Universal-Omega opened pull request 03#3999: Disable survey - 13https://git.io/JCFRj
[23:21:47] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[23:24:23] <Not-4d2d>	 [02mediawiki] 07Universal-Omega closed pull request 03#2395: T7570: Install SkinPerPage - 13https://git.io/JCgzC
[23:24:25] <Not-4d2d>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+1/-0/±1] 13https://git.io/JCFoz
[23:24:26] <Not-4d2d>	 [02miraheze/mediawiki] 07R4356th 03f39691a - T7570: Install SkinPerPage (#2395)
[23:27:57] <icinga-miraheze>	 RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[23:33:51] <icinga-miraheze>	 PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 4 minutes ago with 0 failures
[23:38:37] <Not-4d2d>	 [02mw-config] 07Universal-Omega closed pull request 03#3998: T7570: Install SkinPerPage - 13https://git.io/JC2iU
[23:38:38] <Not-4d2d>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JCbZf
[23:38:40] <Not-4d2d>	 [02miraheze/mw-config] 07R4356th 0302a290b - T7570: Install SkinPerPage (#3998)
[23:39:35] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[23:44:20] <Universal_Omega>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki && sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki=loginwiki on mw* / jbr* / test3
[23:44:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:46:32] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.94, 7.61, 6.10
[23:48:40] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.30, 7.49, 5.76
[23:49:03] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.51, 8.22, 6.53
[23:50:37] <icinga-miraheze>	 PROBLEM - cp12 Disk Space on cp12 is WARNING: DISK WARNING - free space: / 4222 MB (10% inode=96%);
[23:51:19] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.74, 6.30, 5.57
[23:51:41] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.88, 7.32, 6.47
[23:52:05] <icinga-miraheze>	 RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[23:52:19] <Not-4d2d>	 [02mw-config] 07Universal-Omega closed pull request 03#3999: Disable survey - 13https://git.io/JCFRj
[23:52:21] <Not-4d2d>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JCbxH
[23:52:22] <Not-4d2d>	 [02miraheze/mw-config] 07Universal-Omega 0321ddb58 - Disable survey (#3999)
[23:52:24] <Not-4d2d>	 [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7 - 13https://git.io/vbvb3
[23:52:25] <Not-4d2d>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7
[23:53:15] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[23:54:04] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.24, 6.47, 6.27
[23:57:41] <Not-4d2d>	 [02miraheze/mw-config] 07Universal-Omega pushed 0314 commits to 03Universal-Omega-patch-6 [+2/-0/±21] 13https://git.io/JCNYF
[23:57:43] <Not-4d2d>	 [02miraheze/mw-config] 07Universal-Omega 039cb7ef7 - Merge branch 'master' into Universal-Omega-patch-6
[23:57:44] <Not-4d2d>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3991: Remove unnecessary extension loads - 13https://git.io/JcqPl
[23:58:38] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.