[00:01:25] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 45315 MB (6% inode=99%); [00:02:42] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.09, 18.68, 15.27 [00:05:27] PROBLEM - graylog2 Disk Space on graylog2 is CRITICAL: DISK CRITICAL - free space: / 42576 MB (5% inode=99%); [00:05:47] PROBLEM - cnt.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cnt.miraheze.org could not be found [00:05:47] PROBLEM - trollpasta.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for trollpasta.com could not be found [00:05:48] PROBLEM - www.recyclerie-uzege.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.recyclerie-uzege.org could not be found [00:06:18] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CRITICAL: Full, 5850 files, 29.94MB, 2021-06-27 00:05:00 (2.3 weeks ago) [00:08:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.60, 21.84, 18.41 [00:09:26] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 43246 MB (6% inode=99%); [00:11:27] PROBLEM - graylog2 Disk Space on graylog2 is CRITICAL: DISK CRITICAL - free space: / 41775 MB (5% inode=99%); [00:12:28] RECOVERY - trollpasta.com - reverse DNS on sslhost is OK: SSL OK - trollpasta.com reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:12:29] RECOVERY - cnt.miraheze.org - reverse DNS on sslhost is OK: SSL OK - cnt.miraheze.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:12:40] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.27, 21.45, 18.99 [00:12:48] RECOVERY - www.recyclerie-uzege.org - reverse DNS on sslhost is OK: SSL OK - www.recyclerie-uzege.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:14:30] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.12, 3.45, 2.66 [00:15:25] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 43248 MB (6% inode=99%); [00:16:28] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 3.06, 3.34, 2.72 [00:16:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.98, 20.32, 19.22 [00:17:25] PROBLEM - graylog2 Disk Space on graylog2 is CRITICAL: DISK CRITICAL - free space: / 41958 MB (5% inode=99%); [00:20:48] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [00:24:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.72, 22.01, 20.29 [00:26:40] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.40, 23.57, 21.07 [00:27:24] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 44742 MB (6% inode=99%); [00:28:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.94, 20.75, 20.35 [00:30:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.25, 20.20, 20.16 [00:34:45] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:36:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.00, 7.08, 6.35 [00:38:44] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.99, 6.16, 6.09 [01:04:39] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [01:11:22] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.81, 18.66, 17.30 [01:13:22] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.10, 17.63, 17.06 [01:14:46] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-2 [+0/-0/±1] 13https://git.io/JCIBH [01:14:48] [02miraheze/mw-config] 07The-Voidwalker 03f07fc61 - silence mediawiki deprecation warnings post 1.35 [01:14:50] [02mw-config] 07The-Voidwalker created branch 03The-Voidwalker-patch-2 - 13https://git.io/vbvb3 [01:14:51] [02mw-config] 07The-Voidwalker opened pull request 03#3997: silence mediawiki deprecation warnings post 1.35 - 13https://git.io/JCIBQ [01:15:44] miraheze/mw-config - The-Voidwalker the build has errored. [01:16:33] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-2 [+0/-0/±1] 13https://git.io/JCIRl [01:16:34] [02miraheze/mw-config] 07The-Voidwalker 03786ded1 - Update LocalSettings.php [01:16:36] [02mw-config] 07The-Voidwalker synchronize pull request 03#3997: silence mediawiki deprecation warnings post 1.35 - 13https://git.io/JCIBQ [01:17:34] miraheze/mw-config - The-Voidwalker the build passed. [01:27:25] PROBLEM - graylog2 Disk Space on graylog2 is CRITICAL: DISK CRITICAL - free space: / 42925 MB (5% inode=99%); [01:31:27] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 44661 MB (6% inode=99%); [01:57:25] PROBLEM - graylog2 Disk Space on graylog2 is CRITICAL: DISK CRITICAL - free space: / 42831 MB (5% inode=99%); [02:17:24] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 43265 MB (6% inode=99%); [02:33:24] PROBLEM - graylog2 Disk Space on graylog2 is CRITICAL: DISK CRITICAL - free space: / 42899 MB (5% inode=99%); [03:17:04] [02mw-config] 07Universal-Omega reviewed pull request 03#3997 commit - 13https://git.io/JCLe8 [03:17:39] [02mw-config] 07Universal-Omega closed pull request 03#3996: Add privacy wiki as import option, please. - 13https://git.io/Jc7jZ [03:17:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JCLew [03:17:42] [02miraheze/mw-config] 07DebuggerByte 0333d8b2e - Add privacy wiki as import option, please. (#3996) [03:18:44] miraheze/mw-config - Universal-Omega the build passed. [03:19:08] [02mw-config] 07Universal-Omega closed pull request 03#3995: Added a setting for MultiBoilerplate - 13https://git.io/JcQsV [03:19:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JCLvJ [03:19:11] [02miraheze/mw-config] 07Njsmooth 03c1fb758 - Added a setting for MultiBoilerplate (#3995) [03:20:06] miraheze/mw-config - Universal-Omega the build passed. [03:20:26] [02mw-config] 07Universal-Omega closed pull request 03#3994: Adding PageImages Namespaces to my Wiki - 13https://git.io/Jc2E6 [03:20:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JCLvR [03:20:29] [02miraheze/mw-config] 07Njsmooth 03d82a12f - Adding PageImages Namespaces to my Wiki (#3994) [03:21:18] miraheze/mw-config - Universal-Omega the build passed. [03:36:17] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [06:20:38] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.59, 7.84, 6.48 [06:22:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.23, 7.54, 6.53 [06:22:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.92, 7.85, 6.53 [06:24:33] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.76, 6.76, 6.35 [06:28:42] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.04, 6.14, 6.19 [07:11:29] .ask Voidwalker re your patch, why not sample it at 1 in 1000 or something too [07:11:29] RhinosF1: I'll pass that on when Voidwalker is around. [07:11:33] Morning all [07:20:05] Reception123: are any imports or anything running? [07:20:17] yes [07:20:29] two imports currently [07:20:43] Ok [07:20:52] Ping me when they finish [07:21:10] ok, though I've had to restart them otherwise they would've finished a while ago [07:22:18] Yeah but I don't want to kill them again to try and work out what stupid state our servers are in at the moment [07:24:26] Southparkfan: is there any chance we could alert on low idle php-fpm workers [07:29:03] Reception123: things depooled last night [07:29:14] I think load issues need to br UBN! [07:29:16] be [07:29:32] What I don't get is why it's happening all of a sudden, everything was fine before [07:29:50] Reception123: that's what i plan to look at [07:34:48] Reception123: see 2 filed tasks [07:35:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.33, 8.10, 6.15 [07:35:48] seen, not really sure how to do that though unfortunately :( [07:35:56] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.13, 6.90, 5.67 [07:36:05] Reception123: wikimedia have one [07:36:56] you mean like one that pings people if something goes under a threashold? [07:37:00] *threshold [07:37:06] Okay that's not helpful [07:37:07] Yep [07:37:17] They #_page [07:37:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.94, 7.99, 6.35 [07:37:28] The data is in Grafana [07:37:38] We just need to make icinga scream at us based on it [07:37:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.61, 6.47, 5.67 [07:38:19] RhinosF1: and great, graylog is also down again [07:38:27] (i.e. no longer taking in any new logs) [07:39:00] Reception123: void keeps deleting search indexes [07:39:33] yeah, his PR should fix things though when we get rid of the deprecation spam [07:41:07] !log deleted oldest graylog search index [07:41:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:41:22] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.05, 6.41, 6.09 [07:41:24] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 69702 MB (9% inode=99%); [07:51:24] deleting doesn't seem to have done it [08:05:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.33, 6.96, 5.98 [08:07:23] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.86, 6.06, 5.77 [08:11:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.85, 6.45, 5.97 [08:13:22] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.77, 5.80, 5.79 [12:14:10] PROBLEM - miraheze.ml - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.ml could not be found [12:14:16] PROBLEM - db1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for db1.miraheze.wiki could not be found [12:14:16] PROBLEM - www.fortis.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.fortis.wiki could not be found [12:14:17] PROBLEM - lakehub.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lakehub.ga could not be found [12:14:17] PROBLEM - www.trollpasta.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.trollpasta.com could not be found [12:14:19] PROBLEM - thelonsdalebattalion.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for thelonsdalebattalion.co.uk could not be found [12:14:19] PROBLEM - podpedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for podpedia.org could not be found [12:14:23] PROBLEM - phabdigests.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for phabdigests.bots.miraheze.wiki could not be found [12:14:24] PROBLEM - tot.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tot.wiki could not be found [12:20:57] RECOVERY - miraheze.ml - reverse DNS on sslhost is OK: SSL OK - miraheze.ml reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [12:21:09] RECOVERY - db1.miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - db1.miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [12:21:09] RECOVERY - www.fortis.wiki - reverse DNS on sslhost is OK: SSL OK - www.fortis.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [12:21:10] RECOVERY - lakehub.ga - reverse DNS on sslhost is OK: SSL OK - lakehub.ga reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [12:21:11] RECOVERY - www.trollpasta.com - reverse DNS on sslhost is OK: SSL OK - www.trollpasta.com reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [12:21:16] RECOVERY - podpedia.org - reverse DNS on sslhost is OK: SSL OK - podpedia.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [12:21:21] RECOVERY - thelonsdalebattalion.co.uk - reverse DNS on sslhost is OK: SSL OK - thelonsdalebattalion.co.uk reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [12:21:24] RECOVERY - phabdigests.bots.miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - phabdigests.bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [12:21:25] RECOVERY - tot.wiki - reverse DNS on sslhost is OK: SSL OK - tot.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [12:57:04] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'm.miraheze.org' expires in 15 day(s) (Thu 29 Jul 2021 12:53:14 GMT +0000). [13:41:38] !log [rhinos@jobrunner4] sudo -u www-data php /srv/mediawiki/w/maintenance/rollbackEdits.php --wiki=allthetropeswiki --user=Gethbot --summary=[[phab:T7635|Request]] --bot (END - exit=32512) [13:41:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:51:48] !log [rhinos@jobrunner4] sudo -u www-data php /srv/mediawiki/w/maintenance/rollbackEdits.php --wiki=allthetropeswiki --user=Gethbot --summary=[[phab:T7635 [13:51:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:52:19] !log Logging via /usr/local/bin/logsalmsg "sudo -u www-data php /srv/mediawiki/w/maintenance/rollbackEdits.php --wiki=allthetropeswiki --user=Gethbot --summary="[[phab:T7635|Request]]" --bot (END - exit=2)" [13:52:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:04:59] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JCtNV [14:05:00] [02miraheze/mw-config] 07RhinosF1 035d06765 - ignore: save files [14:05:54] miraheze/mw-config - RhinosF1 the build passed. [14:09:25] [02mw-config] 07The-Voidwalker reviewed pull request 03#3997 commit - 13https://git.io/JCtAW [14:09:29] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-2 [+0/-0/±1] 13https://git.io/JCtAl [14:09:30] [02miraheze/mw-config] 07The-Voidwalker 03a3beaf4 - Update LocalSettings.php [14:09:32] [02mw-config] 07The-Voidwalker synchronize pull request 03#3997: silence mediawiki deprecation warnings post 1.35 - 13https://git.io/JCIBQ [14:10:31] miraheze/mw-config - The-Voidwalker the build passed. [14:10:44] Voidwalker: hi [14:10:48] [02mw-config] 07The-Voidwalker closed pull request 03#3997: silence mediawiki deprecation warnings post 1.35 - 13https://git.io/JCIBQ [14:10:49] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JCtAr [14:10:51] [02miraheze/mw-config] 07The-Voidwalker 03c61370b - silence mediawiki deprecation warnings post 1.35 (#3997) [14:10:57] hi [14:10:57] [tell] Voidwalker: 2021-07-13 - 07:11:29UTC ask Voidwalker re your patch, why not sample it at 1 in 1000 or something too [14:11:16] !log locally committed oauth changes on test3 [14:11:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:11:45] miraheze/mw-config - The-Voidwalker the build passed. [14:13:16] !log deleted three oldest graylog indexes due to disk exhaustion [14:13:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:13:25] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JCtxv [14:13:26] RECOVERY - graylog2 Disk Space on graylog2 is OK: DISK OK - free space: / 156929 MB (21% inode=99%); [14:13:27] [02miraheze/mw-config] 07RhinosF1 037223ef1 - add pubkey for oauth [14:13:54] logging back up [14:14:17] miraheze/mw-config - RhinosF1 the build passed. [14:17:30] ty Voidwalker [14:17:38] Voidwalker: did you see my UBNs? [14:20:50] I don't really have time for anything other than a quick fix unfortunately. I'll have to check in later today unless something is absolutely broken [14:20:52] [02miraheze/mw-config] 07RhinosF1 deleted branch 03The-Voidwalker-patch-2 [14:20:53] [02mw-config] 07RhinosF1 deleted branch 03The-Voidwalker-patch-2 - 13https://git.io/vbvb3 [14:22:24] Voidwalker: I have no clue what's going on [14:22:59] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 6 minutes ago with 0 failures [14:35:07] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.11, 7.29, 5.97 [14:35:46] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.92, 9.47, 6.86 [14:36:42] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.65, 7.60, 6.11 [14:38:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.84, 7.52, 6.10 [14:38:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.40, 7.49, 6.25 [14:39:08] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.62, 7.51, 6.35 [14:40:21] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.53, 7.19, 6.16 [14:40:44] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.07, 6.67, 6.11 [14:42:17] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.51, 6.60, 6.07 [14:43:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.21, 7.28, 6.91 [14:45:08] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.20, 6.37, 6.28 [14:45:34] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.31, 6.62, 6.71 [14:56:52] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.54, 6.75, 6.21 [15:02:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.34, 7.24, 6.59 [15:04:39] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.35, 6.50, 6.40 [15:04:42] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.80, 7.28, 6.46 [15:06:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.87, 6.88, 6.43 [15:08:42] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.33, 6.26, 6.25 [15:27:23] looks like we have some long running >12h php-fpm processes on mw9 [15:27:56] or not, I might be reading that wrong [15:28:17] * Voidwalker tired and doing too many things [15:34:50] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.32, 6.70, 6.29 [15:36:46] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.24, 6.35, 6.20 [15:39:55] best option is to see what kind of expensive requests we might be getting and throttle them [15:43:22] Voidwalker: ah, I just saw that there has to be at least 15% available (r.e. graylog). I tried deleting one index and that didn't do anything [15:44:55] yeah, if it gets to the flood stage watermark, elasticsearch won't remove itself from readonly until there is 15% disk space available. If you catch it before that happens, you're "fine", but still need to worry about it [15:45:48] How much storage is there anyway? [15:46:22] ah [15:50:15] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JCqOp [15:50:17] [02miraheze/ssl] 07Reception123 03f348477 - fortis.wiki redirect [15:52:40] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JCq3E [15:52:42] [02miraheze/mw-config] 07The-Voidwalker 0321c1d54 - bump down one major version [15:53:35] miraheze/mw-config - The-Voidwalker the build passed. [16:05:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.39, 6.94, 6.48 [16:09:31] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.59, 6.36, 6.36 [16:16:42] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.30, 6.97, 6.45 [16:18:42] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.38, 6.70, 6.42 [16:19:07] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.61, 6.85, 6.42 [16:21:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.76, 6.85, 6.27 [16:22:10] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.06, 7.40, 6.74 [16:24:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.85, 6.71, 6.56 [16:26:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.42, 6.54, 6.51 [16:27:31] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.49, 6.72, 6.46 [16:35:20] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.87, 7.78, 6.82 [16:36:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.98, 7.21, 6.75 [16:38:05] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.06, 6.45, 6.52 [16:39:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.29, 6.26, 6.40 [16:50:48] Voidwalker: still around? [16:50:58] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.51, 6.77, 6.28 [16:52:27] I've got some time, yeah [16:52:55] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.67, 6.72, 6.32 [16:55:00] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JCqux [16:55:02] [02miraheze/ssl] 07Reception123 03950148d - Update wiki.mirahezebots.org.crt [16:56:21] Voidwalker: wanted to ask if you had any idea what to do about https://phabricator.miraheze.org/T7597#153370 [16:56:22] [url] ⚓ T7597 Cargo table errors | phabricator.miraheze.org [16:56:27] what I should insert there [16:56:45] and also that if you're comfortable with it now we can probably silence the deprecation warnings for graylog [16:57:07] I've already merged that :) [16:57:21] and had to fix it too, because the mw docs suck [16:57:41] ah ok, sounds good then [16:57:53] and of course if you have a little time to look into V3 that would be great too :) [16:57:53] not sure on cargo, tbh, possibly something finicky with the table definition, I'll try and see [16:58:32] oh yeah, almost forgot about that, whoops [16:59:07] probably going to be a bit lower priority though, given the high priority tasks on phab [16:59:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.77, 7.14, 6.63 [16:59:35] Yeah, definitely those need to be done before [17:00:18] might have to lower the priority of T7634 and 7633 with how little information we have [17:05:35] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.95, 8.05, 6.90 [17:06:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.03, 7.28, 6.58 [17:07:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.27, 7.27, 6.75 [17:08:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.64, 7.17, 6.61 [17:09:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.14, 8.08, 7.20 [17:09:54] Voidwalker: for alert on low workers [17:09:59] The data is in grafana [17:10:08] We just need to make icinga alert based on it [17:11:04] that may be, but an alert I don't really know how to fix is not a useful alert [17:11:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.86, 7.69, 7.15 [17:12:04] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.72, 6.71, 6.57 [17:12:36] Voidwalker: yeah I'm not sure what's screwing with capacity yet [17:12:50] We are due to pool another 2 mw servers soon [17:13:19] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.27, 6.57, 6.65 [17:14:41] Which might rebalance stuff [17:15:16] that might honestly just be what we need [17:15:54] Yeah [17:16:15] Originally we said with bullseye coming soon we wanted to roll it out together [17:16:23] But we're looking at 2.5 weeks [17:16:29] Plus testing time [17:17:27] My idea was to go test3, add mw1[2-3], reimage mw[8-11], decom jobrunner[3-4] [17:17:37] Voidwalker: ^ [17:19:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.18, 6.92, 6.75 [17:19:29] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.75, 6.38, 6.77 [17:25:07] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.94, 5.80, 6.44 [17:25:15] icinga-miraheze: calm down [17:27:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.02, 6.63, 6.42 [17:29:38] RECOVERY - www.wiki.mirahezebots.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mirahezebots.org' will expire on Mon 11 Oct 2021 15:54:44 GMT +0000. [17:30:59] RECOVERY - wiki.mirahezebots.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mirahezebots.org' will expire on Mon 11 Oct 2021 15:54:44 GMT +0000. [17:31:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.84, 6.60, 6.48 [17:31:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.51, 6.91, 6.63 [17:34:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.62, 7.52, 6.81 [17:36:48] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.15, 6.63, 6.57 [17:37:31] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.54, 6.23, 6.51 [17:46:42] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.61, 6.63, 6.32 [17:48:41] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.20, 7.09, 6.52 [17:50:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.44, 7.07, 6.57 [17:52:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.48, 7.00, 6.58 [17:52:36] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.38, 7.43, 6.76 [17:53:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.51, 7.05, 6.74 [17:54:34] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.66, 7.46, 6.85 [17:55:35] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.50, 6.68, 6.65 [17:56:17] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.76, 7.80, 6.92 [17:58:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.42, 7.31, 6.85 [17:58:30] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.35, 6.79, 6.70 [17:59:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.43, 7.35, 6.92 [18:01:18] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.48, 6.87, 6.56 [18:01:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.45, 7.03, 6.84 [18:02:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.63, 6.62, 6.69 [18:02:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.13, 7.00, 6.79 [18:03:15] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.96, 7.52, 6.82 [18:05:11] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.77, 6.93, 6.70 [18:05:32] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.81, 6.65, 6.76 [18:07:06] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.77, 6.64, 6.63 [18:10:05] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.23, 6.77, 6.82 [18:10:58] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.20, 7.03, 6.81 [18:11:31] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.61, 7.08, 6.86 [18:12:15] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.75, 6.44, 6.79 [18:12:53] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.41, 6.62, 6.70 [18:13:30] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.41, 7.59, 7.08 [18:15:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.49, 6.87, 6.88 [18:16:05] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.55, 6.22, 6.60 [18:17:29] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.87, 6.17, 6.62 [18:23:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.07, 6.82, 6.73 [18:25:26] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.21, 7.21, 6.87 [18:27:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.39, 6.94, 6.78 [18:29:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.84, 7.02, 6.92 [18:29:27] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.33, 6.74, 6.73 [18:31:12] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.65, 6.61, 6.78 [18:35:07] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.65, 8.14, 7.39 [18:35:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.18, 7.46, 7.02 [18:35:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.05, 8.17, 7.19 [18:36:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.35, 7.63, 6.86 [18:37:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.83, 8.00, 7.24 [18:38:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.54, 7.50, 6.91 [18:40:54] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.56, 7.65, 7.54 [18:41:24] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.46, 7.11, 7.11 [18:43:32] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.56, 6.29, 6.70 [18:46:44] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.29, 6.36, 6.66 [18:49:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.21, 7.40, 7.20 [18:50:37] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.32, 7.13, 6.93 [18:51:25] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.36, 6.83, 7.02 [18:52:37] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.01, 7.34, 7.02 [18:54:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.76, 7.06, 6.95 [18:56:33] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.82, 6.22, 6.66 [18:56:38] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.25, 6.26, 6.75 [18:59:23] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.19, 6.06, 6.63 [19:04:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.12, 6.48, 6.57 [19:04:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.71, 7.23, 6.69 [19:05:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.99, 7.65, 7.17 [19:06:38] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.84, 6.19, 6.44 [19:06:58] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.50, 6.55, 6.51 [19:09:20] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.11, 6.34, 6.75 [19:47:47] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.51, 7.34, 6.51 [19:48:17] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.07, 7.76, 6.65 [19:49:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.80, 7.12, 6.22 [19:50:17] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.73, 7.71, 6.77 [19:50:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.34, 6.79, 6.15 [19:51:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.31, 7.51, 6.82 [19:54:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.58, 7.47, 6.77 [19:57:34] PROBLEM - cp13 Current Load on cp13 is WARNING: WARNING - load average: 1.77, 1.64, 1.22 [19:58:18] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.99, 7.69, 7.06 [19:58:45] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.59, 6.71, 6.59 [19:58:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.09, 6.65, 6.39 [19:59:19] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.32, 7.75, 7.18 [19:59:34] RECOVERY - cp13 Current Load on cp13 is OK: OK - load average: 0.96, 1.38, 1.17 [20:02:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.74, 8.40, 7.30 [20:02:47] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.22, 9.13, 7.42 [20:05:06] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.52, 7.83, 7.58 [20:08:35] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.61, 7.63, 7.43 [20:10:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.67, 7.35, 7.47 [20:12:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.84, 7.57, 7.70 [20:18:17] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.01, 6.01, 6.78 [20:18:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.17, 7.38, 7.33 [20:20:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.28, 6.75, 7.11 [20:22:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.56, 7.25, 7.23 [20:24:38] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.92, 7.18, 7.21 [20:25:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.72, 6.88, 6.94 [20:26:13] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.11, 7.50, 7.20 [20:26:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.56, 8.12, 7.73 [20:26:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.81, 8.22, 7.59 [20:28:12] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.43, 7.67, 7.32 [20:28:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.87, 7.74, 7.65 [20:28:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.76, 7.52, 7.41 [20:32:16] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.04, 7.49, 7.27 [20:32:38] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 12.54, 8.74, 7.82 [20:34:22] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.74, 9.46, 8.29 [20:34:54] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.94, 8.73, 7.67 [20:40:22] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.26, 7.30, 7.77 [20:40:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.45, 7.76, 7.65 [20:44:23] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.39, 7.78, 7.77 [20:46:38] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.06, 7.54, 7.93 [20:52:18] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.25, 7.34, 7.27 [20:52:24] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.01, 7.98, 7.83 [20:54:15] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.99, 7.47, 7.32 [20:54:28] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.46, 5.98, 6.72 [20:54:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.41, 8.27, 8.04 [20:58:11] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.07, 8.18, 7.58 [21:00:09] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.89, 7.24, 7.31 [21:00:23] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.04, 6.80, 6.86 [21:00:26] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.24, 7.85, 7.93 [21:02:20] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.76, 6.49, 6.72 [21:04:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.38, 8.74, 7.87 [21:06:18] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.41, 7.56, 7.12 [21:06:29] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.06, 8.62, 8.12 [21:08:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.98, 7.46, 7.14 [21:10:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.48, 7.42, 7.66 [21:16:31] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.63, 7.64, 7.88 [21:18:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.96, 6.07, 6.66 [21:26:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.53, 6.94, 7.83 [21:28:05] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.84, 5.87, 6.60 [21:34:51] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.50, 8.30, 7.26 [21:36:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.28, 8.21, 7.36 [21:36:38] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.21, 7.21, 7.42 [21:36:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.95, 7.52, 7.11 [21:38:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.18, 7.22, 7.11 [21:38:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.02, 7.51, 7.35 [21:38:38] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.58, 7.17, 7.37 [21:42:25] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.06, 7.03, 7.22 [21:44:36] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.40, 6.41, 6.78 [21:44:39] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.06, 7.31, 7.36 [21:46:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.40, 5.95, 6.63 [21:46:38] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.63, 7.42, 7.40 [21:48:23] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.55, 6.21, 6.79 [21:52:24] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.79, 7.01, 7.03 [21:54:39] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.13, 7.99, 7.43 [21:56:23] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.89, 6.42, 6.79 [21:58:40] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.62, 7.21, 7.28 [22:05:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.27, 7.37, 6.72 [22:06:04] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.24, 7.48, 6.79 [22:07:01]