[00:00:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.33, 7.35, 6.70 [00:03:05] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.37, 19.98, 17.24 [00:03:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.61, 8.97, 7.28 [00:04:26] RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 8245 MB (21% inode=96%); [00:04:44] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.85, 9.05, 7.52 [00:06:03] PROBLEM - privacywiki.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for privacywiki.xyz could not be found [00:06:05] PROBLEM - bots1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bots1.miraheze.wiki could not be found [00:06:07] PROBLEM - www.hoolehistoryheritagesociety.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.hoolehistoryheritagesociety.org.uk could not be found [00:06:07] PROBLEM - dariawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dariawiki.org could not be found [00:06:08] PROBLEM - infectowiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for infectowiki.com could not be found [00:06:15] PROBLEM - wiki.teessidehackspace.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.teessidehackspace.org.uk could not be found [00:06:16] PROBLEM - housing.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for housing.wiki could not be found [00:06:18] PROBLEM - pastport.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pastport.org could not be found [00:07:02] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.56, 22.19, 18.70 [00:08:04] back so going to deploy now [00:08:23] [02CreateWiki] 07The-Voidwalker closed pull request 03#229: Don't inject RequestWikiAIJob if model file is not set - 13https://git.io/JlePj [00:08:24] [02miraheze/CreateWiki] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JleSF [00:08:26] [02miraheze/CreateWiki] 07Universal-Omega 03eb851fe - Don't inject RequestWikiAIJob if model file is not set (#229) [00:08:39] Thanks, Voidwalker! [00:08:48] np [00:09:00] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.42, 22.18, 19.15 [00:09:14] oh, and I've still got some dependabot stuff to go through, hmm [00:09:23] miraheze/CreateWiki - The-Voidwalker the build passed. [00:09:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.38, 7.26, 6.64 [00:11:14] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.62, 2.70 [00:11:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.36, 7.44, 6.80 [00:12:05] [02miraheze/mediawiki] 07The-Voidwalker pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/Jle9V [00:12:07] [02miraheze/mediawiki] 07The-Voidwalker 035b3564b - Update CreateWiki [00:12:20] [02mw-config] 07The-Voidwalker closed pull request 03#4008: Set wgCreateWikiPersistentModelFile to false - 13https://git.io/JleXs [00:12:21] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jle9P [00:12:23] [02miraheze/mw-config] 07Universal-Omega 039f709da - Set wgCreateWikiPersistentModelFile to false (#4008) [00:12:43] [02miraheze/CreateWiki] 07The-Voidwalker deleted branch 03Universal-Omega-patch-3 [00:12:45] [02CreateWiki] 07The-Voidwalker deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vpJTL [00:12:46] [02mw-config] 07The-Voidwalker deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [00:12:47] RECOVERY - privacywiki.xyz - reverse DNS on sslhost is OK: SSL OK - privacywiki.xyz reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:12:48] [02miraheze/mw-config] 07The-Voidwalker deleted branch 03Universal-Omega-patch-3 [00:12:50] RECOVERY - bots1.miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - bots1.miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:12:54] RECOVERY - www.hoolehistoryheritagesociety.org.uk - reverse DNS on sslhost is OK: SSL OK - www.hoolehistoryheritagesociety.org.uk reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:12:56] RECOVERY - dariawiki.org - reverse DNS on sslhost is OK: SSL OK - dariawiki.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:12:56] RECOVERY - infectowiki.com - reverse DNS on sslhost is OK: SSL OK - infectowiki.com reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:13:09] RECOVERY - housing.wiki - reverse DNS on sslhost is OK: SSL OK - housing.wiki reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.55, 3.11, 2.62 [00:13:17] RECOVERY - wiki.teessidehackspace.org.uk - reverse DNS on sslhost is OK: SSL OK - wiki.teessidehackspace.org.uk reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:17] RECOVERY - pastport.org - reverse DNS on sslhost is OK: SSL OK - pastport.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:13:23] miraheze/mw-config - The-Voidwalker the build passed. [00:15:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.38, 7.67, 7.01 [00:17:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.64, 7.35, 6.97 [00:18:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.06, 7.51, 7.79 [00:19:00] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.73, 19.50, 19.51 [00:20:48] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 7.97, 8.07, 7.99 [00:21:37] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.12, 6.40, 6.69 [00:24:31] Is it possible to have all of my notifications completely wiped? I keep having random notifications that are months old deciding to mark themselves as unread and it is annoying [00:24:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.82, 7.44, 8.00 [00:24:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.20, 7.36, 7.74 [00:26:34] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.25, 8.34, 8.04 [00:26:35] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JleQS [00:26:37] [02miraheze/RemovePII] 07Universal-Omega 0340a7ddb - Update README.md [00:26:38] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [00:27:34] miraheze/RemovePII - Universal-Omega the build passed. [00:27:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.36, 7.38, 7.06 [00:30:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.54, 7.77, 7.91 [00:33:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.50, 7.59, 7.18 [00:34:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.98, 8.67, 8.14 [00:34:31] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.40, 7.84, 7.89 [00:35:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.48, 7.51, 7.20 [00:37:17] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.57, 7.28, 6.61 [00:38:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.98, 7.84, 7.87 [00:38:28] @Void @Universal Omega why was the AI work removed? Seems anti-professional to commitments made by SRE to utilise the technology with no good reason provided? [00:39:17] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.73, 7.25, 6.69 [00:39:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.07, 8.67, 7.68 [00:40:25] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.91, 8.22, 8.01 [00:41:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.96, 7.79, 7.50 [00:41:54] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:41:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.73, 20.64, 20.04 [00:42:53] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:42:58] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:43:19] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.54, 6.68, 6.58 [00:43:20] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:43:40] Because we may not be able to support it. If it turns out not to be the source of these problems, then I apologize. If it is the cause, then we would be minimizing the end-user impact at least until we can mitigate the performance issues. [00:43:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.91, 19.95, 19.86 [00:44:03] I will be monitoring this. If there is no improvement by this time tomorrow, I will revert the changes. [00:44:25] The process is high memory intensive, which is why SRE isolated it. There was a capacity plan made to support this which SRE have taken no interest in executing. [00:45:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.31, 7.49, 7.91 [00:46:04] With the lack of communication, I am not impressed that SRE seek to undo technology so heavily supported by the community. Seems wasteful of the months of time invested. [00:46:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.95, 7.69, 7.93 [00:46:25] Is there something tracking that at the moment? If so, I can try to move on that now, depending the access it would require. [00:47:13] Or is that just what's in the budget, and no one's actually moved to do so yet? [00:47:14] https://phabricator.miraheze.org/T7139 - which SRE are preventing on arbitrary deadlines [00:47:15] [url] ⚓ T7139 MediaWiki Capacity Proposal | phabricator.miraheze.org [00:47:52] Ah, hadn't seen that for a while, I'll look over [00:49:08] Either way, I'm hoping the AI change will be temporary. I don't want to leave it disabled if at all possible. I'll also see if there's something I can do to account for any requests that are missed during this time. [00:49:38] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.17, 7.66, 7.85 [00:49:46] But the fact it’s taken that removing positive impact technology that the community support without community engagement when plans have been proposed for a long period of time that SRE have chosen to not do because they want to wait summarises perfectly SRE’s failings over the past few months which the survey recently conducted highlights [00:51:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.29, 7.68, 7.84 [00:51:53] If attempts had been made to engage myself as the developer of the relevant technology, I could have immediately identified the memory issues are likely being linked so appropriate action could be taken rather than an entirely uninformed approach [00:53:42] Hmm, do you want me to revert then, so you can review a better approach? I fully understand that this is not ideal, and that "not ideal" is an understatement. [00:54:41] I linked the better approach, the approach agreed at the time by myself, John, Paladox and Reception [00:55:48] Though that approach now requires Ferran because of policy requiring the EM for Infra to approve servers [00:56:56] Yeah, I'm looking at that, and I don't know why it was decided in April to wait an additional 3 months before adding the servers [00:57:29] It'll reduce the migration, sure, but overall is not sustainable. [00:59:03] RhinosF1 wanted to wait for Bullseye... I think it could've been done and later upgraded, but I do understand why they wanted to wait. [00:59:57] Holding back a capacity increase to minimise a 5-10 minute job in a few months doesn’t seem like a reasonable reason in my opinion [01:00:09] Especially when capacity is an issue [01:03:19] Yeah. I completely agree that we should've moved forward with that, as we've had a problem for months, but the reason the AI was temporarily removed was because of https://phabricator.miraheze.org/T7626 as from what it seemed, the AI was a potential cause of that, however I understand the issues you have with us doing that. We should've communicated the decision and reasons better. Apologies for that. [01:03:19] [url] ⚓ T7626 redis-server is occasionally killed for OOM | phabricator.miraheze.org [01:03:31] I'm removing T7145 as a blocker to T7139 [01:03:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.22, 7.12, 7.23 [01:03:39] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.88, 6.10, 6.73 [01:04:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.47, 8.52, 7.64 [01:04:46] It’s definitely the cause, but that risk was well known when John, Paladox and Reception agreed to do it. Therefore removing it months down the line when the risk was known and agreed to is rather poor as it undervalues peoples work and effort [01:04:49] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [01:05:30] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [01:05:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.89, 7.44, 7.34 [01:05:40] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03revert-4008-Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JleAw [01:05:42] [02miraheze/mw-config] 07The-Voidwalker 0369e3419 - Revert "Set wgCreateWikiPersistentModelFile to false (#4008)" [01:05:43] [02mw-config] 07The-Voidwalker created branch 03revert-4008-Universal-Omega-patch-3 - 13https://git.io/vbvb3 [01:05:58] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:05:59] [02mw-config] 07The-Voidwalker opened pull request 03#4009: Revert "Set wgCreateWikiPersistentModelFile to false" - 13https://git.io/JleAo [01:06:15] I understand, and once again I apologize for that. [01:06:56] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:06:59] miraheze/mw-config - The-Voidwalker the build passed. [01:08:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.72, 7.38, 7.43 [01:09:19] [02mw-config] 07The-Voidwalker closed pull request 03#4009: Revert "Set wgCreateWikiPersistentModelFile to false" - 13https://git.io/JleAo [01:09:20] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JlexT [01:09:22] [02miraheze/mw-config] 07The-Voidwalker 0385fa0b3 - Revert "Set wgCreateWikiPersistentModelFile to false (#4008)" (#4009) [01:09:23] [02miraheze/mw-config] 07The-Voidwalker deleted branch 03revert-4008-Universal-Omega-patch-3 [01:09:25] [02mw-config] 07The-Voidwalker deleted branch 03revert-4008-Universal-Omega-patch-3 - 13https://git.io/vbvb3 [01:09:59] I'm re-enabling, and going to push on T7139 [01:10:21] miraheze/mw-config - The-Voidwalker the build passed. [01:10:40] According to recent changes, nothing should have been lost either. [01:13:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.42, 5.90, 6.66 [01:15:08] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: CRITICAL: Diff, 111159 files, 28.62GB, 2021-06-20 01:15:00 (4.4 weeks ago) [01:17:59] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.84, 5.83, 6.66 [01:18:04] PROBLEM - bacula2 APT on bacula2 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [01:20:06] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 20 packages available for upgrade (7 critical updates). [01:32:25] In the meantime, can we see if it's feasible to stop running AI jobs on jobrunner3 specifically? I'm hoping that could fix the redis problem without actually causing other problems. [01:35:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.78, 7.14, 6.34 [01:36:08] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.02, 7.67, 6.55 [01:37:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.34, 7.19, 6.46 [01:38:03] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.15, 7.11, 6.49 [01:39:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.98, 6.56, 6.30 [01:39:59] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.86, 6.42, 6.31 [02:02:54] Tasks are open for new servers [02:03:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.83, 6.25, 6.04 [02:05:37] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.74, 6.06, 6.00 [02:34:06] I feel kinda dumb, jobrunner3 is set as jobrunner::intensive, and we can just disable that [02:36:14] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-4 [+0/-0/±1] 13https://git.io/JlvsV [02:36:16] [02miraheze/puppet] 07The-Voidwalker 03d8ee2ea - jobrunner3 cannot run intensive jobs [02:36:17] [02puppet] 07The-Voidwalker created branch 03The-Voidwalker-patch-4 - 13https://git.io/vbiAS [02:36:44] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-4 [+0/-0/±1] 13https://git.io/JlvsS [02:36:45] [02miraheze/puppet] 07The-Voidwalker 03b193dac - Update jobrunner4.yaml [02:36:52] [02puppet] 07The-Voidwalker opened pull request 03#1817: jobrunner3 cannot run intensive jobs - 13https://git.io/Jlvs7 [02:37:51] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JlvGL [02:37:52] [02miraheze/puppet] 07The-Voidwalker 038a5b07b - jobrunner3 cannot run intensive jobs (#1817) [02:37:54] [02puppet] 07The-Voidwalker closed pull request 03#1817: jobrunner3 cannot run intensive jobs - 13https://git.io/Jlvs7 [02:42:22] [02puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-2 - 13https://git.io/vbiAS [02:42:24] [02miraheze/puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-2 [02:42:26] [02puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-4 - 13https://git.io/vbiAS [02:42:27] [02miraheze/puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-4 [02:42:40] [02miraheze/puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 [02:42:41] [02puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 - 13https://git.io/vbiAS [02:43:43] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is CRITICAL: CRITICAL: Diff, 104810 files, 52.01GB, 2021-06-20 02:42:00 (4.4 weeks ago) [02:44:41] [02puppet] 07The-Voidwalker deleted branch 03revert-1770-paladox-patch-2 - 13https://git.io/vbiAS [02:44:42] [02miraheze/puppet] 07The-Voidwalker deleted branch 03revert-1770-paladox-patch-2 [03:21:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.81, 6.96, 6.16 [03:21:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.96, 6.85, 6.12 [03:23:33] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.76, 6.48, 6.08 [03:23:33] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.91, 6.57, 6.11 [03:27:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.60, 6.68, 6.07 [03:27:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.03, 7.24, 6.47 [03:27:33] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.50, 7.36, 6.50 [03:29:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.61, 7.34, 6.60 [03:29:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.70, 7.36, 6.62 [03:31:11] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.79, 6.36, 6.10 [03:33:24] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.18, 8.56, 7.15 [03:33:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.90, 8.57, 7.17 [03:37:20] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.29, 7.94, 7.27 [03:39:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.33, 7.23, 7.16 [03:43:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.67, 5.98, 6.65 [03:43:32] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.43, 5.95, 6.63 [04:04:55] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.07, 6.74, 6.40 [04:06:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.98, 6.33, 6.28 [04:35:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.36, 7.22, 6.04 [04:35:33] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.10, 7.96, 6.30 [04:37:28] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.14, 6.67, 5.98 [04:37:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.38, 6.84, 6.08 [04:39:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.13, 6.29, 5.97 [05:48:33] PROBLEM - wlhmn.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wlhmn.tk' expires in 15 day(s) (Fri 06 Aug 2021 05:43:15 GMT +0000). [05:52:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JlvAy [05:52:02] [02miraheze/ssl] 07MirahezeSSLBot 03694be7e - Bot: Update SSL cert for wlhmn.tk [06:06:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.97, 6.49, 5.59 [06:08:05] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.96, 5.92, 5.48 [06:11:09] RECOVERY - db13 APT on db13 is OK: APT OK: 50 packages available for upgrade (0 critical updates). [06:15:16] RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [06:17:36] RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [06:20:20] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.11, 7.79, 6.50 [06:21:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.69, 6.99, 6.35 [06:23:42] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.55, 6.79, 6.34 [06:24:18] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.21, 6.66, 6.35 [06:30:08] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 104 packages available for upgrade (0 critical updates). [06:30:11] RECOVERY - wlhmn.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wlhmn.tk' will expire on Tue 19 Oct 2021 04:51:52 GMT +0000. [06:35:44] RECOVERY - db12 APT on db12 is OK: APT OK: 71 packages available for upgrade (0 critical updates). [06:38:59] [02puppet] 07RhinosF1 commented on commit 038a5b07bc5692064bfa82685bedda8bcdfd11a8b5 - 13https://git.io/Jlfke [06:59:41] RECOVERY - db11 APT on db11 is OK: APT OK: 62 packages available for upgrade (0 critical updates). [07:35:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.23, 7.39, 6.42 [07:36:03] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.67, 7.27, 6.18 [07:37:59] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.79, 6.74, 6.12 [07:39:54] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.98, 6.35, 6.25 [08:10:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.81, 7.26, 6.36 [08:14:00] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.64, 6.53, 6.26 [08:34:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.07, 7.33, 6.21 [08:36:23] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.73, 6.81, 6.10 [08:36:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.27, 6.84, 6.19 [08:38:18] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.19, 6.22, 5.97 [08:38:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.07, 6.18, 6.03 [08:50:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.63, 7.50, 6.45 [08:50:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.97, 7.65, 6.51 [08:52:48] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.14, 7.16, 6.47 [08:52:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.16, 7.33, 6.56 [08:54:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.47, 6.19, 6.19 [08:54:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.06, 6.67, 6.42 [09:04:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.09, 8.49, 6.89 [09:04:43] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.63, 7.00, 6.45 [09:06:41] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.37, 7.80, 6.84 [09:08:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.26, 6.85, 6.61 [09:09:57] !log [rhinos@jobrunner4] Disabling puppet for mediawiki update [09:10:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:10:06] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.01, 6.86, 6.86 [09:10:14] ^ that's a test not live [09:10:37] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.35, 6.14, 6.38 [09:11:17] !log [rhinos@jobrunner4] Disabling puppet for mediawiki update [09:11:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:11:59] I'm stopping puppet but no mediawiki changes [09:12:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.14, 6.24, 6.62 [09:13:18] !log [rhinos@jobrunner4] enabled puppet [09:13:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:39:56] [02puppet] 07RhinosF1 opened pull request 03#1818: Mediawiki: add some cookbooks - 13https://git.io/JlfN2 [09:40:51] That's not ready to merge [09:41:05] I'll add the files in a bit [09:55:26] [02puppet] 07RhinosF1 synchronize pull request 03#1818: Mediawiki: add some cookbooks - 13https://git.io/JlfN2 [09:55:48] [02puppet] 07RhinosF1 synchronize pull request 03#1818: Mediawiki: add some cookbooks - 13https://git.io/JlfN2 [09:56:25] [02puppet] 07RhinosF1 synchronize pull request 03#1818: Mediawiki: add some cookbooks - 13https://git.io/JlfN2 [09:57:06] [02puppet] 07RhinosF1 synchronize pull request 03#1818: Mediawiki: add some cookbooks - 13https://git.io/JlfN2 [09:57:22] [02puppet] 07RhinosF1 synchronize pull request 03#1818: Mediawiki: add some cookbooks - 13https://git.io/JlfN2 [09:59:16] [02puppet] 07RhinosF1 synchronize pull request 03#1818: Mediawiki: add some cookbooks - 13https://git.io/JlfN2 [09:59:29] [02puppet] 07RhinosF1 synchronize pull request 03#1818: Mediawiki: add some cookbooks - 13https://git.io/JlfN2 [10:09:20] [02puppet] 07RhinosF1 synchronize pull request 03#1818: Mediawiki: add some cookbooks - 13https://git.io/JlfN2 [10:10:21] [02puppet] 07RhinosF1 synchronize pull request 03#1818: Mediawiki: add some cookbooks - 13https://git.io/JlfN2 [10:10:50] [02puppet] 07RhinosF1 synchronize pull request 03#1818: Mediawiki: add some cookbooks - 13https://git.io/JlfN2 [10:17:39] PROBLEM - thepolicyhub.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for thepolicyhub.org.uk could not be found [10:24:37] [02puppet] 07RhinosF1 synchronize pull request 03#1818: Mediawiki: add some cookbooks - 13https://git.io/JlfN2 [10:25:26] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [10:27:05] Reception|away: ^ should work [10:29:26] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: breaking stuff, last run 3 minutes ago with 0 failures [10:31:18] can I ask why the puppet scripts are a) mediawiki-specific and b) written in python and not bash if they practically only shell out to other scripts [10:31:25] RECOVERY - thepolicyhub.org.uk - reverse DNS on sslhost is OK: SSL OK - thepolicyhub.org.uk reverse DNS resolves to cp13.miraheze.org - CNAME FLAT [10:31:25] !log made & reverted test3 RO to check script works [10:31:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:32:33] majavah: a) because every other mediawiki bin script is there for now, that's another issue b) I decided [10:34:07] it's part of my plan to make mediawiki upgrades less painful [10:41:04] in theory there's a lot more they could do [12:05:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.53, 7.28, 6.41 [12:05:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.08, 7.41, 6.32 [12:07:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.33, 7.17, 6.38 [12:09:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.89, 6.77, 6.33 [12:13:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.51, 7.69, 6.91 [12:16:25] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.00, 7.86, 6.92 [12:18:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.80, 7.17, 6.79 [12:19:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.94, 7.08, 7.03 [12:24:08] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.42, 6.79, 6.73 [12:27:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.50, 5.84, 6.58 [12:45:15] [02puppet] 07RhinosF1 synchronize pull request 03#1818: Mediawiki: add some cookbooks - 13https://git.io/JlfN2 [12:53:47] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'm.miraheze.org' expires in 7 day(s) (Thu 29 Jul 2021 12:53:14 GMT +0000). [12:54:03] Hmm [13:34:22] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.04, 8.14, 6.77 [13:35:55] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.37, 8.58, 7.04 [13:37:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.74, 7.98, 7.01 [13:38:20] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.00, 7.74, 6.94 [13:42:16] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.04, 6.78, 6.73 [13:49:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.11, 7.10, 6.95 [13:51:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.11, 6.83, 6.88 [13:53:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.76, 6.57, 6.79 [13:56:28] !log [rhinos@jobrunner4] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=helioswiki heliosrisingheroes_pages_full.xml --username-prefix wikia:heliosrisingheroes (END - exit=2) [13:56:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:57:22] !log moved dump to screen [13:57:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:05:55] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.76, 7.48, 6.99 [14:06:18] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.54, 8.22, 7.34 [14:08:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.30, 7.10, 7.04 [14:09:52] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.62, 6.66, 6.77 [14:13:59] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.01, 6.40, 6.79 [14:29:06] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.90, 6.14, 5.57 [14:33:05] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.55, 6.36, 5.85 [14:34:50] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.61, 8.32, 6.82 [14:35:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.60, 8.64, 7.40 [14:36:48] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.00, 7.51, 6.71 [14:38:47] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.65, 8.16, 7.03 [14:41:05] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.78, 7.26, 6.35 [14:41:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.40, 7.39, 7.24 [14:43:04] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.93, 7.01, 6.37 [14:45:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.00, 9.04, 7.84 [14:47:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.91, 6.86, 6.12 [14:49:05] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.99, 6.42, 6.36 [14:54:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.85, 8.00, 7.94 [14:57:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.12, 7.24, 8.00 [14:57:39] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.74, 6.27, 6.28 [15:03:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.62, 6.73, 7.42 [15:05:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.45, 6.53, 7.29 [15:05:53] !log [rhinos@jobrunner4] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=helioswiki heliosrisingheroes_pages_full.xml --username-prefix wikia:heliosrisingheroes (END - exit=0) [15:05:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:09:32] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.85, 5.49, 6.67 [15:10:28] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.49, 5.93, 6.76 [15:35:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.60, 7.31, 6.63 [15:41:47] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.72, 6.55, 6.55 [15:49:35] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.51, 6.94, 6.61 [15:51:33] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.22, 6.62, 6.53 [15:59:00] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.59, 7.07, 6.39 [15:59:33] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.71, 7.81, 7.03 [16:00:25] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.83, 6.58, 5.73 [16:01:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.21, 6.44, 5.85 [16:01:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.78, 7.37, 6.96 [16:03:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.12, 9.03, 7.61 [16:05:28] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.79, 6.21, 5.90 [16:06:14] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.08, 6.43, 5.97 [16:09:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.07, 7.64, 7.49 [16:10:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.68, 7.56, 7.46 [16:14:46] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.08, 7.53, 7.42 [16:15:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.07, 6.22, 5.92 [16:17:26] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.65, 6.43, 6.04 [16:17:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.08, 8.36, 7.77 [16:19:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.31, 7.96, 7.69 [16:23:54] Reception|away: can you merge my puppet pr? [16:23:58] With the bin scripts [16:24:44] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.82, 7.75, 7.75 [16:24:47] will have a look in a few minutes [16:26:34] Reception|away: we can't accidentally come out of read only next MediaWiki upgrade [16:26:59] Because the steps will include simplying typing check-read-only [16:27:54] ah, so this is part of the MW automation plan then [16:28:25] 😊 [16:31:48] Reception|away: yes [16:32:10] I tested all the scripts [16:33:10] I'm going to find a way to test the whole sequence [16:33:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.80, 8.71, 7.88 [16:34:42] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.05, 8.48, 7.97 [16:35:06] [02puppet] 07Reception123 closed pull request 03#1818: Mediawiki: add some cookbooks - 13https://git.io/JlfN2 [16:35:07] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+4/-0/±2] 13https://git.io/JlU2a [16:35:09] [02miraheze/puppet] 07RhinosF1 03cf87ba5 - Mediawiki: add some cookbooks (#1818) [16:36:23] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.59, 6.99, 6.60 [16:36:33] Reception|away: please pull puppet then make sure it deploys fine [16:36:40] To ensure I wasn't stupid with puppet syntax [16:38:23] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.40, 6.36, 6.41 [16:41:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.17, 7.89, 7.97 [16:45:22] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.88, 7.16, 6.74 [16:47:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 7.88, 8.28, 8.12 [16:49:22] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.40, 6.51, 6.60 [16:49:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.77, 7.82, 7.96 [16:51:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.06, 7.79, 7.92 [16:52:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.06, 7.57, 7.95 [16:53:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.21, 7.20, 7.69 [17:02:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.68, 7.42, 7.56 [17:03:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.94, 7.56, 7.43 [17:05:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.11, 7.19, 7.31 [17:06:22] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.80, 7.19, 7.51 [17:14:15] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.74, 7.55, 7.43 [17:16:13] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.84, 7.34, 7.37 [17:19:25] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.31, 19.74, 17.15 [17:19:33] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.22, 9.13, 7.99 [17:20:09] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.29, 9.02, 7.98 [17:21:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.02, 18.04, 16.83 [17:21:26]