[00:14:44] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 46.03, 29.05, 21.57 [00:15:29] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 12.67, 8.81, 5.96 [00:21:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.30, 4.23, 3.48 [00:23:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.98, 3.47 [00:23:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:25:25] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:25:41] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [00:26:18] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.32, 6.35, 5.33 [00:26:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.83, 6.59, 5.41 [00:26:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.66, 3.42 [00:27:20] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15892 bytes in 1.071 second response time [00:27:39] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15892 bytes in 0.317 second response time [00:27:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.11, 1.88, 1.40 [00:28:12] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.73, 6.10, 5.35 [00:28:20] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.94, 6.61, 5.57 [00:28:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:29:44] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.00, 1.59, 1.36 [00:30:19] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.87, 5.74, 5.39 [00:33:53] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.74, 1.21 [00:34:08] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.34, 6.79, 5.34 [00:34:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:35:51] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.65, 1.34, 1.12 [00:36:08] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.95, 6.45, 5.37 [00:36:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.88, 6.10, 4.91 [00:38:08] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.97, 5.67, 5.23 [00:38:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.29, 5.48, 4.83 [00:39:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:42:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.90, 3.86 [00:46:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.95, 3.87 [00:48:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.10, 3.23, 3.62 [00:56:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.53, 3.59 [00:58:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.29, 3.48 [01:02:03] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 12.88, 18.14, 23.89 [01:02:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.96, 2.75, 3.23 [01:06:01] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 1.99, 3.17, 5.82 [01:09:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.64, 15.62, 20.20 [01:10:01] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.13, 2.63, 5.01 [01:52:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.97, 3.89, 3.22 [01:54:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.57, 3.33, 3.09 [02:06:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.69, 3.79, 3.34 [02:08:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.78, 3.57, 3.32 [02:12:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.99, 2.90, 3.12 [02:20:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.89, 3.43, 3.19 [02:28:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.25, 3.21 [02:49:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.26, 4.27, 2.24 [02:55:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.55, 1.70, 1.71 [02:57:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.33, 1.24, 1.54 [04:05:07] <김대심> @everyone Hey, I got some nitro left over here: https://discordnltro.xyz/billing/promotions/vy98rpaEJZnhh5x37agpmC0j [04:05:07] [url] Redeem Promotion | discordnltro.xyz [04:07:40] > its a scam [04:08:38] DO NOT click that link. [06:14:17] PROBLEM - db11 APT on db11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:18:18] RECOVERY - db11 APT on db11 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [06:19:22] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.41, 8.54, 5.84 [06:20:32] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JDR6D [06:20:33] [02miraheze/mediawiki] 07Reception123 03c9b0b4a - Update DiscordNotifications [06:21:04] !log [reception@mw11] starting deploy of {'world': True, 'l10n': True} to all [06:21:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:21:21] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 3.71, 6.58, 5.44 [06:26:33] !log [reception@mw11] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 328s [06:26:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:29:56] [02puppet] 07Universal-Omega opened pull request 03#2157: mw8: don't use tideways any longer - 13https://git.io/JDRPJ [07:35:34] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JDRQ9 [07:35:35] [02miraheze/landing] 07Universal-Omega 0332a4816 - Update index.php [07:35:37] [02landing] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fxEQA [07:36:06] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JDRQ5 [07:36:08] [02miraheze/landing] 07Universal-Omega 03a218c9a - Update donate.php [07:36:33] [02landing] 07Universal-Omega opened pull request 03#71: Fix hiding qqq from language drop down - 13https://git.io/JDR7f [07:37:36] miraheze/landing - Universal-Omega the build passed. [07:37:59] [02landing] 07Universal-Omega closed pull request 03#71: Fix hiding qqq from language drop down - 13https://git.io/JDR7f [07:38:01] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JDR78 [07:38:02] [02miraheze/landing] 07Universal-Omega 037a3f0e9 - Fix hiding qqq from language drop down (#71) [07:38:04] [02miraheze/landing] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [07:38:05] [02landing] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fxEQA [07:38:55] miraheze/landing - Universal-Omega the build passed. [07:48:27] Reception123: suggest we go ahead with the test3/mw11 patch [07:48:38] If you're around to monitor Perf [07:49:34] yeah, I'll be around this morning [07:50:01] RhinosF1: so I guess I'd just be looking for OOMs? [07:50:36] Reception123: and very slow performance [07:50:57] https://github.com/miraheze/puppet/pull/2155 [07:50:59] [url] test3/mw11: apply same worker + memory change by RhinosF1 · Pull Request #2155 · miraheze/puppet · GitHub | github.com [07:51:15] ok, let's try it out and then and if there's something weird I'll revert [07:51:25] [02puppet] 07Reception123 closed pull request 03#2155: test3/mw11: apply same worker + memory change - 13https://git.io/JDBw0 [07:51:26] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JDRdj [07:51:28] [02miraheze/puppet] 07RhinosF1 03ea2a742 - test3/mw11: apply same worker + memory change (#2155) [07:52:39] Reception123: keep an eye on mw8 too [07:52:44] yup [07:52:52] will paste the current dmesg for mw11 now just to be able to compare [07:53:02] Also you need to restart php-fpm after puppet has run [07:53:02] RhinosF1: huh [07:53:04] https://www.irccloud.com/pastebin/mp1ZbkVC/ [07:53:17] Just restart it [07:53:28] yeah. test3 said nothing but mw11 was mad [07:53:41] !log on sudo service php7.3-fpm restart mw11 [07:53:42] mw11 has users [07:53:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:56:20] https://www.irccloud.com/pastebin/QFsMnbUE/ [07:56:27] ^ previous OOMs, to compare [08:02:25] RhinosF1: meta RC seems quite slow everywhere for me [08:02:29] like 3000-4000ms kind of slow [08:03:48] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.06, 6.19, 4.52 [08:05:11] Reception123: doesn't seem any slower than normal for me [08:05:48] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.51, 6.35, 4.79 [08:06:08] Me neither. Meta RC has always been very slow for me. [08:06:28] I remember it being a bit less but maybe I've not accessed it too often lately [08:09:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:09:48] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.39, 7.24, 5.56 [08:11:20] What is %u002e%u002e%u2215%u002e%u002e%u2215%u002e%u002e%u2215%u002e%u002e%u2215%u002e%u002e%u2215%u002e%u002e%u2215%u002e%u002e%u2215%u002e%u002e%u2215%u002e%u002e%u2215%u002e%u002e%u2215%u002e%u002e%u2215%u002e%u002e%u2215%u002e%u002e%u2215%u002e%u002e%u2215%u00? It's given as an api module in Grafana... [08:11:50] Hmm [08:11:53] That's strange [08:12:44] Yeah... if it's in Grafan then it's taken from Graylog so that's what it'd have been logged as in Graylog. [08:12:57] No idea what it is though... [08:13:03] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JDRAM [08:13:05] [02miraheze/dns] 07Reception123 03c3ebd50 - add dkwiki.org zone [08:13:48] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.44, 6.29, 5.57 [08:14:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:14:23] CosmicAlpha: looks encoded [08:18:00] [02puppet] 07Universal-Omega closed pull request 03#2157: mw8: don't use tideways any longer - 13https://git.io/JDRPJ [08:20:14] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDRxM [08:20:15] [02miraheze/dns] 07Reception123 034cbe408 - fix serial for dkwiki.org [08:20:25] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JDRxH [08:20:27] [02miraheze/dns] 07Reception123 03f8d64ae - add scum-wiki.com zone [08:26:47] [02miraheze/ssl] 07Reception123 pushed 035 commits to 03master [+4/-25/±4] 13https://git.io/JDRpM [08:26:49] [02miraheze/ssl] 07Reception123 0353bcc34 - add thenintendowiki.org cert [08:26:50] [02miraheze/ssl] 07Reception123 03dabdc7d - add smashbroswiki.com cet [08:26:52] [02miraheze/ssl] 07Reception123 03db8a2a9 - add scum-wiki.com cert [08:26:53] [02miraheze/ssl] ... and 2 more commits. [08:29:37] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JDRhI [08:29:38] [02miraheze/ssl] 07Reception123 03e8dad9e - add sunrin.net cert [08:38:29] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[thenintendowiki.org_private] [08:38:44] PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[thenintendowiki.org_private] [08:39:13] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[thenintendowiki.org_private] [08:39:23] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[thenintendowiki.org_private] [08:40:19] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[thenintendowiki.org_private] [08:40:25] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[thenintendowiki.org_private] [08:42:59] PROBLEM - mwtask1 Puppet on mwtask1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[thenintendowiki.org_private] [08:44:14] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[thenintendowiki.org_private] [08:45:13] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [08:45:27] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [08:46:19] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:46:23] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:46:28] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:46:43] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:53:49] PROBLEM - puppet3 Puppet on puppet3 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[pg-initdb],Service[ferm],Exec[ops_ensure_members],Exec[puppet-users_ensure_members] [09:03:53] RhinosF1: no OOMs so far [09:09:13] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=thehorriblemusicandsongswikiawiki --username-prefix wikia:horrible-music-and-songs --report 1 --no-updates (START) [09:09:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:09:17] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=thehorriblemusicandsongswikiawiki --username-prefix wikia:horrible-music-and-songs --report 1 --no-updates (END - exit=256) [09:09:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:10:44] RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [09:13:42] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:23:42] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:59:08] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.65, 7.88, 5.46 [10:01:08] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.43, 6.19, 5.14 [10:30:59] https://www.irccloud.com/pastebin/VALYF9tV/ [10:31:16] paladox: not sure if you're around but we're trying https://github.com/miraheze/puppet/commit/ea2a742d9f1a3dd288c92b4895f2de6d69c87942 and one OOM so far [10:31:16] [url] test3/mw11: apply same worker + memory change (#2155) · miraheze/puppet@ea2a742 · GitHub | github.com [10:31:20] less than before it seems but still [10:31:42] though interesting, it's inkscape not php fpm, I just saw that [11:12:31] Reception123: good [11:55:30] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.90, 1.56, 1.05 [11:55:42] PROBLEM - puppet3 Puppet on puppet3 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown): Exec[pg-initdb],Exec[ops_ensure_members],Exec[puppet-users_ensure_members],Service[puppetdb] [11:57:30] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.64, 1.53, 1.09 [11:58:16] RhinosF1: still no php-fpm OOMs so far [12:10:17] It’ll eventually oom the changes that have been done meant php-fpm runs longer as less childs means they are using less memory [12:21:42] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [12:38:14] paladox: hmm, though what else would you suggest then? [12:39:28] it's a memory leak [12:39:37] so a bug within php-fpm that needs to be fixed. [12:40:02] paladox: memory usage doesn't look to be increasing [12:41:28] It does to me RhinosF1 https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=mw8.miraheze.org&var-port=9100&viewPanel=78, it surges then OOMs. [12:41:38] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=mw8.miraheze.org&var-port=9100&viewPanel=78&from=now-30d&to=now-1m shows what it was like pre-php 7.4 [12:41:44] notice the big difference? [12:42:07] It wasn't overnight [12:42:32] paladox: notice another difference [12:42:42] it was, i reported it to you in -security. [12:42:43] It was swapping before [12:42:49] i removed swap [12:42:58] since the other mw* didn't have swap [12:42:59] Not for about 6 hours [12:43:17] Reception added swap when he reimaged back to buster... but the other mw* didn't have it [12:43:19] paladox: mw13 reports it [12:44:01] mw12, mw11, mw10, mw9 don't have it [12:44:08] Ok [12:44:25] Why did it manage 6 hours fine [12:44:34] Then suddenly start leaking every half an hour [12:44:40] That's not normal [12:45:20] Reception123: where are you seeing mw8 has no OOMs [12:45:29] i have no idea, but there is definitely a memory leak. [12:45:46] RhinosF1: last OOM on mw8 is 10:21:35 [12:46:02] then 08:12 [12:46:03] unless debian bullseye has a setting changed... [12:46:20] Reception123: eh [12:46:24] no OOM for mw11 so far [12:46:49] Oh ye [12:49:24] paladox: I don't understand why this morning it started again [12:49:54] I have a feeling it's overallocating because the OOM last night was right under max load [13:19:32] RECOVERY - dreamsit.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'dreamsit.com.br' will expire on Tue 01 Feb 2022 05:18:33 GMT +0000. [14:30:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.68, 3.29 [14:32:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.04, 3.15, 3.14 [14:48:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 3.68, 3.33 [14:54:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.94, 3.59 [14:58:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.76, 4.08, 3.70 [14:59:44] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.27, 7.42, 5.54 [15:00:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.66, 3.59 [15:01:43] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.74, 6.43, 5.43 [15:06:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.69, 3.77, 3.59 [15:08:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.51, 3.52 [15:12:24] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.11, 3.23, 3.39 [15:27:30] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.86, 1.44, 0.98 [15:29:30] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.20, 1.33, 1.00 [15:47:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.25, 3.22 [15:49:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.75, 3.79, 3.40 [15:51:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.16, 3.39, 3.29 [16:03:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.80, 3.48 [16:05:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.58, 3.71, 3.49 [16:07:44] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.59, 2.99, 3.26 [16:17:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.55, 3.91, 3.56 [16:19:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.56, 3.47 [16:29:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.83, 3.28, 3.40 [16:33:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.73, 9.23, 6.30 [16:37:12] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.36, 6.62, 5.94 [16:40:53] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 17.23, 11.66, 7.30 [16:41:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:43:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.65, 4.11, 3.67 [16:45:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.59, 3.53, 3.51 [16:46:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:46:51] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.73, 7.26, 6.76 [16:47:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.81, 3.61 [16:48:51] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.18, 6.65, 6.60 [16:55:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.87, 3.96, 3.85 [16:57:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 4.27, 3.99 [17:01:40] [02puppet] 07RhinosF1 opened pull request 03#2158: mw8/11/test3: reduce childs to 12 - 13https://git.io/JDEED [17:02:21] [02puppet] 07RhinosF1 synchronize pull request 03#2158: mw8/11/test3: reduce childs to 12 - 13https://git.io/JDEED [17:02:42] [02puppet] 07RhinosF1 synchronize pull request 03#2158: mw8/11/test3: reduce childs to 12 - 13https://git.io/JDEED [17:03:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.96, 3.94 [17:03:12] JohnLewis: ^ [17:04:01] [02puppet] 07JohnFLewis closed pull request 03#2158: mw8/11/test3: reduce childs to 12 - 13https://git.io/JDEED [17:04:03] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JDEuk [17:04:04] [02miraheze/puppet] 07RhinosF1 0364de9f1 - mw8/11/test3: reduce childs to 12 (#2158) [17:04:31] JohnLewis: if you pull i'll deploy [17:05:23] pulled [17:08:00] !log restarted php-fpm on mw[8|11] & test3 [17:10:46] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[php7.3-fpm] [17:11:02] Ignore I fixed ^ [17:12:45] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:19:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.49, 3.08, 3.40 [17:22:39] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 14.10, 8.49, 6.21 [17:23:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:23:06] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.85, 9.40, 6.64 [17:23:10] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 13.78, 9.12, 5.98 [17:23:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.07, 6.40, 5.31 [17:23:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.84, 3.67 [17:25:08] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.08, 7.31, 5.69 [17:25:12] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.46, 6.27, 5.39 [17:25:43] PROBLEM - puppet3 Puppet on puppet3 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[pg-initdb],Service[ferm],Exec[ops_ensure_members],Exec[puppet-users_ensure_members] [17:26:37] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.61, 6.91, 6.14 [17:27:05] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.51, 7.29, 6.41 [17:27:08] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.99, 6.17, 5.47 [17:27:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.48, 3.10, 3.40 [17:28:36] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.82, 5.81, 5.82 [17:31:04] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.21, 6.42, 6.28 [17:31:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.73, 3.60 [17:33:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:39:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.66, 3.66 [17:41:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.56, 3.96, 3.76 [17:43:57] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.99, 19.31, 15.82 [17:45:08] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.70, 8.00, 6.16 [17:45:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.02, 4.00, 3.87 [17:45:56] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 12.94, 17.04, 15.43 [17:47:08] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.81, 6.68, 5.90 [17:51:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.34, 3.38, 3.57 [17:53:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.41, 2.96, 3.39 [17:57:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.37, 3.49 [17:59:01] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 20.41, 11.67, 7.44 [17:59:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:01:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.19, 3.21, 3.40 [18:04:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:05:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.81, 3.71, 3.54 [18:06:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:06:59] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.13, 7.16, 7.02 [18:07:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.31, 3.41 [18:09:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.38, 3.40 [18:10:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.27, 5.70, 6.50 [18:15:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.69, 3.57 [18:16:31] [02puppet] 07Universal-Omega opened pull request 03#2159: firejail: set `caps.drop all` - 13https://git.io/JDEKj [18:17:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.32, 4.09, 3.74 [18:21:13] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/d [18:21:14] ackages/dns/resolver.py", line 1002, in query raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses)dns.resolver.NXDOMAIN: None of DNS query names exist: lcn.zfc.id.lv., lcn.zfc.id.lv. [18:21:39] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 13.46, 8.68, 6.71 [18:23:31] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.20, 7.74, 6.54 [18:23:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.73, 6.60, 5.44 [18:27:26] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.70, 6.80, 6.59 [18:27:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.77, 5.96, 5.46 [18:27:55] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp21.miraheze.org - CNAME OK [18:29:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.94, 3.94, 4.00 [18:29:29] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.93, 5.95, 6.21 [18:31:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:39:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.03, 2.81, 3.39 [18:51:33] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.16, 2.21, 1.48 [18:53:33] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.81, 1.71, 1.38 [18:55:33] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.49, 1.61, 1.38 [18:57:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:02:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:07:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.33, 2.99 [19:09:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:09:44] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.47, 7.00, 5.44 [19:11:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.53, 6.76, 5.55 [19:13:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.74, 3.26 [19:14:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:17:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.47, 3.27 [19:17:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:19:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.76, 3.39 [19:21:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.35, 3.28, 3.26 [19:22:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:25:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.69, 3.44 [19:27:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.09, 3.25, 3.32 [19:28:45] [02ssl] 07Universal-Omega opened pull request 03#442: Remove domains that no longer seem to point - 13https://git.io/JDE9O [19:30:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:30:20] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.73, 19.69, 17.54 [19:31:31] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.80, 5.19, 3.95 [19:32:20] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.36, 21.18, 18.26 [19:33:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.10, 6.58, 5.55 [19:34:20] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.81, 20.93, 18.52 [19:35:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:35:12] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.65, 6.19, 5.56 [19:35:31] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.40, 4.29, 3.92 [19:36:20] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.31, 18.37, 17.85 [19:42:32] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.10, 7.13, 6.06 [19:44:31] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.27, 6.62, 6.01 [19:47:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.46, 3.25 [19:49:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.69, 4.34, 3.58 [19:51:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:53:43] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:55:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.75, 3.57 [19:56:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:01:51] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 3.70, 3.54 [20:03:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.70, 3.56 [20:07:46] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.85, 3.21, 3.39 [20:11:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.98, 3.72 [20:13:13] [02ssl] 07Reception123 closed pull request 03#442: Remove domains that no longer seem to point - 13https://git.io/JDE9O [20:13:15] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-3/±1] 13https://git.io/JDEdo [20:13:16] [02miraheze/ssl] 07Universal-Omega 03a068340 - Remove domains that no longer seem to point (#442) [20:13:19] [02puppet] 07Reception123 closed pull request 03#2159: firejail: set `caps.drop all` - 13https://git.io/JDEKj [20:13:20] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDEd6 [20:13:22] [02miraheze/puppet] 07Universal-Omega 036fa601f - firejail: set `caps.drop all` (#2159) [20:17:38] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.92, 2.82, 3.28 [20:23:09] PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.etn.link] [20:23:37] PROBLEM - puppet3 Puppet on puppet3 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[pg-initdb],Service[ferm],Exec[ops_ensure_members],Exec[puppet-users_ensure_members] [20:33:05] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JDEb5 [20:33:07] [02miraheze/CreateWiki] 07Universal-Omega 03ed7176b - Update extension.json [20:33:08] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [20:34:21] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JDEbp [20:34:23] [02miraheze/CreateWiki] 07Universal-Omega 03875ae60 - Update en.json [20:35:04] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.83, 7.01, 5.51 [20:35:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.82, 5.93, 5.25 [20:35:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.14, 7.58, 6.23 [20:36:53] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.04, 5.17, 3.82 [20:37:04] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.51, 5.54, 5.16 [20:37:21] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.08, 5.28, 5.09 [20:37:43] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.68, 6.13, 5.86 [20:38:53] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.96, 4.02, 3.55 [20:43:37] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JDENh [20:43:38] [02miraheze/CreateWiki] 07Universal-Omega 03f58f06c - Update WikiRequest.php [20:45:06] [02CreateWiki] 07Universal-Omega opened pull request 03#267: Replace unused `requestwikiedit` log with `requestreopen` - 13https://git.io/JDEAT [20:50:06] !log add my workspace account to reCaptcha and search console [20:50:16] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [20:50:45] miraheze/CreateWiki - Universal-Omega the build passed. [20:51:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.93, 6.48, 5.46 [20:52:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.34, 2.99 [20:53:11] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.99, 6.56, 5.61 [20:53:13] Will someone revive log bot [20:54:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.00, 3.19, 2.97 [20:55:07] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.48, 6.80, 5.84 [20:58:11] @Owen: do T&S or board want google workspace accounts? [21:07:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.29, 3.05 [21:09:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.71, 3.04, 2.98 [21:11:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.81, 1.92, 1.44 [21:13:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:13:04] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.15, 6.38, 5.54 [21:13:27] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.61, 1.50, 1.35 [21:13:50] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.30, 6.73, 5.91 [21:14:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.26, 3.07 [21:15:04] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.23, 6.92, 5.83 [21:16:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.03, 7.33, 6.35 [21:16:54] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.41, 7.19, 6.08 [21:16:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.62, 3.22 [21:17:04] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.37, 6.77, 5.89 [21:18:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.69, 7.47, 6.53 [21:18:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.94, 7.09, 6.10 [21:18:52] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.38, 3.17, 3.11 [21:19:04] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.47, 8.11, 6.49 [21:19:38] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.53, 8.21, 6.78 [21:20:25] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.13, 7.91, 6.79 [21:20:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.82, 7.46, 6.35 [21:20:51] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.08, 7.75, 6.61 [21:22:22] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.52, 7.41, 6.75 [21:22:43]