[00:02:20] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 5029 MB (20% inode=94%); [00:03:22] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.54, 22.22, 18.07 [00:03:37] RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 6364 MB (16% inode=96%); [00:04:07] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:17] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 16.82, 10.34, 7.47 [00:04:55] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.41, 8.11, 6.54 [00:05:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 51.222.25.132/cpweb [00:06:54] PROBLEM - miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.wiki could not be found [00:07:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:09:47] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.59, 3.10, 2.45 [00:10:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.81, 7.81, 6.99 [00:11:17] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.02, 23.91, 21.10 [00:11:43] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.64, 2.93, 2.46 [00:12:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.27, 8.39, 7.30 [00:13:48] RECOVERY - miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:14:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.20, 7.35, 7.71 [00:14:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.33, 7.84, 7.22 [00:15:14] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.89, 22.93, 21.14 [00:17:11] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.49, 22.30, 21.10 [00:19:08] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.37, 24.73, 22.13 [00:20:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.01, 8.63, 8.04 [00:20:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.31, 7.99, 7.35 [00:21:05] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.77, 22.82, 21.77 [00:24:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.77, 7.24, 7.65 [00:24:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.47, 7.38, 7.27 [00:28:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.83, 6.13, 6.78 [00:28:55] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.23, 18.51, 20.18 [00:32:05] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:36:43] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.57, 21.28, 20.57 [00:36:54] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.19, 7.52, 7.20 [00:38:42] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.08, 21.47, 20.78 [00:40:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.38, 6.07, 6.71 [00:40:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.89, 18.91, 19.94 [00:40:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.20, 6.29, 6.76 [01:04:09] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.74, 21.38, 18.36 [01:04:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.11, 7.87, 6.52 [01:05:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.78, 8.49, 6.76 [01:06:07] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.68, 21.32, 18.67 [01:06:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.49, 2.89 [01:06:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.63, 7.04, 6.39 [01:06:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.12, 7.57, 6.65 [01:08:05] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.57, 20.14, 18.59 [01:08:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.77, 3.18, 2.85 [01:08:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.76, 6.19, 6.15 [01:08:57] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.58, 6.53, 6.38 [01:21:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.14, 2.92 [01:23:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 2.95, 2.88 [01:29:56] Oh welcome back JohnLewis and paladox! [01:30:27] thanks! [01:32:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.47, 3.19 [01:36:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.07, 3.10 [01:52:53] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.57, 2.05, 1.44 [01:53:55] PROBLEM - mon2 Puppet on mon2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [01:54:48] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.84, 1.61, 1.36 [02:03:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JRacS [02:03:43] [02miraheze/puppet] 07paladox 035e52a0a - base::puppet: Fix apt key [02:05:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRaCm [02:05:41] [02miraheze/puppet] 07paladox 03dc7c518 - Fix [02:07:36] PROBLEM - ldap2 Puppet on ldap2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:07:38] PROBLEM - puppet3 Puppet on puppet3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:08:34] PROBLEM - dbbackup1 Puppet on dbbackup1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:09:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRaWT [02:09:25] [02miraheze/puppet] 07paladox 03d623358 - Fix [02:09:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.21, 3.03 [02:10:21] !log puppet3: sudo apt-key del "6F6B 1550 9CF8 E59E 6E46 9F32 7F43 8280 EF8D 349F" [02:10:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:10:52] !log all host: sudo apt-key del "6F6B 1550 9CF8 E59E 6E46 9F32 7F43 8280 EF8D 349F" [02:11:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:11:35] PROBLEM - services4 Puppet on services4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:11:41] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:11:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.61, 2.97, 2.97 [02:14:17] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.76, 22.38, 17.56 [02:14:18] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.28, 8.91, 6.73 [02:14:37] RECOVERY - dbbackup1 Puppet on dbbackup1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:14:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.50, 8.93, 6.76 [02:15:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRall [02:15:27] [02miraheze/puppet] 07paladox 03018abf4 - puppet3: Reduce heap usage to 2gb for puppetserver [02:15:51] PROBLEM - db12 Puppet on db12 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown) [02:16:06] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown) [02:16:24] PROBLEM - gluster3 Puppet on gluster3 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown) [02:16:26] PROBLEM - mail2 Puppet on mail2 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 2 minutes ago with 13 failures. Failed resources (up to 3 shown) [02:16:35] PROBLEM - cloud5 Puppet on cloud5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown) [02:16:38] PROBLEM - db11 Puppet on db11 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 2 minutes ago with 5 failures. Failed resources (up to 3 shown) [02:16:40] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 694 failures. Last run 2 minutes ago with 694 failures. Failed resources (up to 3 shown) [02:16:52] PROBLEM - services3 Puppet on services3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:16:53] PROBLEM - gluster4 Puppet on gluster4 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown) [02:16:58] PROBLEM - test4 Puppet on test4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:17:07] PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown) [02:17:26] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown) [02:17:27] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown) [02:17:32] PROBLEM - cp13 Puppet on cp13 is CRITICAL: CRITICAL: Puppet has 677 failures. Last run 3 minutes ago with 677 failures. Failed resources (up to 3 shown) [02:17:37] PROBLEM - phab2 Puppet on phab2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:17:42] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 696 failures. Last run 2 minutes ago with 696 failures. Failed resources (up to 3 shown) [02:17:52] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 692 failures. Last run 3 minutes ago with 692 failures. Failed resources (up to 3 shown) [02:17:53] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 686 failures. Last run 3 minutes ago with 686 failures. Failed resources (up to 3 shown) [02:17:53] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown) [02:18:01] PROBLEM - cloud4 Puppet on cloud4 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/puppet],File[/etc/update-motd.d/97-last-puppet-run] [02:18:03] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 694 failures. Last run 2 minutes ago with 694 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/robots.php],File[/srv/mediawiki/sitemap.php],File[/usr/local/bin/fileLockScript.sh],File[/usr/local/bin/foreachwikiindblist] [02:18:16] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 411 failures. Last run 2 minutes ago with 411 failures. Failed resources (up to 3 shown): File[/usr/local/bin/nginx-prometheus-exporter],File[antiguabarbudacalypso.com],File[antiguabarbudacalypso.com_private],File[spiral.wiki] [02:18:22] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 704 failures. Last run 3 minutes ago with 704 failures. Failed resources (up to 3 shown) [02:19:06] What is with all the failures? [02:20:12] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.94, 23.53, 19.72 [02:20:27] RECOVERY - mail2 Puppet on mail2 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [02:20:34] RECOVERY - cloud5 Puppet on cloud5 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [02:20:35] RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [02:20:46] RECOVERY - services3 Puppet on services3 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [02:20:50] RECOVERY - gluster4 Puppet on gluster4 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [02:21:02] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:21:14] puppet OOM'd [02:21:20] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:21:22] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:21:26] RECOVERY - cp13 Puppet on cp13 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [02:21:32] RECOVERY - phab2 Puppet on phab2 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [02:21:34] RECOVERY - services4 Puppet on services4 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [02:21:37] RECOVERY - ldap2 Puppet on ldap2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:21:45] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [02:21:47] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:21:50] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:21:51] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [02:21:52] RECOVERY - mon2 Puppet on mon2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:21:55] RECOVERY - cloud4 Puppet on cloud4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:22:08] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:22:15] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [02:22:22] RECOVERY - gluster3 Puppet on gluster3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:22:39] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:22:47] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 198 failures. Last run 2 minutes ago with 198 failures. Failed resources (up to 3 shown): File[munwiki.info],File[munwiki.info_private],File[mh142.com],File[mh142.com_private] [02:22:56] RECOVERY - test4 Puppet on test4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:23:47] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:23:55] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:24:15] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:26:08] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.93, 19.80, 19.49 [02:28:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 3.89, 6.79, 7.75 [02:28:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.45, 3.18 [02:28:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.33, 7.23, 7.95 [02:30:18] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.51, 3.10, 3.09 [02:34:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRaRc [02:34:20] [02miraheze/puppet] 07paladox 0339f7b01 - puppetserver: Increase ReservedCodeCacheSize to 1g [02:39:37] PROBLEM - ldap2 Puppet on ldap2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown) [02:40:18] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.12, 5.33, 6.46 [02:40:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.62, 5.31, 6.61 [02:42:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRa05 [02:42:14] [02miraheze/puppet] 07paladox 0359d6354 - puppetdb: Increase Xmx to 1G [02:44:45] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [02:45:56] PROBLEM - cloud4 Puppet on cloud4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:46:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.26, 3.03 [02:48:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.62, 2.95, 2.94 [02:51:58] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.60, 21.72, 18.86 [02:55:58] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.65, 18.56, 18.32 [03:05:36] RECOVERY - ldap2 Puppet on ldap2 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [03:13:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.47, 3.22 [03:13:55] RECOVERY - cloud4 Puppet on cloud4 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:15:16] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.39, 2.70, 2.96 [03:24:30] paladox: you may want to update your phab profile :) (says you aren’t sre) [03:25:01] done [03:29:34] welcome back!! i missed you both :) [03:37:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRaPX [03:37:02] [02miraheze/puppet] 07paladox 03367bbd2 - nginx: Remove delaycompress from logrotate file [03:39:22] !log cp12: gzip access.log.1 [03:39:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:42:55] !log cp13: gzip access.log.1 [03:42:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:43:48] !log cp3: gzip access.log.1 [03:43:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:27:34] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [04:30:41] Okay [04:30:56] Whoops. [04:34:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.33, 7.04, 5.19 [04:36:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.14, 6.47, 5.20 [04:49:04] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 34.03, 24.19, 18.90 [04:51:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.52, 21.93, 18.74 [04:51:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.96, 6.74, 5.94 [04:52:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.81, 6.48, 5.63 [04:53:10] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.82, 6.02, 5.78 [04:54:55] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.96, 25.25, 20.74 [04:56:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.07, 21.94, 20.08 [04:58:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.74, 19.23, 19.35 [05:00:52] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.21, 6.16, 5.95 [05:21:40] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.41, 6.23, 5.69 [05:23:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.52, 5.56, 5.51 [06:00:47] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRVqc [06:00:48] [02miraheze/dns] 07Reception123 03b9f5b2c - add acme challenge to miraheze.wiki [06:03:54] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRVmf [06:03:55] [02miraheze/dns] 07Reception123 036327091 - Update miraheze.wiki [06:06:02] !log rebooted ns1 (puppet being killed) [06:06:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:27:09] RECOVERY - ns1 APT on ns1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [07:00:04] PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [07:04:58] PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [07:05:01] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRV4P [07:05:03] [02miraheze/dns] 07Reception123 030c57b0e - try to add acme challenge for bots again [07:05:14] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [07:05:22] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [07:05:26] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [07:05:52] PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [07:06:04] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:06:19] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [07:09:24] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [07:11:57] PROBLEM - services4 APT on services4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [07:12:57] PROBLEM - services3 APT on services3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [07:19:06] Reception123: puppet failure on ns1? [07:25:24] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [07:25:38] JohnLewis and paladox you're most welcomed. [07:26:14] John is not connected to IRC right now. [07:28:30] Universal_Omega: yeah, I don't know why it keeps doing that. Before it was killed [07:29:16] Universal_Omega: Oops, why though? [07:29:27] What about paladox? [07:29:54] Reception123: earlier puppet OOMed, causing failures on every single server. Not sure if it's related but might be. Not my area of expertise though. [07:30:17] ah [07:30:55] Ugochimobi: IRC disconnects you after awhile for inactivity, So that's why John isn't. Pala_dox is connected by probably away. [07:31:23] (Underscored username to not do anymore pings. Please don't continue to mention him or it will ping every time) [07:31:42] Oh yeah, that's true [09:05:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.70, 6.92, 5.89 [09:07:00] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.04, 6.52, 5.86 [09:20:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.00, 5.99, 5.55 [09:24:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.77, 7.84, 6.33 [09:26:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.47, 7.14, 6.26 [09:30:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.54, 7.06, 6.34 [09:30:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.46, 8.14, 6.85 [09:32:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.10, 7.00, 6.38 [09:36:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.49, 7.68, 6.93 [09:38:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.18, 6.74, 6.67 [09:38:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.72, 7.60, 7.23 [09:42:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.77, 6.31, 6.80 [09:50:26] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.03, 8.63, 7.25 [09:51:40] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.72, 9.76, 8.08 [09:56:20] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.44, 6.81, 7.03 [09:57:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.35, 7.27, 7.55 [10:00:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.70, 6.02, 6.64 [10:00:34] Not sure why config sync was still going off [10:00:42] .GH Miraheze/puppet [10:00:42] https://github.com/Miraheze/puppet [10:00:59] Someone needs to merge https://github.com/miraheze/puppet/pull/1856 [10:01:00] [url] mediawiki: don't auto sync config by RhinosF1 · Pull Request #1856 · miraheze/puppet · GitHub | github.com [10:06:06] [02puppet] 07JohnFLewis commented on pull request 03#1696: Add some logging to mirahezerenewssl - 13https://git.io/JRwWX [10:06:14] [02puppet] 07JohnFLewis assigned pull request 03#1696: Add some logging to mirahezerenewssl - 13https://git.io/JqYCL [10:08:03] [02puppet] 07JohnFLewis assigned pull request 03#1745: CSP: add iwiki.eu.org mirrors - 13https://git.io/JOSGq [10:08:59] [02puppet] 07JohnFLewis commented on pull request 03#1745: CSP: add iwiki.eu.org mirrors - 13https://git.io/JRwl9 [10:11:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.30, 6.23, 6.77 [10:15:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.56, 7.52, 7.10 [10:17:02] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.27, 7.40, 7.11 [10:18:14] [02puppet] 07JohnFLewis reviewed pull request 03#1789 commit - 13https://git.io/JRwBa [10:18:15] [02puppet] 07JohnFLewis reviewed pull request 03#1789 commit - 13https://git.io/JRwBV [10:18:56] [02puppet] 07JohnFLewis assigned pull request 03#1794: demo for more complex redirects - 13https://git.io/Jnp0y [10:19:13] [02puppet] 07JohnFLewis commented on pull request 03#1799: Whitelist *.archive.org - 13https://git.io/JRwBd [10:19:23] [02puppet] 07JohnFLewis assigned pull request 03#1799: Whitelist *.archive.org - 13https://git.io/Jctix [10:20:09] [02puppet] 07JohnFLewis reviewed pull request 03#1810 commit - 13https://git.io/JRwRY [10:21:41] [02puppet] 07JohnFLewis closed pull request 03#1827: prep for capacity upgrade - 13https://git.io/JBuqW [10:21:42] [02puppet] 07JohnFLewis commented on pull request 03#1827: prep for capacity upgrade - 13https://git.io/JRwRM [10:22:21] [02puppet] 07JohnFLewis assigned pull request 03#1844: mediawiki: [DNM] Only clone mediawiki to staging servers - 13https://git.io/JB556 [10:22:31] [02puppet] 07JohnFLewis commented on pull request 03#1844: mediawiki: [DNM] Only clone mediawiki to staging servers - 13https://git.io/JRwRA [10:24:30] [02puppet] 07JohnFLewis commented on pull request 03#1854: Remove $wgReCaptchaSiteKey from PrivateSettings - 13https://git.io/JRw0P [10:24:40] [02puppet] 07JohnFLewis assigned pull request 03#1854: Remove $wgReCaptchaSiteKey from PrivateSettings - 13https://git.io/JRs4u [10:25:33] paladox: when you're around could you try to renew https://phabricator.miraheze.org/T7760 ? I've tried several times but kept getting Detail: DNS problem: NXDOMAIN looking up TXT for [10:25:34] [url] ⚓ T7760 wildcard.bots.miraheze.wiki expires soon | phabricator.miraheze.org [10:26:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.86, 6.13, 6.65 [10:27:35] [02puppet] 07JohnFLewis closed pull request 03#1855: Use --quiet for LocalisationUpdate cron - 13https://git.io/JRGWx [10:27:37] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRwE9 [10:27:38] [02miraheze/puppet] 07Universal-Omega 030c7d3d4 - Use --quiet for LocalisationUpdate cron (#1855) [10:28:10] [02puppet] 07JohnFLewis closed pull request 03#1856: mediawiki: don't auto sync config - 13https://git.io/JRCyu [10:28:12] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRwEx [10:28:13] [02miraheze/puppet] 07RhinosF1 0374ad560 - mediawiki: don't auto sync config (#1856) [10:30:27] Ty JohnLewis [10:31:03] You're welcome. think I added some comments to a PR of yours in puppet as well [10:32:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 8.00, 6.68, 6.28 [10:32:26] [02puppet] 07RhinosF1 synchronize pull request 03#1810: varnish: force sslrequest. to jbr3 - 13https://git.io/Jcglf [10:32:44] JohnLewis: ^ is one of them [10:32:57] also on the return line too :P [10:33:00] The DNM one you assigned to me [10:33:41] [02puppet] 07RhinosF1 synchronize pull request 03#1810: varnish: force sslrequest. to jbr3 - 13https://git.io/Jcglf [10:34:17] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 15.15, 9.39, 7.29 [10:34:28] Main blocker for deploy system is opening up an ssh account with the ability to SCP files for MediaWiki [10:34:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.79, 7.98, 7.18 [10:35:08] Auto config deploys failed because www-data can't sudo as itself [10:35:26] [02puppet] 07JohnFLewis closed pull request 03#1810: varnish: force sslrequest. to jbr3 - 13https://git.io/Jcglf [10:35:28] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRwzd [10:35:29] [02miraheze/puppet] 07RhinosF1 0362f41ea - varnish: force sslrequest. to jbr3 (#1810) [10:35:55] "can't sudo as itself"? [10:36:43] JohnLewis: the script runs various commands with sudo -u www-data but with auto deploy which you merged my disable it's running as www-data abyway [10:36:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.59, 7.54, 7.12 [10:37:16] So the problem is simply sudo -u www-data is disallowed by the www-data user? [10:38:11] Yes [10:40:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.89, 7.25, 7.15 [10:40:40] Shouldn't be too hard to fix such a quirky problem [10:40:55] Easiest would be to simply allow it [10:41:06] Other idea was create a deploy user but that's more funky [10:42:18] Changing to it to allow it would be fairy simply I think using user in puppet. Would also be nice to ensure uid matches up everywhere because file copying between appservers when we start with the actual syncing to other servers [10:42:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.94, 6.46, 6.76 [10:50:17] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.74, 8.79, 7.63 [10:51:13] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRwwt [10:51:15] [02miraheze/puppet] 07JohnFLewis 037762702 - allow www-data to sudo as itself [10:51:27] RhinosF1: ^ [10:52:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.70, 6.85, 6.75 [10:53:46] JohnLewis: https://git.io/JRwEx can be reverted then [10:54:02] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03revert-1856-patch-65 [+0/-0/±1] 13https://git.io/JRwrv [10:54:03] [02miraheze/puppet] 07JohnFLewis 039e068df - Revert "mediawiki: don't auto sync config (#1856)" [10:54:05] [02puppet] 07JohnFLewis created branch 03revert-1856-patch-65 - 13https://git.io/vbiAS [10:54:07] [02puppet] 07JohnFLewis opened pull request 03#1860: Revert "mediawiki: don't auto sync config" - 13https://git.io/JRwrf [10:54:08] [02puppet] 07RhinosF1 opened pull request 03#1859: Revert "mediawiki: don't auto sync config" - 13https://git.io/JRwrJ [10:54:11] [02puppet] 07JohnFLewis closed pull request 03#1860: Revert "mediawiki: don't auto sync config" - 13https://git.io/JRwrf [10:54:13] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRwrk [10:54:14] [02miraheze/puppet] 07JohnFLewis 034f38274 - Revert "mediawiki: don't auto sync config (#1856)" (#1860) [10:54:16] [02puppet] 07JohnFLewis deleted branch 03revert-1856-patch-65 - 13https://git.io/vbiAS [10:54:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.63, 7.45, 7.38 [10:54:17] [02miraheze/puppet] 07JohnFLewis deleted branch 03revert-1856-patch-65 [10:54:21] Done [10:54:27] [02puppet] 07RhinosF1 commented on pull request 03#1859: Revert "mediawiki: don't auto sync config" - 13https://git.io/JRwrm [10:54:28] [02puppet] 07RhinosF1 closed pull request 03#1859: Revert "mediawiki: don't auto sync config" - 13https://git.io/JRwrJ [10:54:39] Now config should auto deploy on test3! [10:54:43] Using the new system [10:55:00] Now we just need to look into opening up rsync from 1 mw* to the rest [10:56:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.86, 6.77, 6.78 [11:04:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.12, 8.24, 7.70 [11:06:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.04, 7.34, 7.43 [11:09:11] JohnLewis: can you look into https://phabricator.miraheze.org/T7724 [11:09:12] [url] ⚓ T7724 Setup SSH key for remote access between mw11 and mw* | phabricator.miraheze.org [11:09:31] I tried some tests a week ago but ssh agent was just moaning socket didn't exist [11:10:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.66, 5.79, 6.77 [11:18:11] How was mw11 chosen? [11:25:02] JohnLewis: the one that complains least about load [11:25:38] According to our memory of icinga-miraheze in here [11:25:48] Would mw11 still remain 'disposable' though? [11:27:30] What do you mean by disposable JohnLewis [11:32:15] It's all gonna be in puppet so we're talking minutes to fail it over [11:32:36] But short downtime's would be fine [11:32:56] 1) switch canary and staging to true on the new server [11:33:16] 2) activate ssh agent to store the key [11:33:47] 3) switch the ssh key on the other mw*'s over (if needed, not if we keep the private key in puppet) [11:34:19] 1 & 3 would be just values in puppet [11:34:33] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.42, 8.20, 6.95 [11:34:49] We're talking at most minutes [11:35:20] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.88, 7.52, 6.58 [11:36:30] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.54, 7.50, 6.84 [11:37:18] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.67, 6.51, 6.33 [11:40:23] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.53, 6.72, 6.66 [12:03:55] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.99, 7.39, 6.87 [12:04:08] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.92, 8.85, 7.07 [12:06:17] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.34, 20.45, 16.83 [12:08:17] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.52, 18.57, 16.58 [12:30:29] JohnLewis: for low workers, grafana has a concept of Red / amber / green now [12:30:35] We just need icinga to alert on it [12:31:04] That concept has always been there to my knowledge [12:31:48] Probably, it's not new [12:32:05] We had some incidents where workers were low and it seemed a good idea to actually alert [12:39:46] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.39, 6.92, 6.39 [12:41:38] It makes perfect sense to alert of resource starvation [12:41:43] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.16, 6.71, 6.38 [12:45:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.65, 6.89, 6.55 [12:46:10] PROBLEM - wiki.yumeka.icu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.yumeka.icu' expires in 15 day(s) (Thu 26 Aug 2021 12:43:55 GMT +0000). [12:46:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRrJf [12:46:36] [02miraheze/ssl] 07MirahezeSSLBot 0383a9a0b - Bot: Update SSL cert for wiki.yumeka.icu [12:47:43] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.73, 6.13, 6.31 [12:53:48] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.17, 6.33, 6.28 [12:54:26] PROBLEM - civwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'civwiki.org' expires in 15 day(s) (Thu 26 Aug 2021 12:49:23 GMT +0000). [12:55:44] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.09, 7.10, 6.58 [12:57:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.43, 6.59, 6.46 [12:59:00] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRrTg [12:59:01] [02miraheze/ssl] 07MirahezeSSLBot 03fcb4f42 - Bot: Update SSL cert for civwiki.org [13:00:10] RECOVERY - wiki.yumeka.icu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yumeka.icu' will expire on Mon 08 Nov 2021 11:46:22 GMT +0000. [13:06:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.73, 20.21, 17.32 [13:08:23] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.48, 18.08, 16.90 [13:17:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.18, 6.59, 7.95 [13:26:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.39, 7.25, 7.95 [13:28:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.76, 7.91, 8.11 [13:29:19] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.18, 7.14, 7.41 [13:33:15] PROBLEM - mw8 APT on mw8 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [13:34:53] RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Mon 08 Nov 2021 11:58:53 GMT +0000. [13:35:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.48, 8.00, 7.74 [13:35:15] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [13:44:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.25, 7.07, 7.80 [13:46:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.08, 7.29, 7.78 [13:48:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.83, 7.39, 7.74 [13:49:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.42, 7.17, 7.27 [13:51:46] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.55, 6.39, 5.98 [13:52:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.93, 8.13, 7.96 [13:52:58] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.99, 6.36, 5.84 [13:53:47] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.65, 6.02, 5.89 [13:54:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.14, 7.77, 7.85 [13:54:54] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.60, 6.14, 5.82 [13:55:03] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.35, 6.94, 7.29 [14:02:57] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.94, 7.08, 7.11 [14:04:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.91, 8.14, 7.64 [14:06:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.63, 7.66, 7.42 [14:08:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.32, 7.38, 7.48 [14:14:46] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.19, 6.28, 6.79 [14:14:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.30, 7.80, 7.62 [14:18:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.81, 7.96, 7.73 [14:20:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.47, 7.09, 7.03 [14:20:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.00, 8.27, 7.86 [14:20:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.84, 6.48, 6.10 [14:22:37] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.18, 7.51, 7.16 [14:22:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.34, 7.49, 7.63 [14:22:53] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.38, 6.02, 5.98 [14:24:35] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.58, 7.33, 7.13 [14:24:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.73, 7.72, 7.68 [14:26:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.23, 7.61, 7.65 [14:32:53]