[00:02:52] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.03, 16.91, 14.30 [00:05:24] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.10, 20.00, 15.73 [00:05:37] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.01, 5.52, 4.29 [00:07:37] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.21, 4.65, 4.12 [00:07:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.84, 3.79, 3.28 [00:09:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.80, 20.99, 17.14 [00:11:20] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.18, 23.31, 18.43 [00:11:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.91, 3.37, 3.22 [00:12:45] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.49, 6.72, 4.35 [00:16:41] SRE, from `metawiki`: [00:16:41] ```Error 503 Backend fetch failed, forwarded for , 127.0.0.1 [00:16:41] (Varnish XID 1054183329) via cp15.miraheze.org at Sun, 10 Oct 2021 00:16:01 GMT. [00:16:41] ``` [00:17:30] Seems to be resolved. [00:27:03] RECOVERY - bacula2 Bacula Databases db12 on bacula2 is OK: OK: Full, 344679 files, 80.20GB, 2021-10-10 00:26:00 (1.0 minutes ago) [00:29:14] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.11, 23.26, 23.91 [00:30:45] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 1.94, 4.74, 5.64 [00:30:49] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [00:34:45] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.06, 3.33, 4.85 [00:41:00] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.32, 17.79, 20.29 [00:52:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.49, 3.17 [00:54:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.89, 3.33, 3.15 [01:07:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.40, 3.30 [01:09:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.73, 3.43 [01:13:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.61, 3.45 [01:15:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.68, 3.33, 3.37 [01:32:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.50, 3.40 [01:36:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.05, 3.01, 3.25 [02:06:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.61, 3.34 [02:08:15] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.93, 3.35, 3.27 [02:23:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.59, 3.39 [02:29:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.02, 3.39, 3.36 [02:37:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.51, 3.38 [02:39:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.61, 3.43 [02:41:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.48, 3.19, 3.30 [02:56:19] RECOVERY - bacula2 Bacula Databases db13 on bacula2 is OK: OK: Full, 341882 files, 96.13GB, 2021-10-10 02:54:00 (2.3 minutes ago) [02:57:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.49, 3.31 [02:59:01] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.32, 3.14, 3.21 [03:04:59] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 6271 files, 48.98MB, 2021-10-10 03:03:00 (2.0 minutes ago) [03:20:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.80, 3.81, 3.49 [03:22:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.77, 3.39, 3.37 [03:30:16] RECOVERY - bacula2 Bacula Phabricator Static on bacula2 is OK: OK: Full, 88188 files, 9.442GB, 2021-10-10 03:28:00 (2.3 minutes ago) [04:05:41] PROBLEM - mw8 APT on mw8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:06:14] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.86, 24.04, 18.15 [04:06:47] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.54, 8.70, 5.63 [04:06:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.58, 10.18, 6.35 [04:07:12] PROBLEM - mw9 APT on mw9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:07:49] RECOVERY - mw8 APT on mw8 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [04:08:10] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.16, 22.78, 18.44 [04:09:12] RECOVERY - mw9 APT on mw9 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [04:10:11] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.15, 19.33, 17.69 [04:10:44] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 2.64, 6.20, 5.41 [04:10:50] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.04, 7.71, 6.32 [04:12:46] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 2.68, 6.04, 5.87 [05:20:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.58, 3.22, 2.91 [05:24:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 2.98, 2.90 [06:00:18] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.42, 8.18, 5.61 [06:00:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.67, 8.53, 5.87 [06:01:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 6.00, 8.64, 6.14 [06:02:16] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.20, 6.73, 5.39 [06:02:16] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.67, 7.05, 5.63 [06:03:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.75, 7.27, 5.92 [06:04:12] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.31, 6.24, 5.51 [06:07:33] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.46, 6.36, 5.85 [06:18:50] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.62, 7.36, 6.54 [06:21:37] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.15, 2.81 [06:23:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.04, 3.08, 2.82 [06:26:32] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.74, 8.12, 7.19 [06:28:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.87, 7.69, 7.14 [06:32:20] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.53, 7.67, 7.21 [06:34:18] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.84, 7.64, 7.25 [06:38:09] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.60, 5.55, 6.49 [07:14:28] [02puppet] 07Universal-Omega opened pull request 03#2021: mwtask1: set mediawiki::jobqueue::runner::cron - 13https://git.io/JoEph [07:14:47] [02puppet] 07Universal-Omega edited pull request 03#2021: mwtask1: set `mediawiki::jobqueue::runner::cron` - 13https://git.io/JoEph [07:15:18] !log run some scripts on all wikis from mwtask1, logs in my homedir [07:15:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:16:07] dmehus: ty for report [07:16:17] @Owen check task I added you too [07:17:19] [02puppet] 07Reception123 closed pull request 03#2021: mwtask1: set `mediawiki::jobqueue::runner::cron` - 13https://git.io/JoEph [07:17:20] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jouv8 [07:17:22] [02miraheze/puppet] 07Universal-Omega 039c72d25 - mwtask1: set mediawiki::jobqueue::runner::cron (#2021) [07:22:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.67, 3.14 [07:24:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.18, 3.53, 3.16 [07:28:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.67, 3.30 [07:30:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.97, 3.49, 3.29 [07:32:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.01, 3.08, 3.17 [07:33:16] [02puppet] 07RhinosF1 opened pull request 03#2022: runner: fix AF scriptt - 13https://git.io/Jou4s [07:34:35] Reception123: ^ merge please [07:35:28] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jou04 [07:35:30] [02miraheze/puppet] 07RhinosF1 03ddc49f1 - runner: fix AF scriptt (#2022) [07:35:31] [02puppet] 07Reception123 closed pull request 03#2022: runner: fix AF scriptt - 13https://git.io/Jou4s [07:36:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.69, 3.41 [07:36:53] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 12.09, 9.77, 6.86 [07:37:01] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.45, 21.27, 18.92 [07:38:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.60, 3.26, 3.28 [07:39:00] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.37, 19.72, 18.64 [07:40:49] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.24, 6.72, 6.28 [08:06:37] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.50, 7.60, 5.46 [08:06:57] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.74, 19.70, 16.51 [08:08:36] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.64, 6.10, 5.17 [08:08:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.17, 17.88, 16.21 [08:35:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.46, 5.67, 4.67 [08:35:45] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.67, 20.82, 17.01 [08:37:23] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.11, 5.45, 4.71 [08:37:41] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.40, 18.07, 16.46 [08:39:20] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.49, 4.34, 4.39 [09:37:04] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.56, 7.64, 5.23 [09:39:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.26, 5.87, 4.87 [10:04:11] PROBLEM - wiki.caker18.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:04:12] PROBLEM - publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:04:25] PROBLEM - index.pdcommunity.ir - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:04:26] PROBLEM - franchise.franchising.org.ua - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:04:27] PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:04:40] PROBLEM - www.opendatascot.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:04:40] PROBLEM - wiki.thunis.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:04:55] PROBLEM - warmage.necrocity.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:04:57] PROBLEM - data.nonbinary.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:00] PROBLEM - wiki.warframestat.us - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:14] PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:15] PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [10:05:18] PROBLEM - wiki.conworlds.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:24] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.38.69.175/cpweb, 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [10:05:36] Hmm poo [10:06:28] PROBLEM - wiki.jumpbound.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:06:32] PROBLEM - idleon.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:06:34] PROBLEM - wiki.raghuveer.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:06:36] PROBLEM - www.thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:06:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.38.69.175/cpweb, 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [10:06:57] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:07:33] PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:07:42] .op [10:07:42] Attempting to OP... [10:08:45] PROBLEM - wiki.ssij.se - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:59] PROBLEM - cp12 HTTPS on cp12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:38] PROBLEM - cp13 HTTPS on cp13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:10:09] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:11:39] PROBLEM - wiki.caker18.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.caker18.com' expires in 12 day(s) (Sat 23 Oct 2021 07:54:49 GMT +0000). [10:11:41] PROBLEM - publictestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'publictestwiki.com' expires in 9 day(s) (Wed 20 Oct 2021 05:07:03 GMT +0000). [10:11:44] RECOVERY - cp12 HTTPS on cp12 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2989 bytes in 0.327 second response time [10:11:52] PROBLEM - index.pdcommunity.ir - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'index.pdcommunity.ir' expires in 9 day(s) (Wed 20 Oct 2021 00:55:24 GMT +0000). [10:11:53] PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.teessidehackspace.org.uk' expires in 9 day(s) (Wed 20 Oct 2021 04:45:23 GMT +0000). [10:11:57] PROBLEM - franchise.franchising.org.ua - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'franchise.franchising.org.ua' expires in 9 day(s) (Wed 20 Oct 2021 06:33:38 GMT +0000). [10:12:00] PROBLEM - www.opendatascot.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'opendatascot.org' expires in 9 day(s) (Wed 20 Oct 2021 07:37:37 GMT +0000). [10:12:00] PROBLEM - warmage.necrocity.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'warmage.necrocity.co.uk' expires in 9 day(s) (Wed 20 Oct 2021 03:15:07 GMT +0000). [10:12:01] PROBLEM - wiki.thunis.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.thunis.eu' expires in 10 day(s) (Wed 20 Oct 2021 10:34:06 GMT +0000). [10:12:02] PROBLEM - data.nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'data.nonbinary.wiki' expires in 9 day(s) (Wed 20 Oct 2021 03:02:44 GMT +0000). [10:12:02] RECOVERY - cp13 HTTPS on cp13 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2989 bytes in 3.670 second response time [10:12:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:12:03] PROBLEM - wiki.warframestat.us - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.warframestat.us' expires in 9 day(s) (Wed 20 Oct 2021 07:19:36 GMT +0000). [10:12:22] PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonciclopedia.org' expires in 9 day(s) (Wed 20 Oct 2021 07:01:22 GMT +0000). [10:12:32] PROBLEM - wiki.conworlds.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.conworlds.org' expires in 9 day(s) (Wed 20 Oct 2021 00:53:44 GMT +0000). [10:13:22] PROBLEM - wiki.jumpbound.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.jumpbound.com' expires in 13 day(s) (Sun 24 Oct 2021 03:49:03 GMT +0000). [10:13:25] PROBLEM - idleon.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'idleon.info' expires in 9 day(s) (Wed 20 Oct 2021 09:27:12 GMT +0000). [10:13:26] PROBLEM - wiki.raghuveer.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.raghuveer.net' expires in 9 day(s) (Wed 20 Oct 2021 05:56:29 GMT +0000). [10:13:35] PROBLEM - www.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 10 day(s) (Wed 20 Oct 2021 18:16:08 GMT +0000). [10:13:38] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 9 day(s) (Tue 19 Oct 2021 22:59:59 GMT +0000). [10:14:14] PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.villagecollaborative.net' expires in 9 day(s) (Wed 20 Oct 2021 06:50:42 GMT +0000). [10:14:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:15:28] PROBLEM - wiki.ssij.se - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ssij.se' expires in 9 day(s) (Wed 20 Oct 2021 09:50:55 GMT +0000). [10:17:06] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 10 day(s) (Wed 20 Oct 2021 18:16:08 GMT +0000). [10:19:13] PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.grottocenter.org' expires in 9 day(s) (Wed 20 Oct 2021 00:09:24 GMT +0000). [10:33:44] PROBLEM - mw9 APT on mw9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:35:48] RECOVERY - mw9 APT on mw9 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [10:35:58] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.44, 8.64, 5.55 [10:36:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.89, 20.74, 17.11 [10:37:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.61, 7.09, 5.36 [10:38:38] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.43, 18.70, 16.80 [10:40:00] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.57, 5.77, 5.08 [11:06:39] PROBLEM - mw11 APT on mw11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:06:43] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.76, 8.00, 5.39 [11:08:37] RECOVERY - mw11 APT on mw11 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [11:08:42] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.60, 6.72, 5.24 [11:35:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.49, 19.31, 16.88 [11:37:47] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.67, 17.66, 16.57 [11:41:29] .op [11:41:30] Attempting to OP... [12:07:10] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.63, 19.44, 16.21 [12:08:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.90, 6.82, 5.29 [12:09:08] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.19, 18.77, 16.36 [12:10:31] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.85, 6.42, 5.32 [12:22:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.08, 5.52, 5.03 [12:24:32] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.34, 6.76, 5.54 [12:26:30] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.28, 6.27, 5.52 [12:36:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.37, 7.07, 4.97 [12:37:01] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.37, 22.31, 18.25 [12:38:38] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.45, 6.82, 5.14 [12:38:44] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.26, 5.51, 4.80 [12:38:57] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.99, 20.55, 18.12 [12:40:37] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.38, 6.01, 5.05 [12:40:46] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.14, 5.79, 4.96 [12:40:55] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.20, 19.09, 17.87 [12:42:45] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.81, 5.46, 4.95 [12:44:44] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.27, 4.87, 4.80 [13:36:45] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 17.44, 9.98, 6.00 [13:37:10] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.67, 21.37, 17.47 [13:39:10] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.70, 20.40, 17.61 [13:40:42] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.68, 7.44, 5.96 [13:42:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.60, 6.45, 5.77 [13:49:26] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 16 minutes ago with 0 failures [13:50:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.49, 5.44, 4.86 [13:52:46] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.00, 4.98, 4.77 [13:58:19] [02MirahezeDebug] 07paladox closed pull request 03#5: Cleanup - 13https://git.io/JwW8Z [13:58:20] [02miraheze/MirahezeDebug] 07paladox pushed 037 commits to 03master [+0/-0/±8] 13https://git.io/JooHD [13:58:22] [02miraheze/MirahezeDebug] 07paladox 032d158d7 - Merge pull request #5 from miraheze/Universal-Omega-patch-1 [13:58:23] [02MirahezeDebug] 07paladox deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fpk7C [13:58:25] [02miraheze/MirahezeDebug] 07paladox deleted branch 03Universal-Omega-patch-1 [14:06:35] PROBLEM - mw9 APT on mw9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:06:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.52, 9.04, 5.58 [14:07:14] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.46, 20.02, 17.74 [14:08:41] RECOVERY - mw9 APT on mw9 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [14:09:17] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.38, 19.48, 17.80 [14:10:37] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.01, 6.98, 5.58 [14:11:40] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.77, 5.63, 5.03 [14:12:33] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.17, 6.08, 5.42 [14:13:37] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.71, 5.89, 5.18 [14:17:30] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.31, 5.83, 5.37 [14:21:26] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.94, 6.51, 5.68 [14:22:11] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.79, 23.57, 20.02 [14:23:23] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.07, 5.85, 5.53 [14:24:07] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.77, 21.99, 19.87 [14:25:19] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.64, 5.84, 5.55 [14:27:14] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.45, 5.38, 5.42 [14:28:02] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.76, 20.03, 19.61 [14:31:13] @Owen: check phab please [14:36:17] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.96, 22.21, 20.54 [14:36:53] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.94, 7.31, 5.75 [14:37:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.35, 7.76, 6.14 [14:38:14] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.56, 20.31, 20.06 [14:38:48] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.42, 6.23, 5.54 [14:41:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.86, 5.97, 5.49 [14:41:16] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.22, 6.59, 5.99 [14:41:49] @RhinosF1 responded [14:42:51] Ty [14:44:57] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.95, 5.12, 5.28 [14:48:50] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.49, 5.24, 5.24 [14:50:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.28, 5.33, 5.29 [14:54:44] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.01, 5.51, 5.33 [14:56:43] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.45, 5.22, 5.25 [14:58:45] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.56, 5.60, 5.37 [15:00:46] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.85, 5.13, 5.22 [15:03:23] @Owen: can we use yours for now? [15:03:33] I guess [15:04:01] @Owen can you remind me of it over email [15:04:33] Done [15:04:50] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.58, 4.76, 5.05 [15:05:54] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.32, 6.23, 4.93 [15:07:05] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.38, 24.08, 20.45 [15:08:01] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.89, 5.74, 4.92 [15:08:44] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.67, 8.07, 5.99 [15:09:28] PROBLEM - mw12 APT on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:10:50] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.74, 5.95, 5.48 [15:11:30] RECOVERY - mw12 APT on mw12 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [15:12:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.46, 3.96, 3.26 [15:12:37] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.49, 6.92, 6.07 [15:13:00] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.48, 23.39, 21.62 [15:14:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.74, 3.27 [15:14:36] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.83, 5.90, 5.80 [15:16:16] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.45, 3.37, 3.20 [15:16:44] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.10, 4.42, 4.95 [15:20:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.31, 6.14, 5.56 [15:20:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.13, 19.92, 20.33 [15:22:44] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.37, 5.61, 5.44 [15:25:16] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 15.83, 10.40, 7.40 [15:25:18] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.74, 4.33, 4.97 [15:25:47] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.13, 22.16, 21.20 [15:26:30] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.00, 9.81, 6.95 [15:27:30] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 7.33, 9.20, 6.84 [15:27:59] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.40, 7.14, 5.81 [15:28:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.64, 3.32 [15:29:15] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.35, 7.14, 6.74 [15:29:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 3.72, 7.27, 6.43 [15:29:50] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.33, 19.55, 20.25 [15:30:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.61, 3.35 [15:31:27] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.14, 7.59, 6.95 [15:31:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.07, 6.02, 6.08 [15:32:22] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.83, 8.91, 6.76 [15:33:11] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.01, 20.68, 18.44 [15:33:33] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.95, 7.31, 6.93 [15:34:27] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.80, 10.75, 7.03 [15:35:05] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 2.79, 6.69, 6.81 [15:35:23] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.11, 22.63, 19.44 [15:35:43] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.03, 6.41, 6.64 [15:36:44] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.16, 3.32, 3.31 [15:37:08] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.53, 5.70, 6.42 [15:37:11] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.75, 7.26, 5.50 [15:37:26] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.69, 22.23, 19.69 [15:38:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.43, 7.20, 6.79 [15:39:10] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.88, 6.02, 5.26 [15:39:27] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 13.98, 19.22, 18.91 [15:40:25] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.77, 6.66, 6.47 [15:40:43] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.73, 6.60, 6.61 [15:41:05] @Owen: NCSC advise Action Fraud or local police reports [15:42:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.94, 3.75, 3.45 [15:43:34] Action Fraud seems out of scope, local police won’t do anything [15:44:17] @Owen AF cover all cyber crime [15:44:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.25, 3.30, 3.33 [15:45:04] The DWO has simply said it's of a criminal nature [15:45:16] !log mwtask1: /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/cleanupUploadStash.php [15:45:18] I don't expect much action beyond a reference number [15:45:19] cc RhinosF1 Reception123 ^ [15:45:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:45:25] paladox: why [15:45:41] cleanup the temp folders to clean space. [15:46:04] thanks paladox ! [15:46:44] Ok [15:49:58] Failed removing stashed upload with key: 13os2wor2rws.gzlezn.7.png (UploadStashBadPathException) [15:49:58] hmm [15:50:34] If you wish to report to AF, then feel free as long as T&S receives communications + reference number(s) [15:52:11] @Owen: I'm happy to keep you copied on. [15:52:35] Reference number for NCSC is on task but they are more advisory [15:52:41] Sounds good, SRE are likely in a much better position with the information they have to follow up initially at least [15:53:09] @Owen: Can I put the company address for correspondence [15:53:19] I'll probably either do it tommorow or late evening [15:53:20] What’s going on? [15:55:53] @Bukkit: Outage this morning was a DDoS [15:56:24] We're co-ordinating with the National Cyber Security Centre & Action Fraud who are responsible for dealing with Cyber Security Incidents and Cyber Crime [15:57:03] Oh, okay. I was quite confused about what you were talking about with Owen. [15:57:36] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.69, 5.35, 4.74 [15:58:58] @Bukkit: it can be [15:59:18] 850 request per second, sheesh. [15:59:33] oh wow [16:04:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 3.51, 3.36 [16:07:07] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 12.54, 9.74, 6.82 [16:07:13] PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:07:23] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.01, 20.47, 18.36 [16:08:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.75, 3.48 [16:09:14] RECOVERY - mw10 APT on mw10 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [16:09:19] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.57, 6.04, 5.27 [16:09:24] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.74, 19.48, 18.26 [16:10:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.66, 3.32, 3.36 [16:11:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.99, 7.48, 6.55 [16:11:16] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.54, 5.79, 5.28 [16:13:11] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.53, 6.06, 6.14 [16:15:13] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.13, 5.09, 5.10 [16:20:05] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.45, 20.55, 18.66 [16:20:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.26, 3.25 [16:21:05] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.08, 6.75, 5.74 [16:22:02] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.79, 20.57, 18.88 [16:22:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.18, 3.22, 3.23 [16:23:58] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.69, 19.39, 18.65 [16:24:31] RECOVERY - gluster4 Disk Space on gluster4 is OK: DISK OK - free space: / 117879 MB (11% inode=80%); [16:36:06] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.30, 22.48, 19.92 [16:37:19] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.50, 7.36, 6.00 [16:38:04] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.95, 20.63, 19.55 [16:39:20] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.50, 6.26, 5.76 [16:40:02] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.73, 19.79, 19.38 [16:45:02] @Owen: AF report NFRC211004784837 refers, I will email a pdf of the report to you later [16:49:24] They say 28 days for a response [16:50:56] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.90, 5.84, 4.56 [16:51:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.10, 20.54, 19.88 [16:51:42] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.75, 7.60, 6.38 [16:52:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.55, 3.84, 3.40 [16:53:25] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.99, 20.26, 19.86 [16:54:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.78, 3.42 [16:54:49] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.46, 6.22, 5.05 [16:54:53] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.11, 5.01, 5.84 [16:55:38] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.18, 7.81, 6.74 [16:56:14] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.89, 3.50 [16:56:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.73, 7.08, 5.78 [16:57:48] !log root@gluster3:/home/paladox# gluster volume set static performance.cache-refresh-timeout 10 [16:57:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:58:01] !log root@gluster3:/home/paladox# gluster volume set static performance.cache-size 1GB [16:58:15] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.38, 6.68, 5.78 [16:58:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.66, 3.47 [16:58:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:58:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.89, 5.40, 5.77 [17:00:44] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.15, 4.93, 5.55 [17:01:32] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.67, 6.55, 6.52 [17:02:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.87, 3.58 [17:04:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.71, 3.57 [17:05:32] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.55, 8.67, 7.49 [17:06:41] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.67, 7.50, 6.09 [17:06:50] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.06, 5.88, 5.67 [17:07:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.45, 7.94, 7.37 [17:08:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.79, 3.62 [17:08:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.11, 6.97, 5.98 [17:08:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.94, 22.80, 20.66 [17:08:45] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.16, 5.69, 5.64 [17:10:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.34, 3.24, 3.43 [17:10:28] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.08, 6.30, 5.85 [17:10:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.84, 22.20, 20.73 [17:12:40] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.34, 7.51, 6.69 [17:13:07] RECOVERY - gluster3 Gluster Disk Space on gluster3 is OK: DISK OK - free space: /mnt/mediawiki-static 237852 MB (11% inode=80%); [17:13:30] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.63, 5.83, 6.62 [17:14:22] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.46, 3.46 [17:14:44] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.60, 5.71, 5.59 [17:16:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.47, 3.46 [17:16:39] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.77, 6.66, 6.53 [17:16:39]