[00:03:28] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.98, 4.06, 3.20 [00:07:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.84, 19.64, 16.38 [00:08:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.99, 3.35 [00:11:19] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.38, 21.19, 17.62 [00:13:51] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.16, 5.68, 4.12 [00:14:10] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.51, 2.73, 1.90 [00:16:10] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 1.70, 2.38, 1.88 [00:16:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.62, 3.19, 3.27 [00:17:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48953 MB (10% inode=98%); [00:22:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.51, 3.40 [00:24:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.12, 3.08, 3.26 [00:27:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.38, 5.71, 5.31 [00:31:28] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:31:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.29, 4.31, 4.86 [00:32:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.48, 21.87, 22.61 [00:36:38] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.19, 25.37, 23.63 [00:37:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.88, 5.14, 5.06 [00:38:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 13.49, 20.41, 22.02 [00:39:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.90, 4.38, 4.79 [00:44:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.58, 3.38 [00:44:38] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.46, 16.73, 19.79 [00:46:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.94, 3.40, 3.34 [00:57:03] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49806 MB (11% inode=98%); [01:02:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.40, 3.24 [01:04:50] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.22, 3.32, 3.23 [01:10:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.46, 3.29 [01:12:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 3.10, 3.18 [01:13:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.47, 5.99, 4.67 [01:14:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:15:32] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.48, 5.72, 4.73 [01:18:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.86, 3.48 [01:24:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.17, 3.55, 3.47 [01:24:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:28:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.26, 3.15, 3.34 [01:50:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.30, 3.13 [01:52:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.58, 3.25 [01:56:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.81, 3.42 [01:58:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 3.90, 3.49 [02:00:30] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.22, 3.23, 3.29 [02:24:47] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.62, 3.27 [02:26:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.22, 3.44, 3.25 [02:30:34] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.46, 3.23, 3.23 [02:52:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.74, 3.39 [02:56:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.50, 3.35, 3.33 [03:21:42] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.53, 3.69, 3.29 [03:23:38] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.88, 3.02, 3.09 [04:15:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48953 MB (10% inode=98%); [04:40:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.24, 2.91 [04:42:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.31, 2.82, 2.79 [04:49:03] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49762 MB (11% inode=98%); [04:53:47] PROBLEM - scpvncommunitywiki.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'scpvncommunitywiki.xyz' expires in 15 day(s) (Mon 10 Jan 2022 04:48:53 GMT +0000). [04:56:45] PROBLEM - history.sdtef.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'history.sdtef.org' expires in 15 day(s) (Mon 10 Jan 2022 04:52:34 GMT +0000). [05:02:49] PROBLEM - guia.esporo.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guia.esporo.net' expires in 15 day(s) (Mon 10 Jan 2022 04:56:14 GMT +0000). [05:03:42] PROBLEM - guia.cineastas.pt - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guia.cineastas.pt' expires in 15 day(s) (Mon 10 Jan 2022 04:59:32 GMT +0000). [05:06:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.29, 2.82, 1.64 [05:08:06] PROBLEM - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'encyclopediaofastrobiology.org' expires in 15 day(s) (Mon 10 Jan 2022 05:04:27 GMT +0000). [05:10:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.61, 1.69, 1.45 [05:22:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.45, 3.11 [05:24:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.34, 3.35, 3.11 [06:02:54] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'publictestwiki.com' expires in 15 day(s) (Mon 10 Jan 2022 05:57:56 GMT +0000). [06:03:35] PROBLEM - franchise.franchising.org.ua - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'franchise.franchising.org.ua' expires in 15 day(s) (Mon 10 Jan 2022 06:01:27 GMT +0000). [06:04:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 3.47, 3.11 [06:04:44] PROBLEM - index.pdcommunity.ir - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'index.pdcommunity.ir' expires in 15 day(s) (Mon 10 Jan 2022 05:59:14 GMT +0000). [06:05:48] PROBLEM - publictestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'publictestwiki.com' expires in 15 day(s) (Mon 10 Jan 2022 05:57:56 GMT +0000). [06:05:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 3.12, 3.02 [06:06:52] PROBLEM - fc.songcontests.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fc.songcontests.eu' expires in 15 day(s) (Mon 10 Jan 2022 06:02:16 GMT +0000). [06:08:02] PROBLEM - hololive.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hololive.wiki' expires in 15 day(s) (Mon 10 Jan 2022 06:00:29 GMT +0000). [06:08:25] PROBLEM - dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Mon 10 Jan 2022 06:06:14 GMT +0000). [06:09:17] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'enc.for.uz' expires in 15 day(s) (Mon 10 Jan 2022 06:02:55 GMT +0000). [06:09:29] PROBLEM - data.nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'data.nonbinary.wiki' expires in 15 day(s) (Mon 10 Jan 2022 06:05:39 GMT +0000). [06:09:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.43, 3.16 [06:10:03] PROBLEM - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'cyberlaw.ccdcoe.org' expires in 15 day(s) (Mon 10 Jan 2022 06:07:01 GMT +0000). [06:11:41] PROBLEM - electowiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'electowiki.org' expires in 15 day(s) (Mon 10 Jan 2022 06:04:54 GMT +0000). [06:11:44] PROBLEM - archive.stellurgists.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'archive.stellurgists.wiki' expires in 15 day(s) (Mon 10 Jan 2022 06:08:22 GMT +0000). [06:11:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.37, 3.25, 3.11 [06:12:32] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.39, 6.75, 4.64 [06:13:48] PROBLEM - ao90.pinho.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ao90.pinho.org' expires in 15 day(s) (Mon 10 Jan 2022 06:08:59 GMT +0000). [06:13:56] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Mon 10 Jan 2022 06:06:14 GMT +0000). [06:14:09] PROBLEM - www.allthetropes.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'allthetropes.org' expires in 15 day(s) (Mon 10 Jan 2022 06:11:18 GMT +0000). [06:14:32] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.39, 6.71, 4.89 [06:15:17] PROBLEM - beaconspace.unrestrictedlorefare.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'beaconspace.unrestrictedlorefare.com' expires in 15 day(s) (Mon 10 Jan 2022 06:07:39 GMT +0000). [06:15:30] PROBLEM - aman.awiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'aman.awiki.org' expires in 15 day(s) (Mon 10 Jan 2022 06:10:13 GMT +0000). [06:16:10] PROBLEM - allthetropes.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'allthetropes.org' expires in 15 day(s) (Mon 10 Jan 2022 06:11:18 GMT +0000). [06:17:03] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'antiguabarbudacalypso.com' expires in 15 day(s) (Mon 10 Jan 2022 06:09:34 GMT +0000). [06:19:28] PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'adadevelopersacademy.wiki' expires in 15 day(s) (Mon 10 Jan 2022 06:11:54 GMT +0000). [07:12:48] PROBLEM - db13 APT on db13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:14:47] RECOVERY - db13 APT on db13 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [07:16:06] PROBLEM - db13 Current Load on db13 is WARNING: WARNING - load average: 3.60, 7.23, 5.21 [07:18:06] RECOVERY - db13 Current Load on db13 is OK: OK - load average: 1.98, 5.44, 4.80 [07:54:19] PROBLEM - db12 Current Load on db12 is CRITICAL: CRITICAL - load average: 11.71, 11.53, 6.75 [07:58:19] RECOVERY - db12 Current Load on db12 is OK: OK - load average: 1.31, 5.86, 5.49 [12:04:18] Happy Christmas Reception123 [12:27:30] Merry Christmas RhinosF1! [13:01:15] PROBLEM - ldap2 SSH on ldap2 is CRITICAL: Server answer [13:03:15] RECOVERY - ldap2 SSH on ldap2 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [14:39:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48932 MB (10% inode=98%); [14:51:03] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49903 MB (11% inode=98%); [14:59:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.23, 3.48, 3.21 [15:01:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.00, 3.35, 3.20 [15:22:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:27:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:06:36] PROBLEM - archive.a2b2.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'archive.a2b2.org' expires in 15 day(s) (Mon 10 Jan 2022 16:01:51 GMT +0000). [16:10:27] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'speleo.wiki' expires in 15 day(s) (Mon 10 Jan 2022 16:05:11 GMT +0000). [16:10:48] PROBLEM - wiki.danvs.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.danvs.net' expires in 15 day(s) (Mon 10 Jan 2022 16:04:26 GMT +0000). [16:11:03] PROBLEM - wiki.mineland.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mineland.eu' expires in 15 day(s) (Mon 10 Jan 2022 16:06:27 GMT +0000). [16:11:21] PROBLEM - wiki.thunis.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.thunis.eu' expires in 15 day(s) (Mon 10 Jan 2022 16:05:49 GMT +0000). [16:11:21] PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.make717.org' expires in 15 day(s) (Mon 10 Jan 2022 16:07:09 GMT +0000). [16:12:11] PROBLEM - nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonbinary.wiki' expires in 15 day(s) (Mon 10 Jan 2022 16:03:42 GMT +0000). [16:14:48] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'permanentfuturelab.wiki' expires in 15 day(s) (Mon 10 Jan 2022 16:11:54 GMT +0000). [16:15:11] PROBLEM - nenawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nenawiki.org' expires in 15 day(s) (Mon 10 Jan 2022 16:08:25 GMT +0000). [16:15:41] PROBLEM - ndg.nenawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nenawiki.org' expires in 15 day(s) (Mon 10 Jan 2022 16:08:25 GMT +0000). [16:19:17] PROBLEM - mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Mon 10 Jan 2022 16:16:05 GMT +0000). [16:19:45] PROBLEM - wiki.ssij.se - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ssij.se' expires in 15 day(s) (Mon 10 Jan 2022 16:13:21 GMT +0000). [16:20:06] PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'permanentfuturelab.wiki' expires in 15 day(s) (Mon 10 Jan 2022 16:11:54 GMT +0000). [16:20:48] PROBLEM - www.mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Mon 10 Jan 2022 16:16:05 GMT +0000). [16:23:24] PROBLEM - wiki.tulpa.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.tulpa.info' expires in 15 day(s) (Mon 10 Jan 2022 16:16:36 GMT +0000). [16:32:11] PROBLEM - monarchists.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'monarchists.wiki' expires in 15 day(s) (Mon 10 Jan 2022 16:26:15 GMT +0000). [16:32:36] PROBLEM - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ricochetuniverse.com' expires in 15 day(s) (Mon 10 Jan 2022 16:28:54 GMT +0000). [16:34:08] PROBLEM - vitriol.spoon.army - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vitriol.spoon.army' expires in 15 day(s) (Mon 10 Jan 2022 16:27:28 GMT +0000). [16:37:55] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mrlove.wiki' expires in 15 day(s) (Mon 10 Jan 2022 16:32:51 GMT +0000). [16:40:17] PROBLEM - wiki.fraterniterebellion.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fraterniterebellion.com' expires in 15 day(s) (Mon 10 Jan 2022 16:37:25 GMT +0000). [16:40:19] PROBLEM - mediawiki.ecfin.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mediawiki.ecfin.eu' expires in 15 day(s) (Mon 10 Jan 2022 16:31:24 GMT +0000). [16:41:41] PROBLEM - www.fraterniterebellion.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fraterniterebellion.com' expires in 15 day(s) (Mon 10 Jan 2022 16:37:25 GMT +0000). [16:43:53] PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.grottocenter.org' expires in 15 day(s) (Mon 10 Jan 2022 16:41:34 GMT +0000). [16:44:55] PROBLEM - wiki.graalmilitary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.graalmilitary.com' expires in 15 day(s) (Mon 10 Jan 2022 16:39:55 GMT +0000). [16:46:40] PROBLEM - wiki.cdntennis.ca - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cdntennis.ca' expires in 15 day(s) (Mon 10 Jan 2022 16:38:41 GMT +0000). [16:49:02] PROBLEM - miraheze.ml - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.ml' expires in 15 day(s) (Mon 10 Jan 2022 16:42:13 GMT +0000). [16:49:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.35, 3.00 [16:51:22] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.66, 3.14, 2.96 [16:55:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.45, 3.13 [16:56:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:57:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.09, 3.06, 3.03 [17:06:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:10:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:20:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:21:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.42, 3.18 [17:23:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.61, 2.77, 2.97 [17:23:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:33:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:41:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:01:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:06:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.47, 3.14 [18:09:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48953 MB (10% inode=98%); [18:12:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.51, 3.27, 3.20 [18:14:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.68, 1.75, 1.10 [18:16:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.50, 1.09 [18:22:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.52, 3.30 [18:24:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.81, 3.85, 3.44 [18:26:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.64, 3.41 [18:26:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:28:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.41, 3.26, 3.31 [18:31:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:49:04] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49807 MB (11% inode=98%); [19:51:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:56:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:57:51] [02mediawiki] 07HM100 commented on pull request 03#4696: Bump skins/Evelution from `2cc6564` to `000ea9c` - 13https://git.io/JyZGk [20:03:14] [02mediawiki] 07RhinosF1 commented on pull request 03#4696: Bump skins/Evelution from `2cc6564` to `000ea9c` - 13https://git.io/JyZG9 [20:05:37] [02mediawiki] 07RhinosF1 commented on pull request 03#4696: Bump skins/Evelution from `2cc6564` to `000ea9c` - 13https://git.io/JyZGj [20:05:41] [02mediawiki] 07RhinosF1 closed pull request 03#4696: Bump skins/Evelution from `2cc6564` to `000ea9c` - 13https://git.io/JDHHe [20:05:43] [02mediawiki] 07dependabot[bot] commented on pull request 03#4696: Bump skins/Evelution from `2cc6564` to `000ea9c` - 13https://git.io/JyZZe [20:05:52] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/Evelution-000ea9c [20:05:54] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/Evelution-000ea9c - 13https://git.io/vbL5b [20:06:14] [02mediawiki] 07RhinosF1 edited a comment on pull request 03#4696: Bump skins/Evelution from `2cc6564` to `000ea9c` - 13https://git.io/JyZGj [20:15:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:20:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:55:23] [02mediawiki] 07HM100 commented on pull request 03#4696: Bump skins/Evelution from `2cc6564` to `000ea9c` - 13https://git.io/JyZlh [20:56:13] [02mediawiki] 07HM100 edited a comment on pull request 03#4696: Bump skins/Evelution from `2cc6564` to `000ea9c` - 13https://git.io/JyZlh [20:59:25] [02mediawiki] 07RhinosF1 commented on pull request 03#4696: Bump skins/Evelution from `2cc6564` to `000ea9c` - 13https://git.io/JyZ8R [21:20:40] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.43, 7.06, 4.36 [21:22:38] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.58, 21.61, 17.02 [21:24:38] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.68, 20.26, 17.10 [21:26:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:28:28] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.97, 5.33, 4.79 [21:30:25] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.85, 4.33, 4.47 [21:31:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:42:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:47:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:51:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.49, 3.11 [21:53:03] PROBLEM - db11 APT on db11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:53:44] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.90, 3.03, 2.99 [21:54:07] !sre persistent 503s [21:54:14] I see [21:54:27] I imagine db11 might be to do with it [21:54:32] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 12.98, 12.24, 6.46 [21:54:43] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.75, 2.08, 1.47 [21:55:02] RECOVERY - db11 APT on db11 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [21:55:03] db11 is broken Agent [21:55:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48956 MB (10% inode=98%); [21:55:12] RhinosF1: it seems we're back [21:55:25] Yeah, Meta finally loaded [21:55:37] Agent: OOM I think [21:55:42] Ah [21:56:03] At least it auto-reboots it seems [21:56:06] Yes [21:56:11] Very high load too [21:56:41] paladox: ^ [21:56:52] Agent: I can't tell you much other than that [21:57:00] No db == no wiki [21:58:32] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 1.17, 6.01, 5.20 [22:00:01] Yeah, it seems it was momentary. [22:00:02] I just asked because a user on #general said they couldn't access their wiki [22:00:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.41, 1.69, 1.62 [22:05:26] Agent: np, it's definitely an issue [22:05:47] Hopefully it won't get too bad before infra switch next year [22:33:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 3.78, 3.32 [22:35:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.57, 3.38, 3.23 [22:41:03] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49765 MB (11% inode=98%); [22:44:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.89, 6.05, 4.91 [22:45:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:46:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.27, 5.97, 5.02 [22:49:37] PROBLEM - gluster3 Gluster Disk Space on gluster3 is WARNING: DISK WARNING - free space: /mnt/mediawiki-static 221923 MB (9% inode=80%); [22:49:49] !log increase gluster3 by 250gb [22:49:53] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.24, 1.78, 1.31 [22:49:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:50:36] RECOVERY - gluster3 Disk Space on gluster3 is OK: DISK OK - free space: / 221913 MB (17% inode=82%); [22:51:36] !log increase gluster4 by 250gb [22:51:37] RECOVERY - gluster3 Gluster Disk Space on gluster3 is OK: DISK OK - free space: /mnt/mediawiki-static 443739 MB (16% inode=82%); [22:51:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:51:52] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.77, 1.37, 1.21 [22:52:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.30, 6.82, 5.59 [22:52:35] paladox, ty :) [22:52:50] RECOVERY - gluster4 Disk Space on gluster4 is OK: DISK OK - free space: / 221826 MB (17% inode=81%); [22:52:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.53, 3.38 [22:53:17] guess i should do another 100gb as approved by john [22:53:24] since that'll make it 2.7tb [22:54:09] though i want to confirm with him beforehand since proxmox shows something different to grafana [22:54:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.18, 6.24, 5.52 [22:54:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.71, 3.46 [22:56:42] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 3.38, 3.38 [23:00:04] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:06] !log [@mw11] starting deploy of {'l10nupdate': True} to ovlon [23:00:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:07:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.87, 3.56 [23:11:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:13:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.74, 3.58 [23:13:42] paladox, yeah that seems fine. no urgency for further disk size capacity increases so can definitely wait for you to talk to him [23:15:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.05, 3.54, 3.52 [23:16:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.18, 4.07, 3.71 [23:18:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.72, 3.62 [23:20:47] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.55, 4.40, 3.88 [23:21:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:24:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.56, 3.87, 3.83 [23:27:55] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1670s [23:28:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.53, 4.14, 3.94 [23:30:00] !log [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 1793s [23:30:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.56, 3.74 [23:36:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.91, 2.92, 3.39