[00:00:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:01:57] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.44, 5.82, 6.18 [00:02:17] !log [universalomega@test3] [TEST [[github:miraheze/puppet/pull/2121|miraheze/puppet#2121]]] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=test3wiki --update (END - exit=0) [00:02:17] https://github.com/miraheze/puppet/pull/2121 [00:02:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:19] !log [universalomega@test3] [TEST [[github:miraheze/puppet/pull/2121|miraheze/puppet#2121]]] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=test3wiki --update --active (END - exit=0) [00:03:19] https://github.com/miraheze/puppet/pull/2121 [00:03:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.49, 3.14 [00:04:09] !log [universalomega@test3] [TEST [[github:miraheze/puppet/pull/2121|miraheze/puppet#2121]]] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=test3wiki (END - exit=0) [00:04:09] https://github.com/miraheze/puppet/pull/2121 [00:04:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:05:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.94, 3.85, 3.30 [00:07:05] !log [universalomega@test3] [TEST [[github:miraheze/puppet/pull/2121|miraheze/puppet#2121]]] sudo -u www-data php /srv/mediawiki/w/maintenance/test.php --wiki=test3wiki (END - exit=256) [00:07:05] https://github.com/miraheze/puppet/pull/2121 [00:07:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:07:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.80, 3.43, 3.21 [00:09:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.92, 3.22, 3.15 [00:18:45] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb [00:20:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:40:06] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:40:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.38.69.175/cpweb, 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [00:40:46] PROBLEM - cp13 Stunnel Http for mw9 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:40:51] PROBLEM - cp14 Stunnel Http for mw9 on cp14 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:40:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.38.69.175/cpweb, 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [00:41:31] CosmicAlpha, paladox, I'm getting extreme lag, persistent timeout errors, and recurring 503s crosswiki [00:42:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:42:45] RECOVERY - cp13 Stunnel Http for mw9 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15874 bytes in 0.006 second response time [00:42:45] RECOVERY - cp14 Stunnel Http for mw9 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15860 bytes in 0.036 second response time [00:42:54] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:45:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:49:37] dmehus: looking [00:51:33] thanks :) [00:51:36] I don't see much in Grafana right now though every 10 minutes are so we are getting things going down... [00:51:41] yep [00:51:50] needs SRE Infrastructure to investigate, I think [00:51:56] yeah. [00:51:57] not much MWEs can do I suspect [00:52:26] yeah, I don't see anything I can do right now. [00:54:05] 👍 [00:55:44] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.40, 4.44, 3.53 [00:59:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.96, 3.53 [01:01:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.61, 4.24, 3.68 [01:09:58] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.39, 4.78, 5.80 [01:10:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.25, 18.95, 23.79 [01:11:57] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.96, 5.52, 5.94 [01:12:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.93, 3.87 [01:13:58] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.38, 5.54, 5.90 [01:14:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 4.19, 3.98 [01:15:58] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.45, 5.55, 5.84 [01:17:58] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.57, 4.97, 5.61 [01:20:23] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.33, 20.36, 21.70 [01:20:24] SRE, another 503: [01:20:24] ```Error 503 Backend fetch failed, forwarded for , 127.0.0.1 [01:20:24] (Varnish XID 617087116) via cp15.miraheze.org at Fri, 12 Nov 2021 01:19:52 GMT.``` [01:20:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.95, 3.99 [01:21:58] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.35, 5.50, 5.63 [01:22:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.00, 19.61, 21.29 [01:23:58] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.99, 5.03, 5.45 [01:26:23] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.05, 17.73, 20.15 [01:29:58] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.89, 3.46, 4.65 [01:30:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 2.65, 3.30 [01:40:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.24, 3.26 [01:42:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.78, 3.08, 3.20 [01:57:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.73, 3.38 [02:01:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.76, 3.02, 3.20 [02:03:20] Have you tried turning it off and turning it back on? [02:03:28] /s [02:04:29] PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 1.73, 1.32, 0.99 [02:06:29] RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.98, 1.21, 0.99 [02:14:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JX5Cc [02:14:45] [02miraheze/mw-config] 07Universal-Omega 03e9871b9 - Remove deleted wiki [02:15:50] miraheze/mw-config - Universal-Omega the build passed. [02:24:51] !log [@test3] starting deploy of {'config': True} to skip [02:24:52] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [02:24:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:24:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JX54N [02:32:55] [02miraheze/mw-config] 07Universal-Omega 0353e446f - Remove deleted wiki [02:33:52] miraheze/mw-config - Universal-Omega the build passed. [02:41:33] !log [@mw11] starting deploy of {'config': True} to all [02:41:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:40] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 7s [02:41:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:54:26] !log [@test3] starting deploy of {'config': True} to skip [02:54:27] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [02:54:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:54:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:34:02] PROBLEM - wiki.tigcity.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.tigcity.tk' expires in 7 day(s) (Sat 20 Nov 2021 03:31:27 GMT +0000). [03:44:23] PROBLEM - www.famedata.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'famedata.org' expires in 7 day(s) (Sat 20 Nov 2021 03:44:02 GMT +0000). [03:44:50] PROBLEM - famedata.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'famedata.org' expires in 7 day(s) (Sat 20 Nov 2021 03:44:02 GMT +0000). [04:06:47] PROBLEM - petrawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'en.petrawiki.org' expires in 7 day(s) (Sat 20 Nov 2021 04:06:27 GMT +0000). [04:07:59] PROBLEM - en.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'en.petrawiki.org' expires in 7 day(s) (Sat 20 Nov 2021 04:06:27 GMT +0000). [04:09:09] PROBLEM - www.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'en.petrawiki.org' expires in 7 day(s) (Sat 20 Nov 2021 04:06:27 GMT +0000). [04:12:50] PROBLEM - files.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'files.petrawiki.org' expires in 7 day(s) (Sat 20 Nov 2021 04:08:07 GMT +0000). [04:19:07] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49677 MB (11% inode=98%); [04:21:00] PROBLEM - translate.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'translate.petrawiki.org' expires in 7 day(s) (Sat 20 Nov 2021 04:17:44 GMT +0000). [05:24:01] PROBLEM - privacywiki.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'privacywiki.xyz' expires in 15 day(s) (Sun 28 Nov 2021 05:21:11 GMT +0000). [06:04:46] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.73, 5.84, 3.43 [06:06:46] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.71, 6.48, 3.96 [06:08:46] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.57, 7.36, 4.60 [06:10:46] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.66, 7.55, 5.02 [06:12:46] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.81, 8.26, 5.60 [06:18:46] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.25, 7.86, 6.40 [06:26:46] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.76, 6.46, 6.28 [06:29:42] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.41, 5.08, 3.78 [06:31:39] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.77, 4.25, 3.64 [06:33:07] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48952 MB (10% inode=98%); [06:49:32] PROBLEM - hr.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'hr.petrawiki.org' expires in 7 day(s) (Sat 20 Nov 2021 06:49:08 GMT +0000). [07:12:36] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.18, 5.35, 4.37 [07:14:33] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.23, 4.98, 4.36 [07:35:07] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 50660 MB (11% inode=98%); [07:51:15] .mh SAL [07:51:15] https://meta.miraheze.org/wiki/SAL [07:57:47] dmehus: what DB changes that you mentioned in #miraheze-feed-meta? I don't think I've done to much DB stuff lately. [08:52:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.53, 4.84, 4.49 [08:54:00] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.51, 4.33, 4.34 [09:37:58] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.58, 7.84, 6.57 [09:39:56] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.14, 7.66, 6.67 [09:43:51] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.02, 7.35, 6.72 [09:45:48] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.21, 7.28, 6.77 [09:47:46] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.18, 6.51, 6.55 [10:18:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.69, 6.91, 6.14 [10:20:01] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.17, 6.95, 6.25 [10:21:58] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.43, 6.62, 6.21 [10:22:11] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.36, 7.42, 4.72 [10:24:11] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.22, 6.13, 4.56 [10:27:48] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.11, 6.88, 6.48 [10:29:46] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.21, 6.13, 6.24 [11:52:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.33, 6.12, 4.38 [11:54:11] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.99, 5.32, 4.30 [12:07:38] PROBLEM - mc-wiki.failboat.fun - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mc-wiki.failboat.fun' expires in 15 day(s) (Sun 28 Nov 2021 11:59:39 GMT +0000). [13:01:07] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48937 MB (10% inode=98%); [13:24:55] PROBLEM - www.recyclerie-uzege.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.recyclerie-uzege.org' expires in 15 day(s) (Sun 28 Nov 2021 13:16:35 GMT +0000). [13:49:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.50, 3.16 [13:51:39] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.70, 4.01, 3.39 [13:53:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.74, 3.36 [13:55:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.62, 3.35, 3.26 [14:05:07] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49556 MB (11% inode=98%); [14:10:55] Hello, [14:10:55] when trying to create an account this message is given back "reCAPTCHA validation failed: score retrieved from server is lower than the minimum required score. Please try again later. If problems persist, please contact sre-mediawiki@miraheze.org." [14:10:56] This happens on desktop and mobile using different browsers and IP adresses. [14:11:01] anyone got an idea? [14:13:40] Guest35: What's the username you're trying to create account with? [14:16:07] Nudeltime [14:18:27] E-mail stewards@miraheze to create account for you if you are getting this error everytime [14:20:42] alright thanks [14:27:06] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.34, 4.90, 4.49 [14:29:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.88, 4.65, 4.45 [14:41:39] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=monstergirlencyclopediawiki /home/reception/monstergirlencyclopedia_pages_full.xml.7z --username-prefix wikia:monstergirlencyclopedia --report 1 (START) [14:41:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:42:05] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=monstergirlencyclopediawiki /home/reception/monstergirlencyclopedia_pages_full.xml.7z --username-prefix wikia:monstergirlencyclopedia --report 1 (END - exit=256) [14:42:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:43:13] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=monstergirlencyclopediawiki /home/reception/monstergirlencyclopedia_pages_full.xml --username-prefix wikia:monstergirlencyclopedia --report 1 (START) [14:43:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:43:46] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.01, 20.32, 18.65 [14:44:46] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.38, 5.04, 4.21 [14:45:07] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.77, 20.47, 18.07 [14:46:46] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.54, 5.01, 4.29 [14:46:47] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JXbOv [14:46:48] [02miraheze/ssl] 07Reception123 036eba746 - add docse.tk cert [14:49:07] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.09, 19.32, 18.20 [14:49:45] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JXbOj [14:49:47] [02miraheze/ssl] 07Reception123 0322b586b - add conlang.tk cert [14:50:11] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=monstergirlencyclopediawiki /home/reception/monstergirlencyclopedia_pages_full.xml --username-prefix wikia:monstergirlencyclopedia --report 1 (END - exit=256) [14:50:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:51:24] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JXb3B [14:51:25] [02miraheze/ssl] 07Reception123 03f1e0bc2 - add notes.mridulpm.in cert [14:52:44] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.65, 5.04, 4.60 [14:53:31] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.91, 20.09, 19.63 [14:57:34] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+0/-1/±6] 13https://git.io/JXbGt [14:57:36] [02miraheze/ssl] 07Reception123 03f2b3ac1 - renew SSL certs [14:57:37] [02miraheze/ssl] 07Reception123 03c82a645 - remove savage-wiki.com cert (wiki deleted) [14:59:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.35, 5.02, 4.77 [15:00:45] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[savage-wiki.com_private] [15:01:32] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.52, 4.82, 4.72 [15:01:37] PROBLEM - cp13 Puppet on cp13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[savage-wiki.com_private] [15:08:15] RECOVERY - vrcdev.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'vrcdev.wiki' will expire on Thu 10 Feb 2022 13:56:08 GMT +0000. [15:08:55] RECOVERY - wiki.hausos.co - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hausos.co' will expire on Thu 10 Feb 2022 13:52:04 GMT +0000. [15:09:33] RECOVERY - id.altilunium.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'id.altilunium.xyz' will expire on Thu 10 Feb 2022 13:52:57 GMT +0000. [15:13:10] RECOVERY - wiki.alathramc.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.alathramc.com' will expire on Thu 10 Feb 2022 13:53:33 GMT +0000. [15:14:09] RECOVERY - diamowiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'diamowiki.ga' will expire on Thu 10 Feb 2022 13:54:32 GMT +0000. [15:14:29] RECOVERY - savage-wiki.com - RapidSSL on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [15:15:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.69, 20.77, 19.37 [15:17:20] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.37, 20.31, 19.37 [15:17:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.51, 4.90, 4.56 [15:19:32] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.22, 4.52, 4.45 [15:22:08] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48925 MB (10% inode=98%); [15:26:48] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.52, 18.53, 18.08 [15:28:34] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:28:43] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.12, 17.42, 17.72 [15:29:37] RECOVERY - cp13 Puppet on cp13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:36:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 4.01, 3.51 [15:37:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.72, 5.60, 4.98 [15:38:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.87, 3.52 [15:39:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.29, 5.34, 4.95 [15:40:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.21, 3.86, 3.55 [15:41:27] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.63, 4.98, 4.86 [15:44:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.75, 3.58 [15:44:42] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.20, 5.14, 4.68 [15:46:42] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.81, 4.97, 4.67 [15:48:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.14, 3.36 [15:51:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.03, 19.87, 19.02 [15:52:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.79, 3.81, 3.56 [15:54:17] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.42, 5.78, 5.30 [15:55:27] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.30, 19.42, 19.14 [15:56:14] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.42, 5.57, 5.27 [15:56:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.56, 3.52 [15:58:10] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.33, 4.93, 5.06 [16:00:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.65, 3.10, 3.34 [16:18:28] [02mw-config] 07CloakSelf opened pull request 03#4217: Add zhwp as import per T8281 - 13https://git.io/JXbKW [16:19:26] miraheze/mw-config - CloakSelf the build has errored. [16:21:31] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49454 MB (11% inode=98%); [16:22:21] [02mw-config] 07CloakSelf synchronize pull request 03#4217: Add zhwp as import per T8281 - 13https://git.io/JXbKW [16:23:19] miraheze/mw-config - CloakSelf the build has errored. [16:25:42] Deja Vu :/ [16:26:53] [02mw-config] 07CloakSelf synchronize pull request 03#4217: Add zhwp as import per T8281 - 13https://git.io/JXbKW [16:27:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.92, 5.65, 4.99 [16:27:53] miraheze/mw-config - CloakSelf the build passed. [16:28:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.76, 21.12, 19.88 [16:29:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.48, 5.24, 4.93 [16:31:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.72, 6.75, 5.48 [16:32:42] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 11.71, 7.42, 5.25 [16:32:53] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 38.98, 28.38, 22.87 [16:32:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.38.69.175/cpweb, 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [16:33:06] The ], don't match up [16:33:11] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 37.86, 25.37, 20.73 [16:33:13] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.70, 7.11, 5.43 [16:33:13] PROBLEM - cp13 Stunnel Http for mw13 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:33:24] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 12.98, 7.63, 5.45 [16:33:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.41, 7.73, 5.49 [16:33:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:33:47] PROBLEM - cp13 Stunnel Http for mw9 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:33:56] Production has very few workers [16:34:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.35, 6.72, 5.19 [16:34:15] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.38.69.175/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [16:34:17] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:34:25] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.92, 6.82, 5.40 [16:35:06] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.50, 8.52, 6.28 [16:35:08] RECOVERY - cp13 Stunnel Http for mw13 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.006 second response time [16:35:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.53, 7.53, 5.82 [16:35:16] [02mw-config] 07CloakSelf synchronize pull request 03#4217: Add zhwp as import per T8281 - 13https://git.io/JXbKW [16:35:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.55, 7.75, 5.82 [16:35:35] Yeah, caught that [16:35:42] RECOVERY - cp13 Stunnel Http for mw9 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15860 bytes in 0.014 second response time [16:36:20] miraheze/mw-config - CloakSelf the build passed. [16:36:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.77, 6.52, 5.44 [16:37:06] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.92, 7.25, 6.06 [16:38:23] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20011 bytes in 1.133 second response time [16:38:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:39:22] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.75, 6.88, 6.00 [16:39:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.43, 6.38, 5.73 [16:40:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.89, 6.93, 5.90 [16:40:15] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:41:06] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.82, 6.64, 6.13 [16:41:07] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.15, 23.32, 22.21 [16:41:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.78, 6.11, 5.79 [16:41:17] It's still wrong [16:41:22] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.74, 6.02, 5.80 [16:42:13] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.99, 6.15, 5.75 [16:43:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:44:42] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.43, 5.79, 5.93 [16:45:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.58, 3.19 [16:46:42] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.16, 5.76, 5.89 [16:47:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.34, 3.14 [16:48:42] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.65, 5.58, 5.82 [16:49:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.09, 3.20, 3.10 [16:52:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.25, 22.88, 23.98 [16:53:07] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.37, 19.19, 20.33 [16:56:53] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.65, 23.58, 23.85 [16:58:42] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.09, 5.60, 5.52 [17:00:42] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.61, 5.38, 5.44 [17:00:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.54, 23.00, 23.65 [17:03:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.72, 4.63, 5.66 [17:11:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.63, 5.57, 5.63 [17:13:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.01, 5.53, 5.62 [17:15:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.34, 5.57, 5.61 [17:16:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.23, 18.71, 20.40 [17:17:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.81, 4.77, 5.30 [17:18:42] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.51, 4.37, 4.91 [17:21:27] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.04, 4.20, 4.97 [17:22:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.91, 3.33 [17:24:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.80, 3.36 [17:30:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.12, 3.29, 3.27 [17:30:42] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.63, 5.52, 5.04 [17:34:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.64, 3.44 [17:34:42] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.52, 5.04, 4.95 [17:36:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.61, 3.44 [17:37:31] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48944 MB (10% inode=98%); [17:52:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.91, 3.82 [18:00:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.73, 2.79, 3.33 [18:00:46] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.82, 5.28, 4.65 [18:01:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.00, 5.93, 5.05 [18:02:42] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.89, 5.25, 4.70 [18:03:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.16, 5.55, 5.01 [18:04:42] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.73, 4.86, 4.64 [18:05:27] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.62, 5.03, 4.90 [18:08:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.58, 3.43 [18:11:24] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.03, 19.62, 18.64 [18:13:19] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.88, 19.80, 18.83 [18:16:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.17, 3.64, 3.59 [18:17:10] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.33, 20.65, 19.42 [18:19:07] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.96, 20.07, 19.36 [18:22:14] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.78, 5.94, 5.17 [18:22:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.88, 3.92, 3.63 [18:22:42] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.72, 5.01, 4.55 [18:23:07] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.55, 21.69, 20.26 [18:24:10] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.23, 5.37, 5.06 [18:24:42] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.96, 4.98, 4.59 [18:27:26] hmm everythings seems very slow [18:28:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.86, 3.74 [18:29:07] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.73, 20.38, 20.17 [18:30:00] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.74, 5.66, 5.26 [18:33:53] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.76, 5.28, 5.22 [18:34:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.35, 2.91, 3.37 [18:39:24] Reception123: mind trying your import again? Seems to have failed. [18:39:30] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.73, 5.27, 4.91 [18:39:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.74, 21.60, 20.30 [18:40:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.40, 3.47 [18:41:24] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.40, 4.83, 4.78 [18:42:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.10, 3.26, 3.40 [18:45:33] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.92, 5.63, 5.46 [18:46:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.59, 3.52 [18:47:58] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=monstergirlencyclopediawiki /home/reception/monstergirlencyclopedia_pages_full.xml --username-prefix wikia:monstergirlencyclopedia --report 1 (START) [18:48:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:48:07] thanks, that socket error again [18:49:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.91, 5.51, 5.47 [18:50:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.84, 8.24, 6.37 [18:51:00] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.91, 7.35, 6.20 [18:52:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:52:46] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.20, 7.14, 6.20 [18:52:59] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.28, 7.21, 6.28 [18:53:27] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.74, 21.18, 20.12 [18:54:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.79, 6.67, 6.13 [18:56:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.83, 3.63 [18:56:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.89, 6.80, 6.34 [18:57:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:58:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.64, 3.58 [19:00:04] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=monstergirlencyclopediawiki /home/reception/monstergirlencyclopedia_pages_full.xml --username-prefix wikia:monstergirlencyclopedia --report 1 (END - exit=256) [19:00:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.91, 4.22, 3.81 [19:01:07] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.44, 19.96, 20.09 [19:02:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.99, 3.77 [19:04:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.99, 3.79 [19:05:31] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49338 MB (11% inode=98%); [19:07:27] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.54, 18.97, 20.12 [19:08:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.81, 3.74 [19:11:27] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.68, 4.62, 5.00 [19:16:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.73, 3.72 [19:18:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.53, 3.65 [19:24:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.74, 3.87, 3.71 [19:26:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.65, 3.65 [19:28:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 3.80, 3.70 [19:30:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.59, 3.63 [19:33:04] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.53, 20.06, 19.54 [19:34:40] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.78, 5.39, 5.13 [19:35:02] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.18, 19.80, 19.54 [19:36:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.31, 2.77, 3.27 [19:40:30] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.38, 4.59, 4.91 [19:42:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.37, 3.40 [19:42:55] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.41, 20.26, 19.67 [19:44:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.30, 19.82, 19.54 [19:48:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.84, 3.23, 3.35 [19:52:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.79, 3.40, 3.42 [19:53:59] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.45, 5.58, 5.21 [19:54:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.48, 21.23, 20.28 [19:59:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.09, 6.03, 5.48 [20:00:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.81, 3.23, 3.39 [20:00:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.26, 20.24, 20.16 [20:01:31] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48939 MB (10% inode=98%); [20:03:42] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.68, 5.96, 5.59 [20:05:39] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.28, 6.04, 5.65 [20:07:35] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.04, 5.44, 5.47 [20:09:32] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.15, 5.64, 5.53 [20:11:29] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.90, 5.07, 5.34 [20:15:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.21, 6.41, 5.11 [20:16:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.08, 22.19, 20.80 [20:17:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.13, 6.20, 5.18 [20:17:27] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.04, 4.70, 5.10 [20:18:09] RhinosF1? you have any idea why Grafana doesn't load for me? [20:18:26] ugochimobi: what page [20:19:45] the whole of it [20:19:52] from root domain [20:21:04] What error [20:21:18] .mh grafana . [20:21:19] https://grafana.miraheze.org/wiki/. [20:21:39] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1 fine for me [20:23:15] I wonder why it didn't load for me, several times in the past [20:23:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.33, 5.21, 5.21 [20:24:09] Without an error, we can't say [20:25:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.18, 5.22, 5.19 [20:26:00] Then, it just says Cannot load grafana [20:26:00] The it will now give me tips to check if my network is okay, or my browser is supported and some other things [20:29:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.19, 5.55, 5.36 [20:30:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.09, 19.13, 20.18 [20:31:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.05, 6.12, 5.59 [20:32:06]