[00:01:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.02, 4.09, 3.40 [00:03:23] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 33.70, 24.85, 21.65 [00:05:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.90, 3.49 [00:05:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.72, 21.46, 20.79 [00:05:38] [02DataDump] 07ZabeMath synchronize pull request 03#44: Replace usages of deprecated wfWikiID() - 13https://git.io/JyJWq [00:05:52] [02DataDump] 07ZabeMath reviewed pull request 03#44 commit - 13https://git.io/JyUet [00:06:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.98, 3.56 [00:07:15] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.18, 21.69, 20.89 [00:08:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.92, 3.59 [00:10:15] miraheze/DataDump - ZabeMath the build passed. [00:10:26] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 11.89, 7.20, 5.06 [00:10:47] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.72, 3.17, 3.36 [00:10:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.24, 3.25, 2.16 [00:14:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:14:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.64, 3.50 [00:14:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.82, 1.86, 1.82 [00:16:34] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.16, 3.15, 3.34 [00:16:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.43, 1.44, 1.68 [00:19:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:20:12] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.00, 5.68, 5.31 [00:22:09] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.04, 5.69, 5.35 [00:31:04] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.57, 3.41 [00:31:56] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.04, 5.96, 5.91 [00:34:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:34:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.81, 3.34, 3.37 [00:35:51] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.15, 5.91, 5.89 [00:36:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.98, 1.71, 1.42 [00:37:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 1.67, 4.38, 5.33 [00:38:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.36, 1.32 [00:40:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.02, 19.82, 23.53 [00:41:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.05, 4.02, 5.01 [00:45:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:46:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.63, 3.38 [00:54:38] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.93, 17.81, 20.34 [00:55:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:56:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.98, 3.75 [00:58:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 4.20, 3.86 [01:02:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.80, 3.78 [01:04:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.15, 21.26, 20.67 [01:05:37] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.72, 5.66, 4.92 [01:07:34] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.89, 4.83, 4.72 [01:08:38] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.72, 19.18, 20.10 [01:16:59] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.19, 1.60, 1.10 [01:18:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.11, 3.88, 3.75 [01:18:59] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.65, 1.58, 1.15 [01:22:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.82, 3.89, 3.83 [01:26:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.95, 3.86 [01:28:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.69, 3.77 [01:32:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 4.04, 3.86 [01:34:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.77, 3.79 [01:36:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 4.00, 3.88 [01:40:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.85, 3.93, 3.87 [01:44:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.76, 4.15, 3.95 [01:44:53] [02DataDump] 07The-Voidwalker closed pull request 03#44: Replace usages of deprecated wfWikiID() - 13https://git.io/JyJWq [01:44:54] [02miraheze/DataDump] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyUy9 [01:44:56] [02miraheze/DataDump] 07ZabeMath 034fd5856 - Replace usages of deprecated wfWikiID() (#44) [01:49:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:49:50] miraheze/DataDump - The-Voidwalker the build passed. [01:54:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:58:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:04:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.59, 3.60, 3.95 [02:08:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:14:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.63, 3.81 [02:14:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:16:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.79, 3.85 [02:19:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:20:05] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.93, 6.81, 5.66 [02:22:05] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.37, 6.58, 5.71 [02:24:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:26:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.37, 3.50 [02:27:22] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.50, 3.12, 2.01 [02:29:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:30:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.52, 3.53 [02:32:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.64, 3.57 [02:33:17] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.75, 1.66, 1.73 [02:34:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.52, 3.53 [02:35:16] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.03, 1.54, 1.68 [02:35:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:40:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.83, 3.82, 3.60 [02:40:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:42:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.79, 3.50, 3.52 [02:51:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:52:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.71, 3.57 [02:54:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.65, 3.56 [02:56:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.69, 3.96, 3.68 [02:56:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:58:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.79, 3.65 [03:00:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.67, 4.03, 3.75 [03:06:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.64, 3.69 [03:14:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.32, 3.03, 3.40 [03:20:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.24, 4.02, 3.62 [03:32:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.92, 3.63, 3.72 [03:33:34] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [03:42:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 2.99, 3.36 [03:50:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.55, 3.46 [03:52:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.41, 3.41 [03:54:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.16, 3.72, 3.53 [03:56:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.48, 3.46 [03:58:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.26, 3.17, 3.36 [04:04:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.61, 3.45 [04:06:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.42, 3.40 [04:08:33] PROBLEM - ambientguitar.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'ambientguitar.org' expires in 7 day(s) (Fri 31 Dec 2021 04:08:32 GMT +0000). [04:10:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.82, 3.07, 3.26 [04:14:55] PROBLEM - www.ambientguitar.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'ambientguitar.org' expires in 7 day(s) (Fri 31 Dec 2021 04:08:32 GMT +0000). [04:17:00] PROBLEM - wiki.timewisely.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.timewisely.net' expires in 7 day(s) (Fri 31 Dec 2021 04:13:14 GMT +0000). [04:22:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.74, 3.46 [04:28:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.62, 3.49 [04:29:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:34:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:35:46] RhinosF1! [04:36:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.56, 3.88, 3.63 [04:38:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.89, 3.41, 3.48 [04:40:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.68, 3.57 [04:42:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.92, 3.41, 3.48 [04:44:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.28, 3.20, 3.38 [04:48:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.49, 3.45 [04:56:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.73, 3.56 [04:58:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.59, 3.52 [05:04:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.62, 3.13, 3.38 [05:10:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.38, 3.40 [05:12:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.22, 3.34 [05:16:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.79, 3.55 [05:20:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.11, 3.74, 3.57 [05:22:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.50, 3.50 [05:28:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.32, 3.06, 3.33 [05:46:59] PROBLEM - adanfime.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.22415883446080:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../ssl/record/rec_layer_s3.c:1544:SSL alert number 80 [05:53:58] RECOVERY - adanfime.com - LetsEncrypt on sslhost is OK: OK - Certificate 'adanfime.com' will expire on Wed 23 Mar 2022 04:47:12 GMT +0000. [06:12:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 3.47, 3.09 [06:14:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.32, 3.07 [06:16:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.32, 3.30, 3.09 [06:18:51] RECOVERY - mail2 APT on mail2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:22:50] RECOVERY - graylog2 APT on graylog2 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [06:40:44] RECOVERY - puppet3 APT on puppet3 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [06:41:18] PROBLEM - adanfime.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - adanfime.com reverse DNS resolves to unalocated.63.wixsite.com [07:39:38] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JyTQg [07:39:39] [02miraheze/mw-config] 07Reception123 030271fd7 - CC-BY-SA 3.0 for tlhwiki per T8456 [07:39:41] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [07:39:42] [02mw-config] 07Reception123 opened pull request 03#4297: CC-BY-SA 3.0 for tlhwiki per T8456 - 13https://git.io/JyTQ2 [07:40:43] miraheze/mw-config - Reception123 the build passed. [07:40:54] [02mw-config] 07Reception123 closed pull request 03#4297: CC-BY-SA 3.0 for tlhwiki per T8456 - 13https://git.io/JyTQ2 [07:40:56] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyTQX [07:40:57] [02miraheze/mw-config] 07Reception123 03687b434 - CC-BY-SA 3.0 for tlhwiki per T8456 (#4297) [07:40:59] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [07:41:00] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [07:41:52] !log [@mw11] starting deploy of {'config': True} to ovlon [07:41:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:41:59] miraheze/mw-config - Reception123 the build passed. [07:42:02] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 10s [07:42:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:45:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48945 MB (10% inode=98%); [07:47:03] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 52865 MB (11% inode=98%); [07:54:25] !log [@test3] starting deploy of {'config': True} to skip [07:54:26] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [07:54:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:54:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:05:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48846 MB (10% inode=98%); [08:33:05] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26705 MB (5% inode=98%); [09:12:34] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.45, 6.37, 4.77 [09:13:41] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.12, 6.17, 4.73 [09:14:14] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.21, 6.90, 5.14 [09:14:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:14:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.85, 6.46, 5.00 [09:18:34] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.14, 7.12, 5.59 [09:20:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.21, 6.95, 5.81 [09:20:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.67, 6.64, 5.60 [09:21:41] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.53, 6.01, 5.38 [09:22:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.26, 6.32, 5.72 [09:24:11] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.80, 2.18, 1.32 [09:24:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:26:10] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.02, 1.79, 1.28 [09:28:08] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.68, 1.40, 1.20 [10:15:05] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 27350 MB (6% inode=98%); [11:47:14] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.80, 5.99, 4.63 [11:47:41] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.72, 6.46, 4.73 [11:47:45] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.18, 6.10, 4.69 [11:48:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.24, 18.50, 14.35 [11:49:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.23, 5.76, 4.73 [11:49:41] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.42, 6.43, 4.91 [11:49:45] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.32, 5.30, 4.56 [11:50:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:50:38] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.80, 18.50, 14.83 [11:51:41] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.21, 6.11, 4.99 [11:55:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:07:22] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JykVN [12:07:23] [02miraheze/CreateWiki] 07translatewiki 035ad7342 - Localisation updates from https://translatewiki.net. [12:07:24] [url] Main page - translatewiki.net | translatewiki.net [12:07:25] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JykVA [12:07:27] [02miraheze/landing] 07translatewiki 03b18e4a4 - Localisation updates from https://translatewiki.net. [12:07:27] [url] Main page - translatewiki.net | translatewiki.net [12:07:28] [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JykVh [12:07:29] [02miraheze/DataDump] 07translatewiki 03bcb4503 - Localisation updates from https://translatewiki.net. [12:07:30] [url] Main page - translatewiki.net | translatewiki.net [12:07:31] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JykVj [12:07:32] [02miraheze/MirahezeMagic] 07translatewiki 03b1e2aed - Localisation updates from https://translatewiki.net. [12:07:33] [url] Main page - translatewiki.net | translatewiki.net [12:10:07] miraheze/landing - translatewiki the build passed. [12:15:33] miraheze/CreateWiki - translatewiki the build passed. [12:16:40] miraheze/DataDump - translatewiki the build passed. [12:18:22] miraheze/MirahezeMagic - translatewiki the build passed. [12:37:05] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26699 MB (5% inode=98%); [13:04:45] Reception123: ^ [13:06:11] RhinosF1: no access but eh we'll move soon so no need to panic yet [13:52:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.15, 2.77 [13:56:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.59, 3.07, 2.85 [14:16:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.06, 2.80 [14:18:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.06, 2.79, 2.74 [14:21:05] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 32290 MB (7% inode=98%); [14:57:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.18, 2.97 [15:00:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.28, 2.89, 2.91 [15:00:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.71, 1.29, 0.90 [15:02:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.98, 1.15, 0.90 [15:12:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.60, 3.27 [15:16:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.58, 3.72, 3.37 [15:18:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.23, 3.54, 3.35 [15:18:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:20:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.82, 3.47 [15:22:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.51, 3.39 [15:23:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:28:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.13, 3.05, 3.27 [15:33:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:38:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.29, 3.18 [15:38:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:40:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.36, 3.22 [15:44:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.62, 3.77, 3.41 [15:46:03] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.72, 1.54, 1.27 [15:46:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.63, 3.42 [15:48:01] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.99, 1.30, 1.21 [15:50:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.75, 3.50 [15:52:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.50, 3.44 [15:54:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.67, 3.50 [15:56:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.51, 3.30, 3.39 [15:58:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:04:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.54, 3.45 [16:06:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.34, 3.27, 3.35 [16:08:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:09:45] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.54, 6.47, 5.30 [16:10:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.67, 3.50 [16:11:45] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.02, 5.98, 5.27 [16:12:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.45, 4.24, 3.72 [16:16:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.75, 3.66 [16:16:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:18:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 3.96, 3.74 [16:21:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:28:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.91, 3.65, 3.82 [16:40:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.64, 3.62 [16:44:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.18, 3.44 [16:46:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.68, 3.00, 3.34 [16:52:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.92, 3.33, 3.41 [16:54:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.62, 3.50 [16:56:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.23, 3.41, 3.44 [16:58:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 3.17, 3.34 [17:23:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.68, 3.27 [17:23:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:25:04] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.59, 3.28 [17:27:00] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 3.94, 3.44 [17:38:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:38:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.93, 3.80 [17:44:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.88, 4.00, 3.78 [17:46:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.87, 3.76 [17:52:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.80, 3.85, 3.72 [17:58:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.85, 3.55, 3.69 [17:59:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:04:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:06:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.75, 2.90, 3.32 [18:10:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.78, 3.59, 3.49 [18:12:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.39, 3.27, 3.40 [18:24:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:29:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:32:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.46, 3.41 [18:34:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.72, 3.51 [18:36:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.80, 3.56 [18:40:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.93, 3.20, 3.36 [18:41:45] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.30, 6.29, 5.55 [18:41:49] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:52] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.99, 6.24, 3.72 [18:42:14] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.37, 7.55, 5.62 [18:42:15] Here [18:42:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:42:39] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.53, 22.11, 17.33 [18:42:52] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:43:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:43:41] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.55, 6.60, 5.20 [18:43:47] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 20514 bytes in 0.371 second response time [18:44:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.37, 7.25, 5.76 [18:44:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:44:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.20, 21.44, 17.70 [18:44:49] !sre [18:44:52] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:45:04] JohnLewis, paladox, Voidwalker: around? [18:45:13] yes [18:45:21] just got here :) [18:45:41] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.71, 6.28, 5.24 [18:45:45] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.61, 6.14, 5.72 [18:45:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.96, 5.33, 3.96 [18:46:38] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.24, 19.99, 17.62 [18:47:52] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.09, 4.82, 3.94 [18:48:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.58, 6.78, 5.88 [18:48:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:52:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:53:16] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.61, 3.92, 3.56 [18:55:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.87, 3.60 [18:59:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.10, 3.26, 3.40 [19:02:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:08:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.43, 3.42 [19:10:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.69, 3.52 [19:12:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.63, 3.53 [19:14:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.78, 3.22, 3.38 [19:17:48] PROBLEM - phab2 Current Load on phab2 is CRITICAL: CRITICAL - load average: 1.58, 2.79, 1.37 [19:18:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.66, 3.50 [19:19:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:19:47] PROBLEM - phab2 Current Load on phab2 is WARNING: WARNING - load average: 0.29, 1.90, 1.22 [19:20:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.67, 3.53 [19:21:47] RECOVERY - phab2 Current Load on phab2 is OK: OK - load average: 0.04, 1.27, 1.06 [19:22:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 3.26, 3.40 [19:24:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:30:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.31, 3.30 [19:32:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.36, 3.31 [19:34:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.32, 3.16, 3.23 [19:38:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JyIDf [19:38:03] [02miraheze/puppet] 07paladox 0369fadbc - roundcube: Upgrade to 1.5.1 [19:38:04] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:38:06] [02puppet] 07paladox opened pull request 03#2171: roundcube: Upgrade to 1.5.1 - 13https://git.io/JyIDJ [19:41:21] [02puppet] 07paladox closed pull request 03#2171: roundcube: Upgrade to 1.5.1 - 13https://git.io/JyIDJ [19:41:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyIDR [19:41:24] [02miraheze/puppet] 07paladox 03f1fad23 - roundcube: Upgrade to 1.5.1 (#2171) [19:41:25] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:41:27] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [19:51:04] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.56, 3.28 [19:52:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.45, 3.28 [19:54:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.11, 2.95, 3.11 [20:22:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:29:41] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.04, 6.68, 5.74 [20:31:41] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.79, 6.16, 5.68 [20:31:57] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.74, 8.02, 6.48 [20:37:45] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.92, 7.57, 6.83 [20:41:37] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.79, 6.77, 6.69 [20:43:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.50, 3.35 [20:47:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.72, 3.46 [20:48:59] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.83, 6.88, 5.96 [20:49:10] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.60, 7.07, 6.30 [20:49:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.61, 6.99, 6.77 [20:51:07] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.66, 6.66, 6.25 [20:52:51] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.04, 6.42, 6.02 [20:59:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.92, 3.86 [20:59:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.71, 6.17, 6.61 [21:00:57] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.94, 3.87 [21:04:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.99, 3.91 [21:07:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:11:39] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.02, 6.97, 6.69 [21:12:28] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.02, 6.25, 5.87 [21:12:30] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.39, 2.86, 3.39 [21:15:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.44, 7.69, 7.08 [21:16:26] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.48, 6.14, 5.93 [21:17:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.75, 8.00, 7.26 [21:19:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.84, 7.45, 7.14 [21:20:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.37, 3.42 [21:22:22] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.29, 7.28, 6.48 [21:23:15] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.53, 6.35, 6.76 [21:24:22] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.26, 6.35, 6.23 [21:26:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.82, 3.23, 3.36 [21:30:03] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.35, 1.91, 1.21 [21:32:03] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.14, 2.16, 1.38 [21:34:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.52, 1.52, 1.24 [21:44:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.20, 4.08, 3.53 [21:46:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.84, 3.49 [21:48:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.99, 3.59 [21:52:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.85, 3.64 [22:06:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.82, 3.17, 3.35 [22:10:36] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.24, 6.99, 5.67 [22:11:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 8.00, 6.83, 5.55 [22:12:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:13:14] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.45, 7.38, 5.91 [22:13:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.08, 6.69, 5.56 [22:14:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb [22:14:24] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:30] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:40] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:52] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [22:15:16] urgh [22:15:45] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:47] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:28] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.58, 7.95, 6.56 [22:16:29] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.632 second response time [22:16:30] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.501 second response time [22:16:38] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:16:44] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.068 second response time [22:17:10] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:17:21] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:17:32] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.44, 6.72, 5.85 [22:18:14] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.64, 6.93, 5.61 [22:18:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:18:25] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.85, 7.97, 6.71 [22:19:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.86, 4.25, 3.69 [22:19:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.71, 7.99, 6.71 [22:20:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.16, 7.12, 5.85 [22:21:14] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.32, 8.43, 7.02 [22:21:27] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.013 second response time [22:21:57] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 2.557 second response time [22:22:06] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.217 second response time [22:22:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:22:42]