[00:01:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.75, 3.73 [00:02:02] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.50, 20.08, 18.32 [00:04:00] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.30, 19.52, 18.32 [00:09:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.79, 3.83 [00:09:21] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.10, 5.69, 4.70 [00:09:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.66, 22.06, 19.74 [00:11:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 4.22, 4.00 [00:11:17] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.06, 6.46, 6.24 [00:11:18] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.28, 5.68, 4.82 [00:13:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.39, 3.54, 3.77 [00:13:15] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.14, 6.69, 5.30 [00:13:16] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.85, 6.49, 6.29 [00:19:37] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.82, 24.59, 21.99 [00:19:50] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:21:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:21:48] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:48] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.830 second response time [00:23:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.19, 2.85, 3.32 [00:23:48] [02mw-config] 07Universal-Omega edited pull request 03#4341: Add `$wgTemplateStylesNamespaces` to ManageWikiNamespaces - 13https://git.io/JSSk8 [00:23:59] [02mw-config] 07Universal-Omega closed pull request 03#4341: Add `$wgTemplateStylesNamespaces` to ManageWikiNamespaces - 13https://git.io/JSSk8 [00:24:01] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [00:24:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSSrj [00:24:04] [02miraheze/mw-config] 07Universal-Omega 035412a2e - Add `$wgTemplateStylesNamespaces` to ManageWikiNamespaces (#4341) [00:24:05] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [00:24:14] !log [@test3] starting deploy of {'config': True} to skip [00:24:15] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:24:24] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 3.399 second response time [00:25:11] miraheze/mw-config - Universal-Omega the build passed. [00:25:28] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:25:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:25:38] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [00:25:41] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:25:45] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.313 second response time [00:25:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:26:21] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.449 second response time [00:27:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:28:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:28:55] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JSSoq [00:28:56] [02miraheze/mw-config] 07Universal-Omega 03d6d4eee - Enable commentlist api on SCSVG [00:28:58] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [00:28:59] [02mw-config] 07Universal-Omega opened pull request 03#4343: Enable commentlist api on SCSVG - 13https://git.io/JSSom [00:30:00] miraheze/mw-config - Universal-Omega the build passed. [00:33:50] [02mw-config] 07Universal-Omega closed pull request 03#4343: Enable commentlist api on SCSVG - 13https://git.io/JSSom [00:33:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSSon [00:33:53] [02miraheze/mw-config] 07Universal-Omega 038ecc08b - Enable commentlist api on SCSVG (#4343) [00:33:55] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [00:33:56] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [00:34:56] miraheze/mw-config - Universal-Omega the build passed. [00:42:06] !log [@mw11] starting deploy of {'config': True} to ovlon [00:42:34] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 28s [00:42:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:42:44] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:42:51] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [00:43:00] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:43:02] ... [00:43:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:47:58] !log [@test101] starting deploy of {'config': True} to skip [00:47:59] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:48:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:48:15] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:48:18] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [00:48:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:48:31] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:48:32] ... [00:50:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.58, 3.35 [00:51:55] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.15, 6.64, 5.72 [00:52:08] !log [@mwtask111] starting deploy of {'config': True} to scsvg [00:52:14] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 6s [00:52:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:30] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 4.06, 3.56 [00:52:32] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:34] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [00:52:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.90, 22.51, 23.87 [00:52:57] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:58] ... [00:53:49] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.68, 6.23, 5.67 [00:55:04] !log [@test3] starting deploy of {'config': True} to skip [00:55:05] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:55:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:44] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:45] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [00:56:02] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:56:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.74, 3.54 [00:58:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.85, 4.85, 5.90 [01:00:15] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.12, 5.36, 5.95 [01:00:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.83, 4.27, 3.80 [01:02:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.97, 3.74 [01:04:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.91, 5.59, 6.00 [01:13:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [01:13:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.97, 3.77 [01:14:16] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.36, 5.19, 5.49 [01:15:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:15:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.88, 3.61, 3.66 [01:16:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.70, 4.62, 5.24 [01:16:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.00, 18.62, 20.11 [01:17:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.60, 4.48, 3.98 [01:18:15] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.49, 4.27, 5.04 [01:20:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.69, 19.83, 20.25 [01:21:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.02, 3.91, 3.87 [01:22:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.28, 4.88, 5.09 [01:22:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.44, 19.66, 20.16 [01:23:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 4.06, 3.93 [01:24:15] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.67, 4.50, 4.94 [01:25:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.49, 3.56, 3.77 [01:26:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.24, 21.19, 20.66 [01:29:37] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.90, 4.07, 3.92 [01:32:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.56, 5.13, 5.12 [01:32:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.18, 18.72, 19.84 [01:35:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.80, 3.63, 3.80 [01:38:15] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.83, 4.82, 4.99 [01:39:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.98, 3.93 [01:43:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.64, 3.81 [01:44:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.60, 5.32, 5.07 [01:45:16] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.52, 4.04, 3.94 [01:46:15] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.76, 4.33, 4.73 [01:49:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.70, 3.84 [01:50:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.46, 19.99, 19.30 [01:52:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.02, 19.00, 19.03 [01:53:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.84, 3.85 [01:55:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.83, 3.86 [01:56:59] !log install libmemcached-tools on mem1 [01:57:08] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:57:09] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [01:57:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:57:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.99, 3.92 [01:59:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.95, 3.51, 3.74 [01:59:25] !log install libmemcached-tools on mem2 [01:59:36] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:59:38] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [01:59:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.74, 1.42, 0.96 [02:02:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.61, 1.42, 1.01 [02:12:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:13:01] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.00, 6.52, 5.87 [02:14:59] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.41, 6.55, 5.97 [02:16:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JS9lh [02:16:07] [02miraheze/mw-config] 07paladox 03c90526d - Enable local logging temporarily on the new mediawiki cluster [02:16:08] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [02:16:10] [02mw-config] 07paladox opened pull request 03#4344: Enable local logging temporarily on the new mediawiki cluster - 13https://git.io/JS9lj [02:16:53] [02mw-config] 07paladox closed pull request 03#4344: Enable local logging temporarily on the new mediawiki cluster - 13https://git.io/JS9lj [02:16:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JS98n [02:16:56] [02miraheze/mw-config] 07paladox 032dd6352 - Enable local logging temporarily on the new mediawiki cluster (#4344) [02:16:57] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [02:16:59] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [02:17:21] miraheze/mw-config - paladox the build passed. [02:17:42] !log [@test101] starting deploy of {'config': True} to skip [02:17:43] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [02:17:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:17:56] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:17:57] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [02:17:59] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JS98g [02:18:01] [02miraheze/mw-config] 07paladox 036337d68 - Log memcached locally on the new mediawiki cluster [02:18:07] miraheze/mw-config - paladox the build passed. [02:18:07] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:18:08] ... [02:18:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:18:29] !log [paladox@mwtask111] starting deploy of {'config': True} to scsvg [02:18:35] !log [paladox@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 6s [02:18:38] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:18:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:18:44] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:18:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:03] miraheze/mw-config - paladox the build passed. [02:19:10] !log [paladox@test101] starting deploy of {'config': True} to skip [02:19:11] !log [paladox@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [02:19:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:58] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:20:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:20:17] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:20:18] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [02:21:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [02:22:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.89, 7.09, 6.33 [02:23:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:24:11] !log [@test3] starting deploy of {'config': True} to skip [02:24:12] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [02:24:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:24:38] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:24:39] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [02:24:48] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:24:49] ... [02:24:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:25:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.55, 3.17, 3.37 [02:26:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.05, 6.48, 6.26 [02:34:16] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.81, 5.01, 4.20 [02:38:15] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.24, 4.35, 4.13 [02:40:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.99, 3.48, 3.30 [02:41:49] !log [@mw11] starting deploy of {'config': True} to ovlon [02:41:58] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.43, 3.06, 3.16 [02:41:59] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:16] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [02:42:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.65, 3.35 [02:49:44] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.19, 4.06, 3.53 [02:52:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.13, 6.54, 5.90 [02:54:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.42, 6.43, 5.94 [02:57:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.22, 3.95, 3.75 [02:59:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.92, 4.12, 3.85 [03:03:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.23, 3.75, 3.78 [03:09:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.83, 3.78 [03:11:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.91, 3.82 [03:12:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:17:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.25, 4.17, 3.90 [03:19:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.92, 3.84 [03:22:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.16, 4.60, 3.91 [03:23:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.97, 3.88 [03:24:23] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.99, 3.94, 3.75 [03:35:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:42:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:47:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.85, 3.57, 3.87 [03:53:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.91, 3.78, 3.82 [03:57:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.90, 3.61, 3.76 [04:07:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:11:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.45, 2.90, 3.39 [04:13:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:15:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.41, 3.51 [04:21:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.69, 3.61 [04:31:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.97, 3.87 [04:35:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.20, 2.82, 3.40 [04:38:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:41:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.00, 3.28, 3.43 [04:44:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:45:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.56, 3.50 [04:51:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.96, 3.72 [05:13:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.34, 3.91, 3.69 [05:14:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:15:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.65, 3.63 [05:25:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:29:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.06, 3.10, 3.37 [05:51:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.34, 3.18 [05:53:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.40, 3.37, 3.21 [06:08:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.55, 4.05, 3.51 [06:10:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.68, 3.44 [06:12:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.81, 3.51 [06:12:16] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [06:13:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.94, 6.72, 5.74 [06:14:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.61, 3.47 [06:14:16] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:15:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.11, 6.55, 5.80 [06:16:00] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.92, 3.21, 3.33 [06:24:43] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.93, 4.19, 3.66 [06:30:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.14, 3.45, 3.56 [06:35:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:38:24] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.32, 3.21, 3.39 [06:46:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:49:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 3.64, 3.39 [06:51:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:53:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.53, 3.42 [06:55:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.09, 3.39, 3.38 [06:57:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:59:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.69, 3.52 [07:07:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 3.00, 3.27 [07:07:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:19:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:34:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:50:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:00:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:04:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:09:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:13:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.46, 6.70, 5.53 [08:13:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:13:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.08, 7.60, 5.89 [08:15:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.43, 6.14, 5.46 [08:15:39] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.60, 7.17, 5.94 [08:17:36] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.79, 6.70, 5.92 [08:33:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:48:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:58:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:21:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:36:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:40:57] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JSHfa [09:40:58] [02miraheze/dns] 07Reception123 0367409ac - add encyclopediarobotica.org zone [09:41:20] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JSHfr [09:41:22] [02miraheze/dns] 07Reception123 03a91b99a - add wiki.pbptanarchy.tk zone [09:41:48] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JSHfo [09:41:49] [02miraheze/dns] 07Reception123 0383d5d57 - add wikifencing.ru zone [10:09:36] [02miraheze/ssl] 07Reception123 pushed 036 commits to 03master [+6/-0/±6] 13https://git.io/JSHUy [10:09:38] [02miraheze/ssl] 07Reception123 030088ba6 - add encyclopediarobotica.org cert [10:09:39] [02miraheze/ssl] 07Reception123 039448577 - add wiki.pbptanarchy.tk cert [10:09:41] [02miraheze/ssl] 07Reception123 03527621f - add wikifencing.ru cert [10:09:42] [02miraheze/ssl] ... and 3 more commits. [10:20:29] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26707 MB (5% inode=97%); [10:30:10] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 27690 MB (6% inode=97%); [11:13:30] [02ssl] 07Reception123 closed pull request 03#464: Remove egoish.org - 13https://git.io/JSSBK [11:13:32] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JSHIA [11:13:33] [02miraheze/ssl] 07Universal-Omega 03ac8ea3d - Remove egoish.org (#464) [11:13:35] [02ssl] 07Reception123 closed pull request 03#463: Remove dreamsit.com.br - 13https://git.io/JSups [11:13:36] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JSHIx [11:13:38] [02miraheze/ssl] 07Universal-Omega 03d94fff9 - Remove dreamsit.com.br (#463) [11:13:50] [02ssl] 07Reception123 closed pull request 03#462: Remove explicit meta.orain.org redirect - 13https://git.io/JStWP [11:13:52] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSHIh [11:13:53] [02miraheze/ssl] 07Universal-Omega 03ed8a693 - Remove explicit meta.orain.org redirect (#462) [11:16:44] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildtextindex.php --wiki=subcultureswiki --update (END - exit=256) [11:16:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:16:48] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:16:49] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [11:22:53] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[dreamsit.com.br],File[egoish.org] [11:23:43] PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[dreamsit.com.br],File[egoish.org] [11:24:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:29:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:32:37] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wikicryptoswiki --username-prefix w --report 1 --no-updates /home/reception/Wikipedia-cryptos.xml (START) [11:32:38] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26705 MB (5% inode=97%); [11:32:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:34:17] PROBLEM - db13 Disk Space on db13 is CRITICAL: DISK CRITICAL - free space: / 26680 MB (5% inode=97%); [11:35:19] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSHLw [11:35:21] [02miraheze/dns] 07Reception123 030522592 - add MX/SPF records to wikicryptos.org per T8553 [11:36:13] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 27447 MB (6% inode=97%); [11:37:49] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wikicryptoswiki --username-prefix w --report 1 --no-updates /home/reception/Wikipedia-cryptos.xml (END - exit=0) [11:37:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:37:53] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:37:54] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [11:37:56] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=lgballtwiki --username-prefix wikia:lgballt --report 1 --no-updates /home/reception/lgballt_pages_full.xml (START) [11:37:59] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:38:00] ... [11:38:24] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=wikicryptoswiki (START) [11:38:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:38:36] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=lgballtwiki --username-prefix wikia:lgballt --report 1 --no-updates /home/reception/lgballt_pages_full.xml (END - exit=0) [11:38:39] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:41:15] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=wikicryptoswiki (END - exit=0) [11:41:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:41:19] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:41:20] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [11:41:55] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=edhelwiki --username-prefix edhel --report 1 --no-updates /home/reception/edhelwiki.xml (START) [11:41:59] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:42:00] ... [11:42:58] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=lgballtwiki (START) [11:43:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:43:03] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:43:25] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=lgballtwiki (END - exit=0) [11:43:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:43:32] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:46:13] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 27664 MB (6% inode=97%); [11:47:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:47:57] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=edhelwiki --username-prefix edhel --report 1 --no-updates /home/reception/edhelwiki.xml (END - exit=0) [11:48:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:00] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:03] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [11:49:20] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [11:50:03] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [12:06:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.11, 6.43, 5.56 [12:06:43] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSHtR [12:06:44] [02miraheze/landing] 07translatewiki 030b6d78c - Localisation updates from https://translatewiki.net. [12:06:45] [url] Main page - translatewiki.net | translatewiki.net [12:06:46] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JSHt0 [12:06:48] [02miraheze/CreateWiki] 07translatewiki 035698d35 - Localisation updates from https://translatewiki.net. [12:06:48] [url] Main page - translatewiki.net | translatewiki.net [12:06:49] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-1/±9] 13https://git.io/JSHtE [12:06:51] [02miraheze/MirahezeMagic] 07translatewiki 03d96e751 - Localisation updates from https://translatewiki.net. [12:06:51] [url] Main page - translatewiki.net | translatewiki.net [12:07:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:09:32] miraheze/landing - translatewiki the build passed. [12:09:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.79, 7.12, 5.79 [12:10:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.40, 5.70, 5.54 [12:11:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.86, 6.34, 5.14 [12:12:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [12:12:19] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:12:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:12:42] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:13:09] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:29] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.61, 7.90, 5.93 [12:13:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [12:13:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.51, 7.36, 5.67 [12:13:38] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:13:44] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.40, 6.56, 5.87 [12:13:51] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:14:16] miraheze/CreateWiki - translatewiki the build passed. [12:14:35] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:14:35] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.63, 6.32, 5.52 [12:14:39] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:14:48] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:15:01] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:16:22] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.319 second response time [12:16:32] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.23, 7.56, 6.06 [12:16:34] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.308 second response time [12:16:34] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.008 second response time [12:16:46] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.012 second response time [12:16:50] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.331 second response time [12:17:01] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.208 second response time [12:17:16] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.466 second response time [12:17:21] miraheze/MirahezeMagic - translatewiki the build passed. [12:17:24] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.33, 7.84, 6.38 [12:17:25] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw12 [12:17:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.64, 7.12, 6.02 [12:17:41] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.022 second response time [12:17:50] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.329 second response time [12:17:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.94, 7.53, 6.50 [12:18:30] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.56, 7.46, 6.23 [12:18:34] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:18:46] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:19:22] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.31, 6.50, 6.06 [12:19:26] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [12:19:52] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:20:27] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.39, 7.65, 6.43 [12:20:30] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.333 second response time [12:20:45] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23663 bytes in 0.359 second response time [12:21:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.32, 7.11, 6.23 [12:21:59] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:21:59] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:22:21] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:22:24] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:22:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.91, 7.90, 6.68 [12:22:38] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:22:40] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:40] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:22:43] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:22:47] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:23:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.44, 7.21, 6.16 [12:23:15] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.55, 7.50, 6.68 [12:23:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [12:23:44] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.51, 7.36, 6.72 [12:23:55] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.024 second response time [12:23:57] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.905 second response time [12:24:04] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:24] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.45, 8.06, 6.87 [12:24:37] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.177 second response time [12:24:39] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.228 second response time [12:24:41] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 1.928 second response time [12:25:01] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:13] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:25:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.86, 7.42, 6.56 [12:25:17] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:25:22] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:25:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.86, 7.72, 6.68 [12:25:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.70, 7.71, 6.95 [12:26:22] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.81, 7.90, 6.96 [12:26:49] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:27:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.29, 7.50, 6.83 [12:27:22] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.043 second response time [12:27:24] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.957 second response time [12:27:33] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.42, 8.16, 7.21 [12:28:08] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.026 second response time [12:28:26] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.009 second response time [12:28:38] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.011 second response time [12:28:44] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.031 second response time [12:28:48] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.558 second response time [12:28:58] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.343 second response time [12:29:00] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.324 second response time [12:29:07] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.94, 6.96, 6.71 [12:29:19] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.013 second response time [12:29:25] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.323 second response time [12:29:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:29:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:30:17] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.32, 6.20, 6.46 [12:31:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.40, 6.72, 6.65 [12:31:07] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.02, 6.23, 6.36 [12:31:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.40, 6.84, 6.54 [12:31:22] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.90, 7.50, 7.13 [12:31:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.13, 6.37, 6.44 [12:33:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.72, 6.26, 6.37 [12:33:17] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.14, 6.31, 6.74 [12:41:59] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.22, 6.62, 6.37 [12:44:52] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:45:02] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:45:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.61, 8.43, 7.34 [12:45:07] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:45:58] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:51] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.399 second response time [12:47:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.19, 7.28, 6.58 [12:47:07] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.391 second response time [12:47:08] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.415 second response time [12:47:20] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw9 [12:47:25] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw9 [12:47:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [12:47:52] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.99, 6.73, 6.57 [12:48:02] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 4.949 second response time [12:48:26] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw9 [12:48:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [12:49:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.40, 6.63, 6.44 [12:49:20] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [12:49:25] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [12:49:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:49:55] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.59, 7.07, 6.22 [12:50:25] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [12:51:52] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.90, 7.33, 6.42 [12:53:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.79, 7.76, 7.63 [12:54:39] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.69, 7.52, 6.98 [12:55:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.46, 8.64, 7.98 [12:55:44] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.67, 6.35, 6.23 [12:58:18] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:58:25] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:58:32] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:58:33] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:58:35] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.15, 8.18, 7.34 [12:58:38] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:00:15] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.367 second response time [13:00:22] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.323 second response time [13:00:28] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.390 second response time [13:00:29] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.074 second response time [13:00:38] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 0.522 second response time [13:01:36] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.58, 7.15, 6.49 [13:02:30] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.42, 7.81, 7.39 [13:03:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.60, 7.57, 7.89 [13:03:32] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.96, 7.07, 6.52 [13:03:50] PROBLEM - db13 Disk Space on db13 is CRITICAL: DISK CRITICAL - free space: / 26707 MB (5% inode=97%); [13:05:29] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.56, 7.60, 6.78 [13:06:24] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.52, 5.67, 6.62 [13:07:26] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.14, 7.66, 6.92 [13:08:19] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.32, 7.04, 6.40 [13:10:16] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.97, 6.51, 6.27 [13:11:18] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.91, 6.48, 6.64 [13:12:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.98, 6.97, 6.15 [13:14:16] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.12, 6.34, 6.02 [13:16:47] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:16:47] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:17:00] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:08] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.41, 7.65, 7.35 [13:17:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:18:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.41, 6.94, 6.28 [13:18:05] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.65, 8.09, 7.19 [13:18:17] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:18:41] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:18:59] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.460 second response time [13:19:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.83, 7.88, 7.49 [13:19:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:19:56] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.80, 7.31, 6.50 [13:20:02] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.25, 7.77, 7.19 [13:20:13] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.416 second response time [13:20:42] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.243 second response time [13:20:56] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.646 second response time [13:20:57] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.114 second response time [13:21:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.51, 6.48, 6.34 [13:22:00] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.35, 8.60, 7.55 [13:23:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.22, 7.73, 7.50 [13:23:15] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.14, 5.53, 6.02 [13:23:26] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:23:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:23:47] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.94, 6.47, 6.37 [13:24:33] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:25:05] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:25:21] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:25:22] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:27:04] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:27:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.95, 7.40, 7.41 [13:27:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [13:27:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb [13:27:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.10, 7.48, 6.84 [13:27:44] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 9.578 second response time [13:28:21] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:28:25] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:30] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:28:47] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:29:09] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.447 second response time [13:29:17] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.818 second response time [13:29:30] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.244 second response time [13:29:31] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.570 second response time [13:30:19] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.675 second response time [13:30:26] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23674 bytes in 1.899 second response time [13:30:28] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.209 second response time [13:30:38] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.039 second response time [13:30:47] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.924 second response time [13:31:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:31:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:31:48] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.39, 7.89, 7.70 [13:33:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.45, 8.21, 7.68 [13:35:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.42, 7.65, 7.53 [13:39:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.42, 6.63, 6.80 [13:43:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.04, 7.50, 7.40 [13:43:48] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.88, 6.89, 6.28 [13:43:57] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:44:09] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:44:16] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [13:44:30] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:45:20] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:45:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.74, 6.81, 6.22 [13:45:46] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.15, 6.30, 6.14 [13:45:54] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.369 second response time [13:46:06] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.222 second response time [13:46:29] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.759 second response time [13:47:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.49, 7.69, 7.51 [13:47:19] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.362 second response time [13:47:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.53, 7.31, 7.00 [13:49:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.53, 6.55, 6.25 [13:49:42] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.80, 6.91, 6.40 [13:50:43] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:50:48] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:50:50] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:50:52] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:50:57] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:51:02] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:51:22] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.55, 7.27, 7.09 [13:51:41] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.68, 7.53, 6.66 [13:52:44] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:52:45] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.629 second response time [13:52:49] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.878 second response time [13:52:53] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.837 second response time [13:53:02] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.230 second response time [13:53:39] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.16, 7.47, 6.76 [13:55:05] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.367 second response time [13:56:10] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:56:31] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [13:56:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.95, 3.43, 3.19 [13:57:05] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:10] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:57:14] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:57:25] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.43, 7.97, 7.43 [13:57:27] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:57:36] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.64, 6.32, 6.46 [13:58:09] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.671 second response time [13:58:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.97, 3.38, 3.21 [13:59:05] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.559 second response time [13:59:10] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.007 second response time [13:59:12] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.06, 7.39, 7.37 [13:59:13] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.713 second response time [13:59:24] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.537 second response time [14:00:18] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:01:56] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.03, 6.77, 6.44 [14:03:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.04, 7.38, 7.41 [14:05:49] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.59, 7.88, 6.93 [14:09:41] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.67, 7.63, 7.09 [14:10:21] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:10:24] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:10:46] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.88, 7.11, 7.14 [14:11:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.96, 7.33, 7.28 [14:11:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.09, 7.54, 7.27 [14:11:07] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:11:41] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:12:38] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:12:52] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 6.293 second response time [14:12:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.38, 7.39, 7.26 [14:13:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.96, 6.78, 7.12 [14:13:11] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.945 second response time [14:13:42] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.158 second response time [14:15:08] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:15:18] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:15:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.97, 7.65, 6.77 [14:15:34] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.76, 6.84, 6.50 [14:15:35] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:15:44] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:15:49] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:56] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:16:20] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:16:34] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:38] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.154 second response time [14:16:39] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.359 second response time [14:16:40] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:16:44] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:16:45] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:17:12] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:17:13] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.012 second response time [14:17:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:17:19] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:17:32] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23663 bytes in 0.268 second response time [14:18:07] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:18:40] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 9.943 second response time [14:18:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.45, 7.89, 7.53 [14:19:14] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.316 second response time [14:19:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.22, 6.52, 6.48 [14:19:42] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.115 second response time [14:19:48] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.816 second response time [14:19:54] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 6.436 second response time [14:19:56] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.310 second response time [14:20:40] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.319 second response time [14:20:49] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.010 second response time [14:20:50] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.311 second response time [14:20:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.75, 6.99, 7.27 [14:21:14] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.035 second response time [14:21:18] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.621 second response time [14:21:21] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 0.308 second response time [14:22:10] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.032 second response time [14:23:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.10, 5.88, 6.64 [14:23:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.95, 6.79, 6.58 [14:24:01] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:24:02] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:24:41] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.147 second response time [14:25:16] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.33, 7.48, 6.95 [14:25:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.52, 5.87, 6.27 [14:25:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSH3y [14:25:54] [02miraheze/puppet] 07paladox 03d89c6f1 - mariadb: Grant REPLICA MONITOR to exporter on bullseye+ [14:25:55] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [14:25:57] [02puppet] 07paladox opened pull request 03#2263: mariadb: Grant REPLICA MONITOR to exporter on bullseye+ - 13https://git.io/JSH3S [14:25:57] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.021 second response time [14:25:59] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.363 second response time [14:26:35] [02puppet] 07paladox edited pull request 03#2263: mariadb: Grant REPLICA MONITOR to exporter on bullseye+ - 13https://git.io/JSH3S [14:27:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.08, 6.89, 6.90 [14:27:11] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:27:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.28, 6.84, 6.80 [14:27:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.28, 6.21, 6.74 [14:27:15] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:27:19] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:27:30] [02puppet] 07paladox closed pull request 03#2263: mariadb: Grant REPLICA MONITOR to exporter on bullseye+ - 13https://git.io/JSH3S [14:27:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSH3Q [14:27:33] [02miraheze/puppet] 07paladox 038cd23fa - mariadb: Grant REPLICA MONITOR to exporter on bullseye+ (#2263) [14:27:34] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [14:27:35] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [14:29:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.59, 7.45, 7.10 [14:29:11] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.199 second response time [14:29:11] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.71, 5.77, 6.41 [14:29:15] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.258 second response time [14:29:22] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.008 second response time [14:29:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:29:41] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:31:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.46, 7.11, 7.02 [14:33:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.25, 6.17, 6.69 [14:34:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.17, 7.75, 7.27 [14:35:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.00, 7.15, 6.68 [14:36:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.50, 6.75, 6.95 [14:37:00] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.57, 7.30, 6.80 [14:38:10] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.18, 7.10, 6.98 [14:39:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.36, 6.97, 7.84 [14:40:06] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.44, 7.44, 7.11 [14:41:30] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:41:48] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:42:03] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.61, 6.92, 6.97 [14:42:11] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [14:42:16] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:42:24] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:42:54] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.77, 6.62, 6.71 [14:42:54] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:43:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.37, 6.94, 6.67 [14:43:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.84, 7.84, 7.95 [14:43:23] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:43:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSHsz [14:43:47] [02miraheze/puppet] 07paladox 03d535379 - prometheus::redis_export: Fix permissions and update systemd unit [14:43:49] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [14:43:50] [02puppet] 07paladox opened pull request 03#2264: prometheus::redis_export: Fix permissions and update systemd unit - 13https://git.io/JSHsg [14:43:59] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.88, 6.22, 6.71 [14:44:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:45:00] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.680 second response time [14:45:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.09, 7.88, 7.04 [14:45:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.09, 7.34, 7.75 [14:45:57] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.590 second response time [14:46:26] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 2.773 second response time [14:46:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.07, 7.39, 7.14 [14:47:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.49, 7.60, 7.06 [14:47:26] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.026 second response time [14:47:41] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.316 second response time [14:47:51] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:47:54] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.75, 7.86, 7.27 [14:48:05] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:48:07] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:48:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [14:48:36] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:48:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.85, 7.17, 7.09 [14:50:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:50:59] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.07, 7.69, 7.27 [14:51:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.01, 7.48, 7.07 [14:51:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.85, 7.50, 7.65 [14:51:47] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.32, 6.89, 7.07 [14:51:51] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.008 second response time [14:52:11] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.012 second response time [14:52:11] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.357 second response time [14:52:39] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.328 second response time [14:52:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.96, 7.59, 7.28 [14:53:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:53:44] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 2.95, 5.57, 6.57 [14:53:47] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:54:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:54:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.99, 7.94, 7.44 [14:55:01] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.56, 7.74, 7.42 [14:57:00] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 0.356 second response time [14:57:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.85, 7.32, 7.24 [14:57:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSHsb [14:57:41] [02miraheze/puppet] 07paladox 03d0de877 - Update prometheus-redis-exporter.systemd.erb [14:57:43] [02puppet] 07paladox synchronize pull request 03#2264: prometheus::redis_export: Fix permissions and update systemd unit - 13https://git.io/JSHsg [14:57:44] [02puppet] 07paladox synchronize pull request 03#2264: prometheus::redis_export: Fix permissions and update systemd unit - 13https://git.io/JSHsg [14:57:50] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.220 second response time [14:57:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:58:59] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.56, 8.73, 7.82 [14:59:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.87, 7.92, 7.91 [15:00:32] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.94, 6.78, 6.15 [15:01:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.57, 7.19, 7.15 [15:01:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSHGI [15:01:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 7.54, 8.02, 7.96 [15:01:07] [02miraheze/puppet] 07paladox 0394ecf08 - Update redis_exporter.pp [15:01:09] [02puppet] 07paladox synchronize pull request 03#2264: prometheus::redis_export: Fix permissions and update systemd unit - 13https://git.io/JSHsg [15:01:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [15:01:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSHGL [15:01:45] [02miraheze/puppet] 07paladox 0379fcb7f - Update prometheus-redis-exporter.systemd.erb [15:01:47] [02puppet] 07paladox synchronize pull request 03#2264: prometheus::redis_export: Fix permissions and update systemd unit - 13https://git.io/JSHsg [15:01:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [15:02:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSHGt [15:02:15] [02miraheze/puppet] 07paladox 03f89f123 - Update redis_exporter.pp [15:02:16] [02puppet] 07paladox synchronize pull request 03#2264: prometheus::redis_export: Fix permissions and update systemd unit - 13https://git.io/JSHsg [15:02:34] [02puppet] 07paladox closed pull request 03#2264: prometheus::redis_export: Fix permissions and update systemd unit - 13https://git.io/JSHsg [15:02:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JSHGq [15:02:37] [02miraheze/puppet] 07paladox 0371223a8 - prometheus::redis_export: Fix permissions and update systemd unit (#2264) [15:02:38] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [15:02:40] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [15:02:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.84, 7.65, 7.65 [15:03:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.10, 6.69, 6.28 [15:03:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.49, 7.37, 7.72 [15:03:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:03:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.48, 3.04 [15:03:42] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:46] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:03:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:03:56] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:04:07] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:04:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.50, 7.47, 6.61 [15:05:01] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:05:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.63, 6.78, 7.07 [15:05:03] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.65, 6.76, 6.37 [15:05:10] PROBLEM - mw121 JobRunner Service on mw121 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [15:05:10] PROBLEM - mw112 JobRunner Service on mw112 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [15:05:16] PROBLEM - mw102 JobRunner Service on mw102 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [15:05:27] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.40, 7.51, 6.95 [15:05:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.34, 3.25, 3.00 [15:06:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSHGW [15:06:14] [02miraheze/puppet] 07paladox 0316097c8 - prometheus::redis_exporter: Update exporter to 1.33.0 [15:06:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.64, 8.01, 6.92 [15:06:34] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [15:06:38] PROBLEM - mw122 JobRunner Service on mw122 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [15:06:39] PROBLEM - mw101 JobRunner Service on mw101 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [15:06:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.60, 7.51, 7.51 [15:07:05] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.815 second response time [15:07:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:07:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:08:06] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.490 second response time [15:08:12] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.507 second response time [15:08:23] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.84, 7.88, 7.00 [15:08:41] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.34, 7.11, 7.35 [15:09:00] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.71, 7.54, 6.79 [15:09:20] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.14, 7.55, 7.07 [15:09:30] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:09:34] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:38] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:09:50] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:09:50] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:50] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.451 second response time [15:09:56] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 2.639 second response time [15:10:13] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:10:20] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 7.65, 8.05, 7.18 [15:10:41] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.726 second response time [15:11:08] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.31, 7.90, 7.66 [15:11:08] RECOVERY - mw121 JobRunner Service on mw121 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [15:11:16] RECOVERY - mw102 JobRunner Service on mw102 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [15:11:16] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.90, 7.56, 7.13 [15:11:29] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.310 second response time [15:11:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:11:32] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.265 second response time [15:11:33] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.014 second response time [15:11:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:11:44] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.178 second response time [15:11:44] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.311 second response time [15:12:12] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.432 second response time [15:12:18] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.85, 6.58, 6.74 [15:12:52] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:12:57] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.08, 6.29, 6.45 [15:14:48] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.328 second response time [15:15:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.19, 7.67, 7.24 [15:15:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.25, 6.41, 6.75 [15:15:16] We should probably clean up the duplicate entries on the SAL if somebody gets a chance [15:15:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:15:34] (From having two logbots) [15:15:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:15:44] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:16:42] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:17:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.71, 7.01, 7.05 [15:17:19] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:17:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:17:50] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.292 second response time [15:18:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.22, 6.84, 6.85 [15:18:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.43, 3.20 [15:18:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.13, 7.31, 7.14 [15:19:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.04, 7.33, 7.16 [15:19:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.34, 7.96, 7.90 [15:19:10] RECOVERY - mw112 JobRunner Service on mw112 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [15:19:17] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:19:18] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:20:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.85, 3.17, 3.13 [15:20:07] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.61, 6.35, 6.66 [15:20:45] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.334 second response time [15:20:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.43, 6.47, 6.87 [15:21:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.26, 7.35, 7.19 [15:21:16] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.574 second response time [15:21:19] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 2.704 second response time [15:21:21] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.109 second response time [15:21:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:21:38] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:34] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [15:22:43] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:23:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.78, 7.85, 7.19 [15:23:36] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23663 bytes in 0.263 second response time [15:24:01] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.99, 6.82, 6.79 [15:24:44] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 2.878 second response time [15:25:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.29, 7.48, 7.70 [15:25:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.77, 7.43, 7.13 [15:25:58] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.23, 6.28, 6.60 [15:26:13] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26699 MB (5% inode=97%); [15:26:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.24, 7.11, 6.99 [15:27:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.95, 7.28, 7.60 [15:27:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.04, 8.21, 7.46 [15:28:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.18, 6.96, 6.97 [15:29:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:29:48] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:29:52] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:21] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:39] RECOVERY - mw101 JobRunner Service on mw101 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [15:30:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.17, 5.84, 6.55 [15:31:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.87, 5.95, 6.70 [15:32:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.28, 3.15 [15:33:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:33:47] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.017 second response time [15:33:48] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.56, 6.82, 6.64 [15:33:59] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.318 second response time [15:34:30] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.253 second response time [15:34:38] RECOVERY - mw122 JobRunner Service on mw122 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [15:34:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.95, 3.07, 3.09 [15:35:00] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:35:14] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:35:23] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:35:29] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:35:46] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.49, 7.12, 6.75 [15:35:50] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:36:13] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:36:15] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:37:03] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.66, 7.80, 7.20 [15:37:10] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:37:15] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.85, 7.64, 7.34 [15:37:19] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.86, 7.72, 7.62 [15:39:05] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.000 second response time [15:39:07] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.007 second response time [15:39:10] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.40, 7.60, 7.40 [15:39:20] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:22] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw10 [15:39:24] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.636 second response time [15:39:26] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.214 second response time [15:39:26] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:34] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.333 second response time [15:39:35] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:36] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:40] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:40] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:42] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.42, 6.91, 6.77 [15:39:43] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:43] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:47] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:40:02] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 3.581 second response time [15:40:13] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.097 second response time [15:40:22] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.553 second response time [15:41:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.07, 7.44, 7.25 [15:41:19] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.119 second response time [15:41:20] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [15:41:32] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.286 second response time [15:41:38] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.894 second response time [15:41:41] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 2.681 second response time [15:41:41] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.158 second response time [15:41:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.40, 7.32, 6.93 [15:41:42] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.418 second response time [15:42:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.34, 3.12 [15:42:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.71, 7.18, 6.88 [15:43:30] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.238 second response time [15:43:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.09, 6.94, 6.85 [15:43:47] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 1.277 second response time [15:43:48] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.788 second response time [15:44:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.34, 2.99, 3.03 [15:45:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:45:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:45:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.88, 6.27, 6.62 [15:46:13] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 26782 MB (6% inode=97%); [15:46:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.27, 6.50, 6.71 [15:47:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.75, 5.98, 6.76 [15:48:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.61, 3.29 [15:49:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.29, 7.56, 7.37 [15:49:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.70, 7.56, 7.47 [15:50:12] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:50:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.21, 3.36, 3.23 [15:50:35] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.77, 8.39, 7.40 [15:51:01] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:51:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.45, 7.06, 7.20 [15:51:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.49, 7.31, 7.40 [15:51:19] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:51:32] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:51:32] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:51:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:51:47] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:51:55] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:52:59] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.877 second response time [15:53:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.67, 7.71, 7.26 [15:53:18] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 0.327 second response time [15:53:27] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.334 second response time [15:53:28] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.049 second response time [15:53:48] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.728 second response time [15:53:53] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.704 second response time [15:54:16] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.245 second response time [15:55:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.80, 7.72, 7.32 [15:56:27] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.67, 7.78, 7.48 [15:56:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.14, 3.42, 3.31 [15:57:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.11, 8.28, 7.57 [15:57:15] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.87, 5.98, 6.79 [15:57:23] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:57:26] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:57:32] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:36] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:58:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 7.78, 8.08, 7.64 [15:58:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.24, 3.29, 3.27 [15:58:44] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:59:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.56, 6.30, 6.77 [15:59:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.59, 7.76, 7.45 [15:59:13] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:19] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:59:21] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 3.033 second response time [15:59:31] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23663 bytes in 0.785 second response time [15:59:34] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.347 second response time [16:00:22] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.84, 7.73, 7.56 [16:00:41] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.339 second response time [16:01:13] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.578 second response time [16:01:14] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.300 second response time [16:01:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:01:33] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.715 second response time [16:03:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.72, 6.78, 6.90 [16:03:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:04:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.42, 3.29 [16:06:13] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26702 MB (5% inode=97%); [16:07:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [16:07:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [16:08:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.25, 3.39, 3.30 [16:09:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.89, 5.84, 6.58 [16:09:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.84, 7.80, 7.27 [16:09:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:10:11] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.01, 7.73, 7.44 [16:10:18] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.02, 6.62, 6.21 [16:10:32] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:11:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.62, 7.78, 7.33 [16:12:08] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.36, 7.61, 7.44 [16:12:16] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.47, 6.55, 6.24 [16:12:25] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.69, 6.91, 6.36 [16:12:36] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.314 second response time [16:12:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:13:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [16:13:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:17:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:18:01] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.61, 7.10, 7.20 [16:18:18] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.93, 7.58, 6.78 [16:19:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.89, 7.19, 7.21 [16:20:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.00, 7.24, 6.75 [16:21:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.05, 6.77, 6.89 [16:21:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [16:23:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.77, 6.75, 6.89 [16:24:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2607:5300:201:3100::929a/cpweb [16:25:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.23, 7.36, 7.23 [16:26:10] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.56, 6.28, 6.55 [16:26:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:27:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.16, 7.24, 7.04 [16:27:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:27:48] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.91, 7.25, 7.21 [16:29:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.86, 7.37, 7.10 [16:29:47] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.33, 7.98, 7.46 [16:31:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [16:32:16] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:33:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.25, 7.20, 6.71 [16:35:08] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.78, 7.56, 6.88 [16:35:15] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.33, 6.29, 6.70 [16:35:42] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:35:47] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:36:07] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:36:10] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:36:24] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/JSHnb [16:36:26] [02miraheze/puppet] 07JohnFLewis 031e945ed - es: push modified old code [16:37:03] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.34, 7.57, 6.97 [16:37:37] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.327 second response time [16:37:38] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.87, 7.84, 7.76 [16:37:46] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 0.257 second response time [16:37:53] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.71, 7.64, 7.02 [16:38:03] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.005 second response time [16:38:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [16:38:07] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.459 second response time [16:38:27] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.197 second response time [16:39:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.96, 6.39, 6.62 [16:39:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.93, 7.03, 6.95 [16:39:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.27, 3.95, 3.39 [16:39:51] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.18, 7.27, 6.95 [16:40:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:40:22] PROBLEM - es121 Puppet on es121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:40:25] PROBLEM - es111 Puppet on es111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:41:00] PROBLEM - es101 Puppet on es101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:41:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.03, 3.59, 3.33 [16:41:48] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.19, 8.51, 7.43 [16:42:51] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.01, 6.74, 6.16 [16:43:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.09, 3.10, 3.18 [16:43:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.86, 6.58, 6.79 [16:43:31] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.81, 7.14, 7.33 [16:43:46] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.43, 7.77, 7.30 [16:44:50] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.38, 6.22, 6.04 [16:45:01] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:45:14] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:45:17] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:45:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.15, 7.39, 6.65 [16:45:50] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:45:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [16:46:49] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:47:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.96, 7.08, 6.70 [16:47:04] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 7.512 second response time [16:47:16] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 7.498 second response time [16:47:20] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 5.778 second response time [16:47:22] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.72, 6.75, 6.50 [16:49:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.42, 7.00, 6.71 [16:49:14] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:49:23] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.04, 7.74, 7.60 [16:49:32] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:50:08] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:50:17] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:50:46] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:50:46] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:50:58] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.799 second response time [16:51:12] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.985 second response time [16:52:09] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.135 second response time [16:52:19] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23663 bytes in 4.279 second response time [16:52:41] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.789 second response time [16:52:53] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.942 second response time [16:53:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:53:37] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:53:40] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 0.407 second response time [16:53:59] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.381 second response time [16:55:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.40, 6.55, 6.66 [16:57:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [16:58:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.11, 6.19, 5.79 [16:58:49] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.79, 7.20, 6.80 [16:59:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:00:06] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:17] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:00:25] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:00:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.75, 7.05, 6.12 [17:00:46] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.71, 7.62, 6.99 [17:01:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.82, 7.15, 7.04 [17:02:08] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 2.944 second response time [17:02:15] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.375 second response time [17:02:26] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.146 second response time [17:03:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [17:03:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [17:03:38] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:03:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.16, 7.04, 7.04 [17:03:50] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:03:51] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:04:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.22, 7.17, 6.40 [17:05:40] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.546 second response time [17:05:47] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.007 second response time [17:05:48] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.318 second response time [17:06:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.89, 7.22, 7.13 [17:06:33] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.65, 8.04, 6.81 [17:07:15] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:07:17] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:07:18] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:07:59] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.49, 7.38, 6.79 [17:08:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.86, 7.78, 6.88 [17:09:27] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JSHC7 [17:09:28] [02miraheze/puppet] 07JohnFLewis 03d39f830 - add missing ES file + java compat [17:09:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.67, 7.59, 7.20 [17:10:18] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.18, 6.06, 6.69 [17:10:59] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.79, 5.93, 6.74 [17:11:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.45, 7.03, 7.05 [17:12:04] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:07] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:12:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.67, 3.34 [17:12:28] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.90, 8.34, 7.25 [17:13:33] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.509 second response time [17:13:37] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.528 second response time [17:13:40] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.115 second response time [17:13:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.63, 7.46, 7.18 [17:13:47] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:13:48] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.97, 8.00, 7.33 [17:14:01] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 2.693 second response time [17:14:07] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.649 second response time [17:14:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.77, 3.41 [17:15:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:15:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:15:42] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.35, 7.03, 7.06 [17:15:51] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.764 second response time [17:16:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.39, 3.31 [17:16:25] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.28, 7.69, 7.23 [17:18:06] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:18:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.66, 6.81, 6.75 [17:19:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [17:19:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [17:19:38] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.95, 5.96, 6.68 [17:19:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.84, 6.06, 6.73 [17:19:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.50, 7.33, 6.78 [17:20:05] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.319 second response time [17:20:10] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.15, 5.88, 6.62 [17:20:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.45, 7.34, 6.95 [17:21:10] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:21:14] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:22:11] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:22:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.24, 7.22, 6.95 [17:23:07] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.322 second response time [17:23:10] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.007 second response time [17:23:40] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.64, 6.89, 6.78 [17:24:08] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.336 second response time [17:24:17] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.606 second response time [17:25:29] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.92, 6.62, 6.68 [17:25:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:25:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:26:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.54, 3.35 [17:27:26] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.94, 6.34, 6.56 [17:27:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.35, 7.28, 6.95 [17:28:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.35, 3.70, 3.43 [17:28:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.96, 6.28, 6.64 [17:29:13] [02miraheze/puppet] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±3] 13https://git.io/JSHWp [17:29:15] [02miraheze/puppet] 07JohnFLewis 034ec5d6d - es: move ssl to /etc/es path [17:29:16] [02miraheze/puppet] 07JohnFLewis 03bd84954 - es: move ssl to /etc/es path [17:30:18] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSHlt [17:30:20] [02miraheze/puppet] 07JohnFLewis 03b6ec3c7 - syntax [17:30:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.96, 3.37, 3.33 [17:33:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.15, 7.02, 6.85 [17:34:13] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb [17:34:14] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.272 second response time [17:34:15] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:34:34] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:34:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.20, 7.03, 6.84 [17:35:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:36:14] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.867 second response time [17:36:17] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 3.083 second response time [17:36:35] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.296 second response time [17:37:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.32, 7.55, 7.42 [17:37:07] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.86, 6.64, 6.30 [17:37:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.94, 6.09, 6.52 [17:37:59] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:38:00] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:38:01] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:38:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.62, 6.76, 6.76 [17:39:06] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.17, 6.66, 6.36 [17:39:56] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.234 second response time [17:39:56] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.005 second response time [17:41:47] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [17:43:07] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:43:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:43:41] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:45:06] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.010 second response time [17:47:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [17:47:37] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:47:48] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.66, 7.67, 7.10 [17:48:57] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.19, 6.56, 6.25 [17:49:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:49:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.06, 7.08, 6.96 [17:51:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.48, 7.42, 7.11 [17:52:20] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:25] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:26] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:26] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:54:52] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.33, 6.12, 6.23 [17:55:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.27, 7.21, 7.09 [17:55:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.98, 6.75, 6.30 [17:57:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:58:07] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.488 second response time [17:58:32] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.308 second response time [17:58:33] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.133 second response time [17:58:39] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.049 second response time [17:58:43] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.266 second response time [17:59:04] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.97, 6.40, 6.30 [17:59:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:01:31] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.64, 7.51, 7.17 [18:02:44] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.03, 6.57, 6.27 [18:03:29] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.51, 6.89, 6.99 [18:04:43] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.47, 6.81, 6.40 [18:05:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.64, 6.51, 6.80 [18:06:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:06:41] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.09, 7.27, 6.62 [18:07:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.21, 7.54, 6.64 [18:07:29] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:07:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [18:08:15] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:08:32] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.35, 7.35, 6.50 [18:08:37] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:08:38] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:08:39] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:08:40] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.39, 7.52, 6.80 [18:09:07] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.11, 7.03, 6.57 [18:09:21] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:09:26] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.153 second response time [18:09:33] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:10:00] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:10:17] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 7.856 second response time [18:11:19] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.55, 6.33, 6.75 [18:11:21] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.104 second response time [18:11:38] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.922 second response time [18:12:24] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.32, 6.87, 6.56 [18:12:37] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.53, 6.67, 6.63 [18:13:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.32, 7.47, 6.83 [18:13:22] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:38] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:39] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:45] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:51] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:53] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:25] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:27] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:37] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:47] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:16:03] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:16:17] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.95, 6.54, 6.52 [18:16:24] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.320 second response time [18:16:32] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 1.180 second response time [18:16:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.35, 7.80, 7.14 [18:16:59] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.227 second response time [18:17:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.21, 7.39, 6.82 [18:18:00] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.077 second response time [18:18:06] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.478 second response time [18:18:56] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.02, 7.73, 7.18 [18:19:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.09, 7.45, 6.93 [18:20:26] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.014 second response time [18:20:51] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:20:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.48, 7.44, 7.15 [18:21:19] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:26] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw12 [18:21:26] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:27] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:38] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:42] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 1.901 second response time [18:21:51] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.959 second response time [18:21:52] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.800 second response time [18:21:59] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:22:02] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.381 second response time [18:22:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.27, 6.45, 6.06 [18:22:27] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:22:56] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.137 second response time [18:23:19] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.647 second response time [18:23:25] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.343 second response time [18:23:26] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [18:23:28] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.443 second response time [18:23:40] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 3.453 second response time [18:24:01] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.785 second response time [18:24:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.68, 6.19, 6.02 [18:24:24] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.272 second response time [18:24:33] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.756 second response time [18:24:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.45, 6.43, 6.28 [18:25:50] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:26:00] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:26:04] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:26:12] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:26:15] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:26:22] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:26:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.90, 7.11, 6.54 [18:27:42] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:27:45] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 8.818 second response time [18:27:46] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:27:49] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 0.506 second response time [18:27:53] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:27:56] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.036 second response time [18:28:08] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:16] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.345 second response time [18:28:22] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.245 second response time [18:28:25] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.503 second response time [18:28:26] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:28:31] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.658 second response time [18:28:31] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.430 second response time [18:28:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.10, 6.67, 6.44 [18:29:40] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.260 second response time [18:29:41] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.465 second response time [18:29:50] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.357 second response time [18:30:07] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.274 second response time [18:30:16] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 8.532 second response time [18:30:23] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.555 second response time [18:30:54] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.68, 7.16, 6.52 [18:31:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.17, 6.34, 6.78 [18:31:17] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.127 second response time [18:31:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:31:39] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:32:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.36, 6.54, 6.39 [18:32:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.39, 7.47, 7.19 [18:32:50] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.15, 6.72, 6.44 [18:34:39] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.38, 7.51, 7.25 [18:35:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [18:35:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:35:50] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.45, 7.20, 6.74 [18:36:37] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.63, 7.65, 7.31 [18:37:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:38:14] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.52, 6.57, 6.52 [18:38:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.21, 7.54, 7.32 [18:40:25] PROBLEM - es111 Puppet on es111 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 17 minutes ago with 1 failures [18:40:42] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.79, 7.88, 6.96 [18:41:00] PROBLEM - es101 Puppet on es101 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 24 minutes ago with 1 failures [18:41:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.72, 6.96, 6.62 [18:41:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [18:41:38] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:41:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.52, 7.16, 6.94 [18:41:56] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:09] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:22] PROBLEM - es121 Puppet on es121 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 29 minutes ago with 1 failures [18:42:28] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:38] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.42, 7.72, 7.02 [18:43:27] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.76, 7.52, 6.86 [18:43:34] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.007 second response time [18:43:51] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.316 second response time [18:44:08] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.353 second response time [18:44:28] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 2.225 second response time [18:45:07] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.92, 7.17, 6.79 [18:45:32] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:38] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:39] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.35, 7.44, 7.03 [18:45:43] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:55] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:45:59] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:46:31] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.85, 6.46, 6.67 [18:47:07] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.40, 8.16, 7.18 [18:47:17] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.22, 6.57, 6.69 [18:47:36] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.05, 7.34, 7.05 [18:48:24] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.14, 6.13, 6.68 [18:49:05] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.23, 7.77, 7.16 [18:49:34] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.23, 7.92, 7.29 [18:50:41] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:51:18] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:51:30] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:51:31] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.64, 7.07, 7.06 [18:51:33] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:51:39] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:51:47] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 4.741 second response time [18:51:48] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 4.290 second response time [18:51:54] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 3.098 second response time [18:52:05] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.036 second response time [18:52:09] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23663 bytes in 0.280 second response time [18:52:42] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.127 second response time [18:53:12] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.012 second response time [18:53:12] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:53:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:53:17] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:27] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:53:27] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.355 second response time [18:53:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:53:32] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.848 second response time [18:53:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:53:35] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.023 second response time [18:53:42] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:54:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.78, 7.33, 6.99 [18:55:12] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.717 second response time [18:55:12] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.182 second response time [18:55:19] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 3.251 second response time [18:55:28] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.585 second response time [18:55:40] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.108 second response time [18:56:59] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.19, 5.98, 6.63 [18:59:47] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.80, 6.51, 6.74 [19:03:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.84, 6.23, 6.73 [19:03:17] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.71, 6.46, 6.78 [19:04:46] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JSQzZ [19:04:47] [02miraheze/puppet] 07JohnFLewis 03323e4a1 - es: cluster config fixes [19:07:10] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.58, 7.36, 7.06 [19:07:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.140.43/cpweb [19:08:16] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:08:23] PROBLEM - es121 Puppet on es121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Elasticsearch_template[graylog-internal] [19:08:28] PROBLEM - es111 Puppet on es111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Elasticsearch_template[graylog-internal] [19:09:00] PROBLEM - es101 Puppet on es101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Elasticsearch_template[graylog-internal] [19:09:08] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.78, 6.62, 6.82 [19:10:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.03, 7.06, 6.68 [19:11:05] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.55, 5.81, 6.49 [19:11:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.53, 3.22 [19:11:46] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:12:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.90, 6.38, 6.49 [19:13:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 3.22, 3.14 [19:13:45] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.36, 6.85, 6.72 [19:14:14] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQPn [19:14:16] [02miraheze/dns] 07JohnFLewis 03f228b40 - add elasticsearch.mh.o [19:15:07] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JSQXC [19:15:09] [02miraheze/puppet] 07JohnFLewis 03db7270d - es: drop -lb [19:15:33] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [19:15:41] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.76, 6.58, 6.62 [19:16:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:17:52] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.36, 6.64, 6.62 [19:18:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQMs [19:18:25] [02miraheze/puppet] 07paladox 034ad9831 - graylog: Only use elasticsearch class on buster [19:19:33] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:19:50] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.44, 7.21, 6.82 [19:20:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.25, 7.21, 6.53 [19:20:20] !log DELETE FROM revision WHERE rev_id='63249'; on templatewiki [19:20:37] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:20:50] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:03] !log sudo -u www-data php /srv/mediawiki/w/maintenance/storage/checkStorage.php --fix /home/universalomega/wikibackups/templatewiki.xml --wiki=templatewiki [19:21:03] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:10] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:21:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.41, 6.99, 6.73 [19:21:29] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:21:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:21:30] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [19:21:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.57, 7.30, 6.90 [19:21:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:21:59] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:06] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [19:22:34] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.777 second response time [19:22:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQDh [19:22:39] [02miraheze/puppet] 07paladox 038d7ef8e - Install graylog121 graylog class [19:22:46] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.054 second response time [19:23:01] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.107 second response time [19:23:07] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.011 second response time [19:23:20] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.29, 6.76, 6.68 [19:23:42] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 1.907 second response time [19:25:00] RECOVERY - es101 Puppet on es101 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [19:25:19] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQyX [19:25:21] [02miraheze/puppet] 07JohnFLewis 0330fbff4 - es: overwrite defaults with miraheze specifics [19:25:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:25:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.60, 7.55, 7.07 [19:25:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.65, 7.55, 6.85 [19:25:59] RECOVERY - mwtask1 Disk Space on mwtask1 is OK: DISK OK - free space: / 16393 MB (35% inode=89%); [19:27:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.57, 7.15, 6.76 [19:28:00] PROBLEM - graylog121 Puppet on graylog121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 20691EEC35216C63CAF66CE1656408E390CFB1F5 from Apt::Source mongodb] [19:29:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.16, 7.09, 6.82 [19:29:14] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:29:22] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:29:51] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:29:58] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:30:15] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:31] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:30:51] !log ALTER TABLE searchindex ADD FULLTEXT si_title (si_title); on subcultureswiki [19:31:09] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.040 second response time [19:31:18] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.486 second response time [19:31:19] !log ALTER TABLE searchindex ADD FULLTEXT si_text (si_text); on subcultureswiki [19:31:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:31:39] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:31:50] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 1.855 second response time [19:31:51] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [19:31:55] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 1.084 second response time [19:31:57] PROBLEM - graylog121 HTTPS on graylog121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.010 second response time [19:32:19] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 4.869 second response time [19:32:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:28] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:30] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23663 bytes in 0.925 second response time [19:32:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQSe [19:32:33] [02miraheze/puppet] 07paladox 0321471d7 - Update graylog121.yaml [19:32:35] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [19:34:15] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Es_instance_conn_validator[graylog-internal-template],Elasticsearch_template[graylog-internal] [19:34:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:37:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.17, 8.31, 7.38 [19:37:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.92, 6.73, 6.24 [19:37:27] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.70, 7.88, 7.55 [19:37:35] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:38:28] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb [19:38:54] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:39:02] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:39:07] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:39:08] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:39:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.71, 7.46, 6.55 [19:39:15] [02puppet] 07Universal-Omega opened pull request 03#2265: CSP: add `'blob:'` to `img-src` - 13https://git.io/JSQSY [19:39:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.30, 3.05 [19:39:33] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 2.473 second response time [19:40:25] [02puppet] 07Reception123 closed pull request 03#2265: CSP: add `'blob:'` to `img-src` - 13https://git.io/JSQSY [19:40:27] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQSG [19:40:28] [02miraheze/puppet] 07Universal-Omega 03447931d - CSP: add `'blob:'` to `img-src` (#2265) [19:40:59] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 5.816 second response time [19:41:05] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.887 second response time [19:41:14] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.94, 3.05, 2.98 [19:41:14] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.172 second response time [19:41:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.92, 7.20, 6.57 [19:41:17] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.452 second response time [19:41:22] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.41, 8.17, 7.74 [19:41:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+5/-9/±47] 13https://git.io/JSQSn [19:41:36] [02miraheze/puppet] 07paladox 0359e2e94 - Update mongodb module [19:42:11] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:42:28] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:03] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.87, 6.54, 6.76 [19:43:15] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.05, 6.67, 6.44 [19:43:46] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:48] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:44:10] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.675 second response time [19:44:25] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:44:26] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:44:28] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.396 second response time [19:44:30] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:44:48] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:45:08] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:13] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JSQSR [19:45:15] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:45:15] [02miraheze/ManageWiki] 07Universal-Omega 0312521e3 - remove check on $val being true for setting namespace configs [19:45:16] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:17] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [19:45:18] [02ManageWiki] 07Universal-Omega opened pull request 03#322: remove check on $val being true for setting namespace configs - 13https://git.io/JSQS0 [19:45:22] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:45:30] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:45:36] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:45:40] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:45:56] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:46:01] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:46:39] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JSQSg [19:46:40] [02miraheze/ManageWiki] 07Universal-Omega 0397e433c - Update ManageWikiHooks.php [19:46:42] [02ManageWiki] 07Universal-Omega synchronize pull request 03#322: remove check on $val being true for setting namespace configs - 13https://git.io/JSQS0 [19:47:06] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23663 bytes in 0.265 second response time [19:47:52] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.330 second response time [19:48:26] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.726 second response time [19:48:26] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.589 second response time [19:48:34] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.978 second response time [19:49:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.39, 7.06, 6.91 [19:49:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.08, 7.81, 7.80 [19:49:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.54, 6.89, 6.50 [19:51:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.21, 7.36, 7.03 [19:51:10] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.62, 7.93, 7.96 [19:51:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.64, 7.61, 6.72 [19:51:16] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.62, 7.68, 6.82 [19:51:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQSM [19:51:17] [02miraheze/puppet] 07paladox 039791edc - mongodb: Force use of buster repo [19:51:38] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 4.930 second response time [19:51:39] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.041 second response time [19:51:41] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.321 second response time [19:51:50] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.314 second response time [19:52:19] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.310 second response time [19:52:22] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 7.680 second response time [19:52:41] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:53:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.28, 6.28, 6.68 [19:53:02] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.262 second response time [19:53:03] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.012 second response time [19:53:08] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.70, 7.27, 6.69 [19:53:26] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:53:33] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.041 second response time [19:53:37] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.659 second response time [19:53:49] miraheze/ManageWiki - Universal-Omega the build passed. [19:54:10] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:54:13] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 27643 MB (6% inode=97%); [19:54:13] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.286 second response time [19:54:17] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.81, 6.78, 6.01 [19:54:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.42, 3.08 [19:55:05] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.58, 7.73, 7.82 [19:55:26] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.916 second response time [19:56:13] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.625 second response time [19:56:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.91, 6.39, 5.96 [19:56:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.37, 3.09 [19:56:50] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.915 second response time [19:57:11] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.61, 7.32, 6.96 [19:57:14] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.237 second response time [19:57:15] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 3.751 second response time [19:57:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQ9O [19:57:34] [02miraheze/puppet] 07paladox 036eab151 - graylog: Use http proxy to download [19:57:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:58:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.07, 3.35, 3.12 [19:58:59] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.11, 7.56, 7.75 [19:59:10] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.07, 7.58, 7.10 [20:00:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.70, 7.75, 7.78 [20:01:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.68, 5.70, 6.73 [20:01:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.82, 6.63, 6.64 [20:02:56] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:03:06] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.33, 7.41, 7.19 [20:03:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQ96 [20:03:28] [02miraheze/puppet] 07paladox 0355d5dee - Update graylog121.yaml [20:03:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [20:04:09] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:04:10] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:04:55] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.445 second response time [20:06:10] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.232 second response time [20:06:11] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 3.107 second response time [20:06:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.88, 7.46, 7.73 [20:08:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQHf [20:08:43] [02miraheze/puppet] 07paladox 03b0f8035 - Update graylog.pp [20:09:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.90, 6.80, 6.89 [20:09:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQHT [20:09:17] [02miraheze/puppet] 07paladox 039159262 - Update graylog.pp [20:09:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:09:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:09:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQHt [20:09:49] [02miraheze/puppet] 07paladox 03ff16059 - Update graylog.pp [20:10:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.81, 8.25, 7.98 [20:11:01] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.53, 6.12, 6.68 [20:11:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.79, 6.20, 6.67 [20:11:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ff1605955f0f...4bbbecdc0fe6 [20:11:46] [url] Comparing ff1605955f0f...4bbbecdc0fe6 · miraheze/puppet · GitHub | github.com [20:11:47] [02miraheze/puppet] 07paladox 034bbbecd - Fix [20:12:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.92, 6.74, 6.60 [20:12:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:13:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [20:14:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.23, 7.92, 7.91 [20:15:49] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:16:07] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:16:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb [20:16:20] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.05, 7.38, 6.88 [20:16:36] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:16:41] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:17:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:17:33] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:18:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.67, 7.41, 6.94 [20:18:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.87, 8.19, 8.00 [20:19:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.44, 3.25 [20:19:50] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.83, 7.74, 7.16 [20:20:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQH7 [20:20:12] [02miraheze/puppet] 07paladox 0333ac0ae - Update apt.pp [20:20:13] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.027 second response time [20:20:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQHd [20:20:40] [02miraheze/puppet] 07paladox 0394549ee - graylog: Fix apt proxy [20:20:41] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [20:20:43] [02puppet] 07paladox opened pull request 03#2266: graylog: Fix apt proxy - 13https://git.io/JSQHF [20:20:43] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.987 second response time [20:20:47] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.840 second response time [20:20:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.48, 7.85, 7.90 [20:20:59] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.55, 3.07, 3.14 [20:21:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:21:43] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 1.631 second response time [20:21:45] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.16, 7.54, 7.16 [20:21:53] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.018 second response time [20:22:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.05, 7.72, 7.17 [20:22:28] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:22:53] [02puppet] 07paladox closed pull request 03#2266: graylog: Fix apt proxy - 13https://git.io/JSQHF [20:22:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQQT [20:22:56] [02miraheze/puppet] 07paladox 038077ddb - graylog: Fix apt proxy (#2266) [20:22:57] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [20:22:59] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [20:23:07] RECOVERY - graylog121 Puppet on graylog121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:23:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQQt [20:23:14] [02miraheze/puppet] 07paladox 033d03502 - Update graylog-server-default [20:23:32] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:23:35] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:23:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.74, 7.18, 6.37 [20:23:45] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.24, 5.86, 5.82 [20:23:50] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:24:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.33, 7.59, 7.18 [20:24:06] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:25:35] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.284 second response time [20:25:43] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.700 second response time [20:25:43] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:25:49] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:25:55] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:25:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.89, 6.24, 6.74 [20:25:57] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:26:42] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:49] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQQu [20:26:51] [02miraheze/dns] 07paladox 037fd0633 - Add graylog-new [20:27:38] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.44, 6.12, 5.93 [20:27:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.38, 7.31, 6.65 [20:27:42] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.181 second response time [20:27:46] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.537 second response time [20:27:55] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.357 second response time [20:27:56] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.424 second response time [20:27:56] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.901 second response time [20:28:05] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.330 second response time [20:28:40] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.399 second response time [20:28:44] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.504 second response time [20:28:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.02, 6.90, 7.27 [20:29:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.76, 7.44, 7.17 [20:29:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:30:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.46, 6.84, 7.22 [20:31:17] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.67, 7.32, 7.15 [20:31:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:33:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.08, 6.49, 6.53 [20:35:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.22, 7.37, 7.18 [20:37:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.43, 6.12, 6.77 [20:37:23] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 12.32, 8.35, 6.81 [20:37:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.140.43/cpweb [20:37:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:37:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.29, 7.09, 6.43 [20:38:05] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:38:09] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:38:13] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:38:19] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:38:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.03, 6.09, 6.80 [20:39:02] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:39:20] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.03, 7.90, 6.85 [20:39:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.61, 7.05, 6.51 [20:39:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.84, 7.27, 6.85 [20:40:05] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.134 second response time [20:40:05] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.451 second response time [20:40:12] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 1.065 second response time [20:40:21] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.128 second response time [20:40:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.68, 3.73, 3.23 [20:41:04] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 4.778 second response time [20:41:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:41:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:41:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQ5f [20:41:47] [02miraheze/puppet] 07paladox 033feb3c4 - graylog: Set elasticsearch host to elasticsearch.miraheze.org for graylog121 [20:41:48] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [20:41:50] [02puppet] 07paladox opened pull request 03#2267: graylog: Set elasticsearch host to elasticsearch.miraheze.org for gra… - 13https://git.io/JSQ5U [20:41:55] [02puppet] 07paladox edited pull request 03#2267: graylog: Set elasticsearch host to elasticsearch.miraheze.org for graylog121 - 13https://git.io/JSQ5U [20:42:00] [02puppet] 07paladox edited pull request 03#2267: graylog: Set elasticsearch host to elasticsearch.miraheze.org for graylog121 - 13https://git.io/JSQ5U [20:42:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQ5t [20:42:24] [02miraheze/puppet] 07paladox 0364d9b9e - Update graylog121.yaml [20:42:25] [02puppet] 07paladox synchronize pull request 03#2267: graylog: Set elasticsearch host to elasticsearch.miraheze.org for graylog121 - 13https://git.io/JSQ5U [20:42:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.01, 3.23, 3.12 [20:43:15] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.68, 6.26, 6.42 [20:43:32] [02puppet] 07paladox closed pull request 03#2267: graylog: Set elasticsearch host to elasticsearch.miraheze.org for graylog121 - 13https://git.io/JSQ5U [20:43:33] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [20:43:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.98, 6.43, 6.39 [20:43:35] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [20:43:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JSQ54 [20:43:38] [02miraheze/puppet] 07paladox 03afac016 - graylog: Set elasticsearch host to elasticsearch.miraheze.org for graylog121 (#2267) [20:43:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.57, 5.63, 6.32 [20:44:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQ5V [20:44:37] [02miraheze/puppet] 07paladox 0346af653 - Update graylog.pp [20:47:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.54, 7.03, 6.91 [20:50:36] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.52, 1.81, 1.14 [20:51:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.20, 6.79, 6.34 [20:51:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [20:51:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:51:45] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:53] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:51:56] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:52:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.04, 3.79, 3.30 [20:52:34] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.67, 1.43, 1.08 [20:53:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.49, 6.73, 6.38 [20:53:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.45, 7.44, 7.06 [20:53:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:53:49] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 4.798 second response time [20:53:54] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.483 second response time [20:53:58] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.232 second response time [20:55:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.81, 7.27, 7.05 [20:56:38] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:56:42] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:56:42] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.06, 6.94, 6.53 [20:57:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:58:35] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.683 second response time [20:58:40] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.013 second response time [20:58:40] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.95, 6.69, 6.44 [20:58:41] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.322 second response time [20:59:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.73, 6.01, 6.27 [20:59:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.13, 6.25, 6.73 [20:59:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:00:03] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:00:07] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.15, 6.63, 6.18 [21:00:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.97, 3.65 [21:00:38] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.91, 5.91, 6.17 [21:01:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQdh [21:01:47] [02miraheze/puppet] 07paladox 03ba99718 - graylog: Set 1gb heap [21:02:01] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.976 second response time [21:02:03] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.14, 7.41, 6.51 [21:03:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:05:56] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.73, 6.43, 6.34 [21:06:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 3.16, 3.39 [21:07:07] PROBLEM - graylog2 HTTPS on graylog2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 311 bytes in 0.030 second response time [21:10:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.52, 3.49 [21:11:07] PROBLEM - graylog121 Puppet on graylog121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Mongodb_conn_validator[mongodb] [21:12:20] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.05, 6.24, 5.94 [21:12:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.34, 3.25, 3.39 [21:13:25] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.19, 6.74, 6.28 [21:14:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.03, 7.36, 6.37 [21:15:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.58, 6.33, 6.19 [21:15:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [21:15:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.93, 6.89, 6.56 [21:15:53] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:15:54] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.66, 7.45, 6.53 [21:15:56] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:16:02] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:16:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2607:5300:201:3100::929a/cpweb [21:16:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.40, 3.43 [21:16:40] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:17:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:19:42] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.95, 7.63, 6.90 [21:19:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.26, 7.66, 6.84 [21:20:23] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:20:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.21, 3.37 [21:20:49] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 0.158 second response time [21:21:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:21:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.88, 7.47, 6.96 [21:21:39] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.71, 8.20, 7.08 [21:21:42] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.22, 6.20, 6.42 [21:21:53] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:03] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.342 second response time [21:22:05] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.018 second response time [21:22:07] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.18, 6.64, 6.64 [21:22:12] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.921 second response time [21:22:19] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQb1 [21:22:22] [02miraheze/puppet] 07paladox 03e1f3836 - graylog: Downgrade mongodb to 4.4.10 [21:22:25] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 3.471 second response time [21:23:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.45, 6.64, 6.24 [21:23:31] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.75, 7.53, 7.01 [21:23:36] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.32, 7.68, 7.02 [21:23:51] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:55] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.553 second response time [21:24:20] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.429 second response time [21:24:34] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:25:09] RECOVERY - graylog121 Puppet on graylog121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:25:15] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.28, 6.70, 6.33 [21:25:54] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.129 second response time [21:26:01] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.64, 7.49, 6.98 [21:26:29] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.212 second response time [21:26:52] RECOVERY - graylog121 HTTPS on graylog121 is OK: HTTP OK: HTTP/1.1 200 OK - 1418 bytes in 0.053 second response time [21:26:53] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.25, 2.59, 1.24 [21:27:19] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:27:29] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.22, 6.89, 6.59 [21:27:30] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:27:58] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:27:59] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.47, 7.42, 7.02 [21:28:02] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:29:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.77, 7.56, 7.29 [21:29:19] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23663 bytes in 1.847 second response time [21:29:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.37, 6.96, 6.64 [21:29:25] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.014 second response time [21:29:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:29:54] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.345 second response time [21:29:59] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.344 second response time [21:31:19] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.81, 6.74, 6.62 [21:31:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:33:18] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.67, 7.72, 7.19 [21:33:52] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.48, 7.79, 7.25 [21:34:52] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 0.11, 1.40, 1.30 [21:35:06] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.41, 7.10, 6.50 [21:35:06] PROBLEM - graylog121 Puppet on graylog121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [21:35:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.29, 7.40, 7.13 [21:35:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:36:09] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:36:12] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:36:17] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:36:29] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:37:04] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.86, 6.18, 6.23 [21:37:44] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:37:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.52, 7.89, 7.44 [21:38:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQAT [21:38:23] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.856 second response time [21:38:24] [02miraheze/puppet] 07paladox 0368b9d4f - Update graylog2.yaml [21:39:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.62, 7.28, 7.14 [21:39:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.20, 6.97, 6.95 [21:39:44] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.31, 8.15, 7.59 [21:40:14] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:40:40] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.041 second response time [21:40:46] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:40:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQAZ [21:40:54] [02miraheze/puppet] 07paladox 037e90ab7 - Update graylog.pp [21:40:55] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:41:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.74, 7.50, 6.95 [21:41:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:41:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.21, 7.03, 6.98 [21:42:41] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 1.399 second response time [21:42:41] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.463 second response time [21:42:46] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:42:57] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.022 second response time [21:43:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.03, 6.67, 6.72 [21:43:07] RECOVERY - graylog2 HTTPS on graylog2 is OK: HTTP OK: HTTP/1.1 200 OK - 1418 bytes in 0.011 second response time [21:43:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.60, 6.42, 6.76 [21:44:32] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.796 second response time [21:44:38] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.031 second response time [21:44:39] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.421 second response time [21:44:44] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.008 second response time [21:45:39] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:45:48] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:45:58] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:45:59] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:46:30] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:46:37] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 13.94, 9.01, 4.19 [21:47:15] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw8 [21:47:38] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:47:40] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:47:47] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:47:52] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23663 bytes in 4.902 second response time [21:47:54] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.011 second response time [21:47:54] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.335 second response time [21:48:01] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:27] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.329 second response time [21:48:59] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:48:59] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:49:07] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:49:07] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:49:20] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw9 mw12 [21:49:41] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.498 second response time [21:49:44] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.265 second response time [21:50:15] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.619 second response time [21:51:03] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JSQx8 [21:51:05] [02miraheze/ManageWiki] 07Universal-Omega 0393e1db1 - fix setting defaults for ManageWikiNamespacesAdditional [21:51:06] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vpSns [21:51:06] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [21:51:08] [02ManageWiki] 07Universal-Omega opened pull request 03#323: fix setting defaults for ManageWikiNamespacesAdditional - 13https://git.io/JSQx4 [21:51:45] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.045 second response time [21:51:55] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.657 second response time [21:52:00] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.664 second response time [21:53:20] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [21:53:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQxQ [21:53:54] [02miraheze/puppet] 07paladox 03e39f6b3 - openldap: Simplify rule [21:53:55] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [21:53:57] [02puppet] 07paladox opened pull request 03#2268: openldap: Simplify rule - 13https://git.io/JSQx7 [21:54:00] [02puppet] 07paladox edited pull request 03#2268: openldap: Simplify ferm rule - 13https://git.io/JSQx7 [21:54:31] [02puppet] 07paladox closed pull request 03#2268: openldap: Simplify ferm rule - 13https://git.io/JSQx7 [21:54:32] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [21:54:34] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [21:54:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQxN [21:54:37] [02miraheze/puppet] 07paladox 031706217 - openldap: Simplify rule (#2268) [21:54:39] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw12 [21:54:40] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:54:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.55, 7.14, 6.85 [21:55:06] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.351 second response time [21:55:12] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.271 second response time [21:55:25] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.644 second response time [21:55:27] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 3.139 second response time [21:55:39] miraheze/ManageWiki - Universal-Omega the build passed. [21:55:59] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:56:08] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:56:14] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:56:18] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:56:21] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:36] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 2.23, 3.35, 3.77 [21:56:37] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.333 second response time [21:56:38] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [21:58:32] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:58:33] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.47, 7.77, 7.15 [22:00:28] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.44, 7.21, 7.02 [22:00:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQpr [22:00:35] [02miraheze/puppet] 07paladox 031495171 - make new mw cluster use syslog_ng [22:00:36] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 1.24, 2.12, 3.16 [22:00:37] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [22:00:38] [02puppet] 07paladox opened pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:01:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQpX [22:01:03] [02miraheze/puppet] 07paladox 03509ea1a - Update mw102.yaml [22:01:04] [02puppet] 07paladox synchronize pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:01:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQp9 [22:01:35] [02miraheze/puppet] 07paladox 03ee77f6a - Update mw102.yaml [22:01:37] [02puppet] 07paladox synchronize pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:01:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQpQ [22:01:51] [02miraheze/puppet] 07paladox 03a9bdd1b - Update mw111.yaml [22:01:53] [02puppet] 07paladox synchronize pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:02:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQp5 [22:02:01] [02miraheze/puppet] 07paladox 038e40835 - Update mw121.yaml [22:02:02] [02puppet] 07paladox synchronize pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:02:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQpF [22:02:12] [02miraheze/puppet] 07paladox 03e54c2da - Update mw112.yaml [22:02:14] [02puppet] 07paladox synchronize pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:02:19] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 9.114 second response time [22:02:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQpb [22:02:22] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 7.939 second response time [22:02:22] [02miraheze/puppet] 07paladox 03ee1478c - Update mw122.yaml [22:02:24] [02puppet] 07paladox synchronize pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:02:26] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 8.092 second response time [22:02:30] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.676 second response time [22:02:36] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.124 second response time [22:02:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQpA [22:02:41] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23663 bytes in 6.802 second response time [22:02:42] [02miraheze/puppet] 07paladox 032d81060 - Update mwtask111.yaml [22:02:43] [02puppet] 07paladox synchronize pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:02:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQph [22:02:53] [02miraheze/puppet] 07paladox 03e8d134a - Update test101.yaml [22:02:54] [02puppet] 07paladox synchronize pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:04:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSQht [22:04:32] [02miraheze/puppet] 07paladox 03b836bb8 - base::syslog: Allow adjusting hostname [22:04:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQhm [22:04:53] [02miraheze/puppet] 07paladox 03c622720 - Update mw101.yaml [22:04:55] [02puppet] 07paladox synchronize pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:04:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQhY [22:05:00] [02miraheze/puppet] 07paladox 030d9080c - Update mw102.yaml [22:05:02] [02puppet] 07paladox synchronize pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:05:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQhO [22:05:07] [02miraheze/puppet] 07paladox 033beacb8 - Update mw112.yaml [22:05:09] [02puppet] 07paladox synchronize pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:05:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQhn [22:05:20] [02miraheze/puppet] 07paladox 03d4450de - Update mw111.yaml [22:05:22] [02puppet] 07paladox synchronize pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:05:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQhc [22:05:28] [02miraheze/puppet] 07paladox 03b0d919a - Update mw121.yaml [22:05:29] [02puppet] 07paladox synchronize pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:05:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQhC [22:05:35] [02miraheze/puppet] 07paladox 0346e2a44 - Update mwtask111.yaml [22:05:36] [02puppet] 07paladox synchronize pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:05:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQh8 [22:05:44] [02miraheze/puppet] 07paladox 0327c343f - Update test101.yaml [22:05:45] [02puppet] 07paladox synchronize pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:05:52] [02puppet] 07paladox closed pull request 03#2269: make new mw cluster use syslog_ng - 13https://git.io/JSQpK [22:05:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±8] 13https://git.io/JSQh4 [22:05:55] [02miraheze/puppet] 07paladox 03d268e91 - make new mw cluster use syslog_ng (#2269) [22:05:57] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [22:05:58] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [22:07:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 3.50, 6.48, 7.59 [22:07:26] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JSQh2 [22:07:28] [02miraheze/mw-config] 07paladox 036a3be42 - Log to graylog on new mw cluster [22:07:29] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [22:07:31] [02mw-config] 07paladox opened pull request 03#4345: Log to graylog on new mw cluster - 13https://git.io/JSQha [22:07:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:07:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:08:08] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.39, 6.08, 6.68 [22:08:20] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.50, 6.73, 6.11 [22:08:31] miraheze/mw-config - paladox the build passed. [22:09:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.73, 7.98, 7.99 [22:09:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQhH [22:09:58] [02miraheze/puppet] 07paladox 03e8f4544 - Fix [22:10:00] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [22:10:06] [02puppet] 07paladox opened pull request 03#2270: Fix - 13https://git.io/JSQh7 [22:10:18] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.23, 7.43, 6.40 [22:11:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:11:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:12:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.73, 7.25, 7.05 [22:12:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQjf [22:12:28] [02miraheze/puppet] 07paladox 0314256ea - Update mw102.yaml [22:12:30] [02puppet] 07paladox synchronize pull request 03#2270: Fix - 13https://git.io/JSQh7 [22:12:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQjU [22:12:37] [02miraheze/puppet] 07paladox 038b7f95b - Update mw111.yaml [22:12:38] [02puppet] 07paladox synchronize pull request 03#2270: Fix - 13https://git.io/JSQh7 [22:12:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQjT [22:12:51] [02miraheze/puppet] 07paladox 035c9c05b - Update mw112.yaml [22:12:52] [02puppet] 07paladox synchronize pull request 03#2270: Fix - 13https://git.io/JSQh7 [22:13:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQjL [22:13:02] [02miraheze/puppet] 07paladox 03a625d5c - Update mw121.yaml [22:13:04] [02puppet] 07paladox synchronize pull request 03#2270: Fix - 13https://git.io/JSQh7 [22:13:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQjY [22:13:23] [02miraheze/puppet] 07paladox 03d78f141 - Update mw122.yaml [22:13:25] [02puppet] 07paladox synchronize pull request 03#2270: Fix - 13https://git.io/JSQh7 [22:13:31] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:13:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQjO [22:13:33] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:13:33] [02miraheze/puppet] 07paladox 030d741cb - Update mwtask111.yaml [22:13:34] [02puppet] 07paladox synchronize pull request 03#2270: Fix - 13https://git.io/JSQh7 [22:13:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSQj3 [22:13:44] [02miraheze/puppet] 07paladox 030cc877e - Update test101.yaml [22:13:45] [02puppet] 07paladox synchronize pull request 03#2270: Fix - 13https://git.io/JSQh7 [22:13:53] [02puppet] 07paladox closed pull request 03#2270: Fix - 13https://git.io/JSQh7 [22:13:53] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:13:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±8] 13https://git.io/JSQjZ [22:13:56] [02miraheze/puppet] 07paladox 03767b681 - Fix (#2270) [22:13:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.24, 7.48, 7.15 [22:13:57] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:13:57] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [22:13:59] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [22:13:59] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.36, 7.38, 6.63 [22:14:15] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:14:40] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:42] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:44] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:37] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 6.574 second response time [22:15:38] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 6.403 second response time [22:16:46] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.666 second response time [22:16:48] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.493 second response time [22:17:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.02, 7.58, 7.30 [22:18:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.53, 7.72, 6.89 [22:18:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.74, 7.38, 7.92 [22:19:42] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.37, 7.67, 7.35 [22:20:06] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.56, 6.61, 6.59 [22:20:39] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23663 bytes in 9.522 second response time [22:20:57] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 7.642 second response time [22:21:38] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:21:40] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:00] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:03] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:14] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 9.922 second response time [22:22:24] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.30, 8.23, 8.11 [22:23:06] RECOVERY - graylog121 Puppet on graylog121 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [22:23:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.03, 7.70, 7.47 [22:23:57] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:23:58] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.334 second response time [22:24:00] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.83, 7.59, 6.97 [22:24:03] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.456 second response time [22:24:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JS7ec [22:24:06] [02miraheze/puppet] 07paladox 0318b0da3 - graylog: We don't need to support openvz anymore [22:24:10] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:24:17] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JS7e4 [22:24:17] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:24:19] [02miraheze/ManageWiki] 07Universal-Omega 03f472848 - Update ManageWikiHooks.php [22:24:20] [02ManageWiki] 07Universal-Omega synchronize pull request 03#323: fix setting defaults for ManageWikiNamespacesAdditional - 13https://git.io/JSQx4 [22:24:23] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.197 second response time [22:24:25] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 7.581 second response time [22:24:30] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 7.621 second response time [22:24:32] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:24:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.79, 3.12, 2.81 [22:24:51] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:24:51] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:25:02] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:25:03] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:25:04] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:25:16] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:25:16] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:25:17] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:25:27] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:25:40] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.450 second response time [22:25:49] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:25:49] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.026 second response time [22:25:58] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.13, 7.60, 7.04 [22:26:03] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.422 second response time [22:26:10] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:26:10] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:26:13] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.657 second response time [22:26:17] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:26:23] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:26:31] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.767 second response time [22:26:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.87, 3.15, 2.87 [22:26:51] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.606 second response time [22:26:54] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 5.510 second response time [22:27:02] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.690 second response time [22:27:02] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23663 bytes in 0.267 second response time [22:27:06] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.639 second response time [22:27:12] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.023 second response time [22:27:14] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.332 second response time [22:27:17] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.13, 6.92, 6.26 [22:27:21] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.304 second response time [22:27:30] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.046 second response time [22:27:44] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.371 second response time [22:27:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.66, 7.76, 7.18 [22:28:08] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.018 second response time [22:28:08] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.396 second response time [22:28:14] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.475 second response time [22:28:21] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 1.942 second response time [22:28:22] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.342 second response time [22:29:00] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JS7eQ [22:29:01] [02miraheze/ManageWiki] 07Universal-Omega 034b1ba4d - Update ManageWikiHooks.php [22:29:03] [02ManageWiki] 07Universal-Omega synchronize pull request 03#323: fix setting defaults for ManageWikiNamespacesAdditional - 13https://git.io/JSQx4 [22:29:09] miraheze/ManageWiki - Universal-Omega the build passed. [22:29:15] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.54, 6.22, 6.08 [22:29:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:31:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.65, 7.27, 7.21 [22:31:47] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:31:49] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.022 second response time [22:31:51] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.02, 6.11, 6.63 [22:31:54] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:32:15] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:32:21] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:32:27] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:33:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.27, 7.26, 7.24 [22:33:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.08, 6.83, 6.33 [22:33:19] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:33:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:33:44] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.921 second response time [22:33:50] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.103 second response time [22:33:51] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.315 second response time [22:33:55] miraheze/ManageWiki - Universal-Omega the build passed. [22:34:18] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.591 second response time [22:34:20] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.539 second response time [22:34:31] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.427 second response time [22:35:15] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.66, 6.62, 6.31 [22:35:45] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.36, 6.95, 6.88 [22:37:18] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.085 second response time [22:37:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:37:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:37:43] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.40, 6.39, 6.69 [22:38:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.36, 7.19, 7.83 [22:41:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.81, 5.98, 6.62 [22:41:38] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JS7fq [22:41:40] [02miraheze/ManageWiki] 07Universal-Omega 03c7322c4 - Update ManageWikiHooks.php [22:41:41] [02ManageWiki] 07Universal-Omega synchronize pull request 03#323: fix setting defaults for ManageWikiNamespacesAdditional - 13https://git.io/JSQx4 [22:45:56] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JS7fo [22:45:57] [02miraheze/ManageWiki] 07Universal-Omega 03be33f94 - Update ManageWikiHooks.php [22:45:59] [02ManageWiki] 07Universal-Omega synchronize pull request 03#323: fix setting defaults for ManageWikiNamespacesAdditional - 13https://git.io/JSQx4 [22:46:10] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.42, 7.31, 6.70 [22:46:22] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:46:29] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:46:51] miraheze/ManageWiki - Universal-Omega the build passed. [22:46:51] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:46:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.10, 5.41, 6.80 [22:47:00] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:47:03] [02ManageWiki] 07Universal-Omega closed pull request 03#322: remove check on $val being true for setting namespace configs - 13https://git.io/JSQS0 [22:47:06] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [22:47:07] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [22:47:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [22:47:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [22:47:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.61, 7.13, 6.88 [22:47:52] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:48:06] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.63, 7.59, 6.89 [22:48:20] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.215 second response time [22:48:27] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.016 second response time [22:48:46] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.262 second response time [22:48:59] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.347 second response time [22:49:41] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.81, 6.51, 6.68 [22:49:48] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.378 second response time [22:50:03] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.38, 6.54, 6.57 [22:50:44] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:52] miraheze/ManageWiki - Universal-Omega the build passed. [22:51:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.40, 6.81, 6.43 [22:51:03] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:05] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:13] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:20] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:32] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:37] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:55] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:52:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.88, 3.70, 3.29 [22:52:46] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.868 second response time [22:53:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.19, 6.16, 6.24 [22:53:05] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.762 second response time [22:53:10] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 5.607 second response time [22:53:13] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 2.831 second response time [22:53:17] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 2.440 second response time [22:53:31] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.746 second response time [22:53:35] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.337 second response time [22:53:50] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.31, 7.02, 7.06 [22:53:52] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.316 second response time [22:54:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.99, 3.05, 3.10 [22:54:19] [02ManageWiki] 07Universal-Omega closed pull request 03#323: fix setting defaults for ManageWikiNamespacesAdditional - 13https://git.io/JSQx4 [22:54:21] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JS7JW [22:54:22] [02miraheze/ManageWiki] 07Universal-Omega 03dc6542e - fix setting defaults for ManageWikiNamespacesAdditional (#323) [22:54:24] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [22:54:25] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vpSns [22:55:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.97, 6.70, 7.71 [22:55:55] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.19, 6.98, 6.75 [22:56:36] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JS7JV [22:56:38] [02miraheze/mediawiki] 07Universal-Omega 034eddada - Update ManageWiki [22:57:05] !log [universalomega@test3] starting deploy of {'world': True} to skip [22:57:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:57:39] !log [universalomega@mw11] starting deploy of {'world': True} to all [22:57:45] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.19, 7.45, 7.20 [22:57:51] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.32, 6.62, 6.64 [22:57:51] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.04, 6.91, 6.41 [22:58:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:18] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:30] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [22:58:48] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:52] ... [22:59:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:59:43] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.25, 7.25, 7.16 [22:59:44] miraheze/ManageWiki - Universal-Omega the build passed. [22:59:50] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.77, 7.24, 6.60 [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:02] !log [@mw11] starting deploy of {'l10nupdate': True} to ovlon [23:00:03] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:35] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:41] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 38s [23:00:46] ... [23:00:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 3.52, 3.31 [23:00:55] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:56] ... [23:01:11] !log [@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 69s [23:01:27] three bots? [23:01:33] oh nvm [23:01:37] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:39] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [23:01:48] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.10, 7.79, 6.87 [23:02:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:39] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:49] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:02:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.24, 3.40, 3.28 [23:02:56] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [23:03:02] !log [universalomega@test3] finished deploy of {'world': True} to skip - SUCCESS in 355s [23:03:06] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:32] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [23:03:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [23:03:33] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.80, 19.06, 17.22 [23:03:39] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.33, 6.77, 6.61 [23:03:52] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:04:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JS7UY [23:04:10] [02miraheze/mw-config] 07Universal-Omega 031379928 - Remove default override for $wgExemptFromUserRobotsControl [23:04:12] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [23:04:21] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:04:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:04:37] [02mw-config] 07Universal-Omega opened pull request 03#4346: Remove default override for $wgExemptFromUserRobotsControl - 13https://git.io/JS7UO [23:04:40] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [23:04:46] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:46] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:04:48] [02mw-config] 07Universal-Omega edited pull request 03#4346: Remove default override for $wgExemptFromUserRobotsControl - 13https://git.io/JS7UO [23:04:48] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:05:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:03] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:05:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:37] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:05:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.93, 7.87, 7.02 [23:06:04] miraheze/mw-config - Universal-Omega the build passed. [23:06:51] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:07:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.33, 6.50, 7.07 [23:07:37] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.68, 7.79, 7.10 [23:07:49] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.14, 7.79, 7.27 [23:08:13] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:08:15] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:08:23] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.88, 7.41, 6.80 [23:08:47] RECOVERY - test3 APT on test3 is OK: APT OK: 19 packages available for upgrade (0 critical updates). [23:08:55] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.762 second response time [23:09:00] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 8.051 second response time [23:09:13] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.976 second response time [23:09:32] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.02, 7.37, 7.22 [23:09:33] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.79, 19.70, 18.28 [23:09:44] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.028 second response time [23:09:45] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.345 second response time [23:09:55] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.334 second response time [23:10:00] !log [universalomega@mw11] finished deploy of {'world': True} to all - SUCCESS in 741s [23:10:12] [02mw-config] 07Universal-Omega closed pull request 03#4346: Remove default override for $wgExemptFromUserRobotsControl - 13https://git.io/JS7UO [23:10:12] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 2.282 second response time [23:10:13] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [23:10:15] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [23:10:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JS7UX [23:10:17] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.485 second response time [23:10:18] [02miraheze/mw-config] 07Universal-Omega 03709fd84 - Remove default override for $wgExemptFromUserRobotsControl (#4346) [23:10:21] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.88, 7.80, 7.01 [23:10:52] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:11:03] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.434 second response time [23:11:08] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [23:11:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:11:18] miraheze/mw-config - Universal-Omega the build passed. [23:11:29] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.78, 7.79, 7.23 [23:11:32] !log [@mw11] starting deploy of {'config': True} to ovlon [23:11:59] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:12:19] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.40, 7.51, 7.02 [23:12:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:12:50] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 77s [23:12:56] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:13:12] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.472 second response time [23:13:14] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:13:23] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:13:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:13:38] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:13:54] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.011 second response time [23:13:57] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:13:58] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [23:14:10] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:14] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:15:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.89, 7.80, 7.75 [23:15:13] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.025 second response time [23:15:16] PROBLEM - mw101 Check Gluster Clients on mw101 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [23:15:22] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 2.017 second response time [23:15:24] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.94, 7.27, 7.29 [23:15:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:15:38] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.814 second response time [23:16:09] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.366 second response time [23:16:09] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.310 second response time [23:16:15] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.24, 6.18, 6.58 [23:17:18] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 3.48, 6.58, 7.07 [23:18:03] !log [@test101] starting deploy of {'config': True} to skip [23:18:04] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [23:18:44] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:48] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [23:18:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.92, 7.25, 6.88 [23:19:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.52, 8.00, 7.82 [23:19:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:41] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.20, 5.99, 6.62 [23:19:52] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:55] ... [23:20:03] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:08] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:16] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:17] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:24] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:20:33] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:41] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.75, 7.97, 7.19 [23:21:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.51, 7.49, 7.66 [23:21:13] !log [@mwtask111] starting deploy of {'config': True} to scsvg [23:21:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.07, 7.75, 7.40 [23:21:20] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 6s [23:21:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb [23:21:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:21:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JS7TQ [23:22:00] [02miraheze/puppet] 07paladox 03e201150 - dns: Support listening on ipv6 and update some other dns related stuff [23:22:01] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [23:22:03] [02puppet] 07paladox opened pull request 03#2271: dns: Support listening on ipv6 and update some other dns related stuff - 13https://git.io/JS7T7 [23:22:04] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.122 second response time [23:22:07] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.960 second response time [23:22:09] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:14] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.328 second response time [23:22:16] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.608 second response time [23:22:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:21] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [23:22:23] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.277 second response time [23:22:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JS7Td [23:22:25] [02miraheze/puppet] 07paladox 0320a6663 - Update resolv.conf.erb [23:22:27] [02puppet] 07paladox synchronize pull request 03#2271: dns: Support listening on ipv6 and update some other dns related stuff - 13https://git.io/JS7T7 [23:22:28] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.495 second response time [23:22:38] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.362 second response time [23:22:48] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:23:02] ... [23:23:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.79, 7.80, 7.73 [23:23:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.87, 7.51, 7.37 [23:23:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.92, 6.65, 6.81 [23:24:04] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.57, 6.77, 6.57 [23:25:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.58, 7.42, 7.17 [23:25:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.06, 7.47, 7.62 [23:25:12] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.38, 7.12, 7.10 [23:25:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.96, 8.39, 7.70 [23:25:15] !log [@test3] starting deploy of {'config': True} to skip [23:25:16] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [23:25:25] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:25:42] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:25:59] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:26:15] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:26:36] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:26:44] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:26:49] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:26:57] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:05] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [23:27:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.99, 7.98, 7.63 [23:27:50] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:57] ... [23:28:13] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26700 MB (5% inode=97%); [23:28:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:41] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.495 second response time [23:28:44] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.374 second response time [23:29:07] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.99, 7.72, 7.37 [23:29:34] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.943 second response time [23:29:53] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 5.441 second response time [23:30:05] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.253 second response time [23:30:14] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.303 second response time [23:30:44] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.947 second response time [23:31:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:31:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:31:55] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.44, 6.74, 6.67 [23:33:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.51, 6.36, 6.77 [23:33:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.26, 7.70, 7.60 [23:33:16] RECOVERY - mw101 Check Gluster Clients on mw101 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [23:33:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.17, 6.12, 6.74 [23:35:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.06, 7.46, 7.52 [23:35:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.13, 7.08, 7.23 [23:37:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.92, 7.16, 7.23 [23:37:18] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:37:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [23:37:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.34, 7.38, 7.12 [23:37:46] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.63, 7.90, 7.13 [23:38:10] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:38:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:38:34] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 2312s [23:38:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.56, 5.21, 6.44 [23:38:58] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:38:59] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [23:39:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:39:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.54, 7.94, 7.53 [23:39:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.52, 6.78, 6.94 [23:39:44] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.65, 7.76, 7.18 [23:40:10] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.75, 20.08, 18.54 [23:41:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.03, 7.17, 7.29 [23:41:15] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.006 second response time [23:41:27] !log [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 2484s [23:41:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.46, 6.34, 6.76 [23:41:51] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:42:07] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.13, 19.60, 18.55 [23:42:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:42:28] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:42:41] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:42:53] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:42:58] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [23:43:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.86, 7.34, 6.84 [23:43:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.34, 7.01, 7.23 [23:43:07] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:43:34] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:43:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 7.75, 8.28, 7.53 [23:43:59] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:44:22] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.459 second response time [23:45:08] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.705 second response time [23:45:30] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.006 second response time [23:45:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.15, 7.36, 7.27 [23:46:00] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 1.369 second response time [23:46:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.51, 6.89, 6.84 [23:47:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.54, 7.89, 7.14 [23:47:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.52, 7.46, 7.57 [23:47:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JS7tx [23:47:24] [02miraheze/puppet] 07paladox 03beeabbe - base::syslog: force hostname to resolve to ipv6 [23:47:25] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [23:47:27] [02puppet] 07paladox opened pull request 03#2272: base::syslog: force hostname to resolve to ipv6 - 13https://git.io/JS7th [23:48:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JS7qL [23:49:00] [02miraheze/mw-config] 07Universal-Omega 03e380f37 - Set LocalisationUpdate proxy for SCSVG [23:49:02] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [23:49:03] [02mw-config] 07Universal-Omega opened pull request 03#4347: Set LocalisationUpdate proxy for SCSVG - 13https://git.io/JS7qm [23:49:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.33, 5.67, 6.59 [23:49:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.66, 7.74, 7.65 [23:49:33] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:49:40] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:49:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.29, 7.53, 7.31 [23:49:55] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:50:10] miraheze/mw-config - Universal-Omega the build passed. [23:50:27] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:51:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.78, 7.29, 7.52 [23:51:35] !log [universalomega@test101] starting deploy of {'l10nupdate': True} to skip [23:51:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.04, 6.82, 7.09 [23:52:20] !log [universalomega@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 44s [23:52:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JS7qz [23:52:28] [02miraheze/mw-config] 07Universal-Omega 032063d25 - Set LocalisationUpdate proxy for SCSVG (#4347) [23:52:30] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [23:52:31] [02mw-config] 07Universal-Omega closed pull request 03#4347: Set LocalisationUpdate proxy for SCSVG - 13https://git.io/JS7qm [23:52:33] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:52:33] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [23:52:34] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [23:52:47] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:52:52] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to skip [23:52:53] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 1s [23:52:57] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.578 second response time [23:52:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.87, 6.45, 6.70 [23:52:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:07] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:09] !log [universalomega@test101] starting deploy of {'l10nupdate': True} to skip [23:53:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:18] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.197 second response time [23:53:20] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:30] miraheze/mw-config - Universal-Omega the build passed. [23:53:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:53:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:53:38] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.54, 7.30, 7.22 [23:53:41] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.197 second response time [23:54:02] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.633 second response time [23:54:38] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 2.833 second response time [23:54:58] !log [@test3] starting deploy of {'config': True} to skip [23:54:59] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s [23:55:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:13] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:23] <020AAL0BF> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:25] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [23:55:25] ... [23:55:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.91, 7.41, 7.26 [23:56:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JS7q5 [23:56:14] [02miraheze/puppet] 07paladox 03356a119 - redis: Put a space between type and config [23:56:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.42, 20.56, 18.94 [23:57:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [23:58:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.54, 19.49, 18.73 [23:58:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JS7mT [23:58:52] [02miraheze/puppet] 07paladox 033d6584d - Update syslog.pp [23:58:53] [02puppet] 07paladox synchronize pull request 03#2272: base::syslog: force hostname to resolve to ipv6 - 13https://git.io/JS7th [23:58:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.89, 6.81, 6.74 [23:59:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.59, 7.74, 7.76 [23:59:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JS7mt [23:59:20] [02miraheze/puppet] 07paladox 0319a75dc - Update mw101.yaml [23:59:21] [02puppet] 07paladox synchronize pull request 03#2272: base::syslog: force hostname to resolve to ipv6 - 13https://git.io/JS7th [23:59:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JS7mm [23:59:28] [02miraheze/puppet] 07paladox 039ea1690 - Update mw102.yaml [23:59:29] [02puppet] 07paladox synchronize pull request 03#2272: base::syslog: force hostname to resolve to ipv6 - 13https://git.io/JS7th [23:59:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:59:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JS7m3 [23:59:38] [02miraheze/puppet] 07paladox 03ef19b30 - Update mw111.yaml [23:59:39] [02puppet] 07paladox synchronize pull request 03#2272: base::syslog: force hostname to resolve to ipv6 - 13https://git.io/JS7th [23:59:42] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.55, 7.60, 7.35 [23:59:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JS7ms [23:59:47] [02miraheze/puppet] 07paladox 0323b61d8 - Update mw112.yaml [23:59:48] [02puppet] 07paladox synchronize pull request 03#2272: base::syslog: force hostname to resolve to ipv6 - 13https://git.io/JS7th [23:59:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JS7mc [23:59:55] [02miraheze/puppet] 07paladox 03215fbc7 - Update mw121.yaml [23:59:57] [02puppet] 07paladox synchronize pull request 03#2272: base::syslog: force hostname to resolve to ipv6 - 13https://git.io/JS7th