[00:00:05] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.319 second response time [00:00:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 0.62, 2.48, 3.84 [00:00:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 0.74, 4.95, 7.12 [00:00:44] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:00:45] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:00:45] PROBLEM - cp20 Stunnel Http for mon111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:00:46] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:00:51] PROBLEM - cp20 Stunnel Http for mon2 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:00:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 0.55, 4.64, 7.31 [00:01:01] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:01:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 0.59, 4.57, 7.23 [00:01:06] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:01:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 0.28, 3.41, 6.48 [00:01:17] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:01:20] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:01:21] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 0.46, 1.39, 1.92 [00:01:29] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:01:38] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:01:38] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:01:44] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:01:53] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:01:54] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:01:55] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:01:57] PROBLEM - cp21 Stunnel Http for mon111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:01:59] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:03] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:03] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 0.18, 2.41, 6.03 [00:02:05] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:06] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:14] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:15] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:19] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:19] PROBLEM - cp20 APT on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:20] That should probably be downtimesd [00:02:25] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 0.38, 3.40, 6.28 [00:02:30] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:36] PROBLEM - cp21 APT on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:37] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:50] What should? [00:02:57] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 0.24, 3.19, 6.45 [00:03:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 0.16, 3.09, 6.36 [00:03:09] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:03:14] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:03:17] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:03:29] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 207 failures. Last run 3 minutes ago with 207 failures. Failed resources (up to 3 shown): File[vault.agentisai.com_private],File[wiki.eversd.com],File[wiki.eversd.com_private],File[pt.polandballwiki.com] [00:03:41] the cp* alerts as they’ve been switched to new Infra puppet and so the checks from old Infra won’t work it seems [00:03:42] !log [paladox@mwtask111] finished deploy of {'world': True, 'force': True} to scsvg - SUCCESS in 1070s [00:03:44] PROBLEM - cp21 Stunnel Http for mon2 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:03:58] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:00] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 0.35, 1.31, 3.04 [00:04:14] !log [paladox@mwtask111] starting deploy of {'world': True, 'force': True} to scsvg [00:04:18] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:21] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [00:04:23] PROBLEM - mem121 Puppet on mem121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:27] MacFan4000: oh, I have that (mon2 alerts) ignored now. [00:04:28] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:41] PROBLEM - cp30 Disk Space on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:47] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:47] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:59] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:03] CosmicAlpha: I think cp* needs to be downtimes on old infra [00:05:08] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:12] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:12] PROBLEM - cp30 conntrack_table_size on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:12] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:14] PROBLEM - cp30 PowerDNS Recursor on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:14] downtimed* [00:05:15] doing [00:05:15] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 0.18, 0.78, 1.55 [00:05:16] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:22] PROBLEM - cp30 ferm_active on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:23] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:27] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:27] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:34] PROBLEM - cp31 Stunnel Http for mon111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:35] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:44] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:44] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:47] PROBLEM - cp31 NTP time on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:49] PROBLEM - cp31 ferm_active on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:57] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:00] PROBLEM - cp30 NTP time on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:05] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:06] PROBLEM - cp30 APT on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:13] PROBLEM - cp30 Stunnel Http for mon111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:13] PROBLEM - cp30 Stunnel Http for mon2 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:15] PROBLEM - cp31 conntrack_table_size on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:16] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:17] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:27] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:42] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 140s [00:07:00] [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/e8d74a721b9a...f3f4a683671c [00:07:01] [02miraheze/mw-config] 07paladox 031268270 - Revert "Another test" [00:07:03] [02miraheze/mw-config] 07paladox 03f3f4a68 - Revert "Use memcached servers directly for new infra" [00:07:24] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [00:08:02] !log [paladox@test101] starting deploy of {'config': True, 'force': True} to skip [00:08:03] !log [paladox@test101] finished deploy of {'config': True, 'force': True} to skip - SUCCESS in 0s [00:08:10] miraheze/mw-config - paladox the build passed. [00:08:14] MacFan4000: done [00:08:38] None of those are being logged as the bot died [00:08:57] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.230 second response time [00:09:00] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 95s [00:09:10] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.203 second response time [00:09:18] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 23580 bytes in 0.762 second response time [00:09:22] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [00:10:01] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.418 second response time [00:10:51] !log [@mw11] starting deploy of {'config': True} to ovlon [00:10:55] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 3s [00:11:14] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 112s [00:11:57] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.485 second response time [00:12:03] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.38, 6.94, 5.85 [00:12:07] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [00:12:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.36, 3.24, 3.02 [00:12:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 9.45, 5.67, 4.78 [00:12:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.59, 5.66, 5.36 [00:12:57] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.65, 6.41, 5.86 [00:12:58] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.89, 4.51, 4.07 [00:13:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.03, 6.51, 5.80 [00:13:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.17, 6.84, 5.80 [00:13:14] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.31, 3.50, 2.83 [00:13:32] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 85s [00:13:33] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:39] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 7.87, 4.48, 3.61 [00:13:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.73, 3.27, 2.58 [00:15:08] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.260 second response time [00:15:43] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.243 second response time [00:16:02] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.31, 2.05, 1.76 [00:16:15] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.716 second response time [00:16:58] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.03, 5.39, 4.55 [00:17:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.71, 5.21, 4.16 [00:18:17] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:58] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.88, 5.64, 4.75 [00:19:39] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.57, 4.93, 4.18 [00:19:39] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:02] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:16] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 1.540 second response time [00:20:38] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:58] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 7.21, 6.23, 5.08 [00:21:54] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.42, 1.97, 1.83 [00:21:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.04, 3.99, 3.38 [00:21:59] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.262 second response time [00:22:38] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 2.378 second response time [00:22:58] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.88, 5.71, 5.03 [00:23:20] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.90, 1.63, 1.36 [00:23:51] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.37, 2.16, 1.91 [00:23:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.96, 4.04, 3.47 [00:24:29] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:24:31] !log [@test3] starting deploy of {'config': True} to skip [00:24:32] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:24:58] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.26, 6.05, 5.25 [00:25:18] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.36, 1.54, 1.36 [00:25:39] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 7.03, 6.31, 5.01 [00:25:41] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:25:58] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.560 second response time [00:26:58] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.07, 5.62, 5.19 [00:26:59] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.77, 5.86, 5.02 [00:27:46] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.43, 1.90, 1.88 [00:28:46] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/1dded2c33a83 [00:28:48] [02miraheze/mw-config] 07paladox 031dded2c - Use ipv6 addresses for wgCdnServers [00:28:49] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [00:28:51] [02mw-config] 07paladox opened pull request 03#4381: Use ipv6 addresses for wgCdnServers - 13https://github.com/miraheze/mw-config/pull/4381 [00:29:46] miraheze/mw-config - paladox the build passed. [00:30:23] [02mw-config] 07paladox closed pull request 03#4381: Use ipv6 addresses for wgCdnServers - 13https://github.com/miraheze/mw-config/pull/4381 [00:30:24] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f3f4a683671c...95c6556b62a5 [00:30:26] [02miraheze/mw-config] 07paladox 0395c6556 - Use ipv6 addresses for wgCdnServers (#4381) [00:30:27] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [00:30:29] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [00:30:38] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [00:30:57] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.532 second response time [00:31:20] !log [paladox@test101] starting deploy of {'config': True} to skip [00:31:21] !log [paladox@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:31:29] miraheze/mw-config - paladox the build passed. [00:31:51] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:31:53] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:32:01] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 5.90, 6.65, 5.61 [00:32:22] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 5.87, 6.03, 5.53 [00:32:26] RECOVERY - mem121 Puppet on mem121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:35] PROBLEM - cp21 Stunnel Http for phab121 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 5456 bytes in 2.034 second response time [00:32:36] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw121 [00:32:43] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:32:46] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:32:48] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:32:49] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:32:51] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 133s [00:32:52] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:32:52] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:32:55] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:32:56] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:07] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:12] PROBLEM - cp20 Stunnel Http for phab121 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 5456 bytes in 2.042 second response time [00:33:19] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:20] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:21] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:22] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:23] PROBLEM - cp30 Stunnel Http for phab121 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 5456 bytes in 2.270 second response time [00:33:30] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [00:33:30] PROBLEM - cp31 Stunnel Http for phab121 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 5456 bytes in 2.811 second response time [00:33:39] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:42] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:42] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:58] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:34:17] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.41, 5.87, 5.53 [00:35:05] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [00:35:10] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://github.com/miraheze/mw-config/commit/269de6037509 [00:35:12] [02miraheze/mw-config] 07paladox 03269de60 - Move Profile stuff into LS [00:35:13] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [00:35:15] [02mw-config] 07paladox opened pull request 03#4382: Move Profile stuff into LS - 13https://github.com/miraheze/mw-config/pull/4382 [00:35:33] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:35:36] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [00:35:47] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:36:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.88, 3.79 [00:36:15] miraheze/mw-config - paladox the build passed. [00:36:15] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:36:28] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/269de6037509...dfcf0c4fd584 [00:36:29] [02miraheze/mw-config] 07paladox 03dfcf0c4 - Update LocalSettings.php [00:36:31] [02mw-config] 07paladox synchronize pull request 03#4382: Move Profile stuff into LS - 13https://github.com/miraheze/mw-config/pull/4382 [00:36:37] CosmicAlpha: ^ [00:36:45] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 3 backends are down. mw102 mw121 mw122 [00:36:46] it doesn't work in PhpAutoprepend [00:36:50] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.724 second response time [00:36:57] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw101 [00:37:03] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.953 second response time [00:37:05] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/dfcf0c4fd584...fb1357235406 [00:37:07] [02miraheze/mw-config] 07github-actions 03fb13572 - CI: lint code to MediaWiki standards [00:37:08] [02mw-config] 07github-actions[bot] synchronize pull request 03#4382: Move Profile stuff into LS - 13https://github.com/miraheze/mw-config/pull/4382 [00:37:18] paladox: feel free to, I just moved it to PhpAutoPrepend because it was more accurate, and it only stopped working like a week ago. [00:37:23] ok [00:37:25] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.765 second response time [00:37:25] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.632 second response time [00:37:30] miraheze/mw-config - paladox the build passed. [00:37:32] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.228 second response time [00:37:44] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.921 second response time [00:37:47] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.602 second response time [00:37:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7cb0a8dddb2c...6fb713b3ab54 [00:37:57] [02miraheze/puppet] 07paladox 036fb713b - Update php.pp [00:38:01] [02mw-config] 07paladox closed pull request 03#4382: Move Profile stuff into LS - 13https://github.com/miraheze/mw-config/pull/4382 [00:38:02] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [00:38:04] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [00:38:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/mw-config/compare/95c6556b62a5...bb06b2e0d432 [00:38:06] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [00:38:07] [02miraheze/mw-config] 07paladox 03bb06b2e - Move Profile stuff into LS (#4382) [00:38:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.51, 5.79, 5.54 [00:38:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.08, 4.42, 4.00 [00:38:32] !log [paladox@test101] starting deploy of {'config': True, 'force': True} to skip [00:38:33] !log [paladox@test101] finished deploy of {'config': True, 'force': True} to skip - SUCCESS in 0s [00:38:58] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:39:01] miraheze/mw-config - paladox the build passed. [00:39:04] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:39:05] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:39:11] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:39:27] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.311 second response time [00:39:40] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.020 second response time [00:39:41] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 95s [00:39:47] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.866 second response time [00:39:47] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 3.762 second response time [00:39:50] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.792 second response time [00:40:02] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.61, 5.64, 5.52 [00:40:13] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [00:41:03] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.662 second response time [00:41:05] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.996 second response time [00:41:05] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.647 second response time [00:41:09] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.319 second response time [00:41:16] !log [@mw11] starting deploy of {'config': True} to ovlon [00:41:19] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 3s [00:41:27] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.583 second response time [00:44:04] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:06] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:16] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:23] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [00:44:34] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.43, 1.78, 1.57 [00:44:34] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.576 second response time [00:45:13] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 0.78, 1.45, 1.69 [00:45:21] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.208 second response time [00:45:22] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 58s [00:45:24] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:45:24] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:45:26] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:45:38] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:45:38] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.024 second response time [00:45:44] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 1.922 second response time [00:45:56] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:46:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.98, 3.97 [00:46:04] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.598 second response time [00:46:05] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.028 second response time [00:46:14] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.877 second response time [00:46:30] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [00:46:33] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.24, 1.61, 1.53 [00:46:47] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw102 [00:47:22] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.360 second response time [00:47:22] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.922 second response time [00:47:37] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.986 second response time [00:47:44] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.48, 5.81, 5.55 [00:47:57] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.847 second response time [00:48:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.81, 4.07, 4.02 [00:48:22] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 4.083 second response time [00:48:46] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:48:50] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.038 second response time [00:49:10] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:23] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:49:33] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:49:44] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw122 [00:50:25] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.196 second response time [00:50:25] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw101 [00:50:32] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [00:50:40] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 9.538 second response time [00:50:45] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 18 backends are healthy [00:51:33] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.775 second response time [00:51:37] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.084 second response time [00:51:37] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.52, 5.60, 5.55 [00:51:39] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [00:51:42] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:51:44] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:51:45] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.663 second response time [00:52:21] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:26] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:52:32] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:53:29] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:53:36] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:53:40] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:53:50] !log [@test3] starting deploy of {'config': True} to skip [00:53:51] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:53:55] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:54:13] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:54:14] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:54:25] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [00:54:27] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.34, 2.26, 1.82 [00:54:31] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:54:33] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.732 second response time [00:54:33] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 9.702 second response time [00:54:47] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw102 mw122 [00:55:08] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:24] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.313 second response time [00:55:24] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.307 second response time [00:55:33] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [00:55:35] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [00:55:36] !log [paladox@mwtask111] starting deploy of {'l10n': True, 'force': True} to scsvg [00:55:51] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.306 second response time [00:55:54] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.869 second response time [00:56:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.25, 3.79, 3.94 [00:56:15] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:56:20] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [00:56:23] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.314 second response time [00:56:26] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw121 mw122 [00:56:30] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.015 second response time [00:57:25] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 8.882 second response time [00:57:26] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw102 [00:57:42] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.485 second response time [00:57:54] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.931 second response time [00:57:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.79, 4.79, 4.30 [00:57:59] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.286 second response time [00:58:03] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [00:58:06] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:58:06] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:58:30] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.560 second response time [00:58:47] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 18 backends are healthy [00:59:11] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.11, 1.83, 1.74 [00:59:20] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [00:59:23] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.243 second response time [00:59:33] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:59:37] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.39, 6.07, 5.64 [01:00:00] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:00:01] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:00:07] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:00:07] !log [paladox@mwtask111] finished deploy of {'l10n': True, 'force': True} to scsvg - SUCCESS in 271s [01:00:24] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:00:36] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:00:48] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:01:19] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.970 second response time [01:01:19] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 1.284 second response time [01:01:55] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.064 second response time [01:01:58] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.364 second response time [01:02:05] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.297 second response time [01:02:05] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:02:14] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:02:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:02:18] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:02:21] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:02:22] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [01:02:32] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.273 second response time [01:02:52] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.307 second response time [01:02:57] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:03:16] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [01:03:37] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.11, 5.74, 5.61 [01:04:09] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.373 second response time [01:04:16] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.402 second response time [01:04:19] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.406 second response time [01:04:26] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:04:26] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:04:47] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:04:58] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.706 second response time [01:05:03] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 107s [01:05:11] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.31, 1.94, 1.88 [01:05:28] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:05:37] has phab been fully migrated? [01:05:58] paladox: ^ [01:06:15] Nope not yet [01:06:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:06:25] i've been focusing on mw* performance problem [01:06:43] yeh, that's the important part [01:06:45] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:01] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:07:08] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:07:27] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:07:27] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.897 second response time [01:07:37] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.51, 6.10, 5.78 [01:07:54] I can switch it over [01:08:28] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:08:38] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:08:38] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:09:12] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.20, 1.46, 1.69 [01:09:22] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:09:37] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.24, 5.74, 5.68 [01:10:30] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [01:10:38] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.546 second response time [01:10:41] Hopefully it should be done before tomorrow is done and over with. [01:10:49] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw101 mw121 [01:11:01] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.909 second response time [01:11:11] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/839cbc5af5f8 [01:11:13] [02miraheze/dns] 07paladox 03839cbc5 - Move phabricator behind cache proxies [01:11:14] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [01:11:16] [02dns] 07paladox opened pull request 03#246: Move phabricator behind cache proxies - 13https://github.com/miraheze/dns/pull/246 [01:11:18] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw102 mw122 [01:11:24] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:40] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:11:46] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:12:20] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 6 backends are down. mw9 mw10 mw13 mw102 mw112 mw121 [01:12:27] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.024 second response time [01:12:46] [02dns] 07paladox closed pull request 03#246: Move phabricator behind cache proxies - 13https://github.com/miraheze/dns/pull/246 [01:12:48] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/79cb88f7acd0...91d954767f84 [01:12:49] [02miraheze/dns] 07paladox 0391d9547 - Move phabricator behind cache proxies (#246) [01:12:50] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.317 second response time [01:12:51] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.056 second response time [01:12:56] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [01:13:06] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.767 second response time [01:13:21] !log [paladox@mwtask111] starting deploy of {'l10n': True, 'force': True} to scsvg [01:13:31] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [01:13:43] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.295 second response time [01:13:45] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.347 second response time [01:13:55] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.905 second response time [01:14:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6fb713b3ab54...2207c145e090 [01:14:28] [02miraheze/puppet] 07paladox 032207c14 - phabricator: Switch db to db121 and unset read only [01:14:33] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 1.864 second response time [01:15:11] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.83, 1.83, 1.77 [01:15:43] [02puppet] 07Universal-Omega opened pull request 03#2318: MediaWiki: Remove temporarily hardcoded IPs - 13https://github.com/miraheze/puppet/pull/2318 [01:15:56] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 144s [01:16:47] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:17:13] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:17:13] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:17:22] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:17:25] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.681 second response time [01:17:33] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:17:58] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:18:08] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:18:11] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:18:13] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:18:14] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:18:20] [02puppet] 07paladox closed pull request 03#2318: MediaWiki: Remove temporarily hardcoded IPs - 13https://github.com/miraheze/puppet/pull/2318 [01:18:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2207c145e090...057069a59e93 [01:18:23] [02miraheze/puppet] 07Universal-Omega 03057069a - MediaWiki: Remove temporarily hardcoded IPs (#2318) [01:18:51] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.571 second response time [01:19:14] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.34, 1.88, 1.79 [01:19:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/057069a59e93...70dc93130ac2 [01:19:33] [02miraheze/puppet] 07paladox 0370dc931 - phabricator: Restrict https access to varnish [01:19:54] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.314 second response time [01:20:06] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.704 second response time [01:20:12] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 3.178 second response time [01:20:21] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [01:21:00] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.90, 1.58, 0.68 [01:21:14] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.69, 1.72, 1.73 [01:21:17] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.231 second response time [01:21:21] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.310 second response time [01:21:36] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.13, 5.65, 5.98 [01:21:46] PROBLEM - cloud12 SMART on cloud12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:50] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.395 second response time [01:21:53] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.018 second response time [01:21:53] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:03] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:04] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 9.051 second response time [01:22:05] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.152 second response time [01:22:45] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.328 second response time [01:23:14] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.922 second response time [01:23:24] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.307 second response time [01:23:33] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.330 second response time [01:23:58] [02puppet] 07Universal-Omega opened pull request 03#2319: stunnel: remove ovlon - 13https://github.com/miraheze/puppet/pull/2319 [01:24:03] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.077 second response time [01:24:09] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 9.894 second response time [01:24:09] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw101 mw102 [01:24:30] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.015 second response time [01:24:32] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:24:32] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:24:35] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw102 [01:24:40] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.314 second response time [01:25:12] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.26, 1.96, 1.82 [01:25:29] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:25:34] [02puppet] 07Universal-Omega synchronize pull request 03#2319: stunnel: remove ovlon - 13https://github.com/miraheze/puppet/pull/2319 [01:25:40] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.260 second response time [01:25:59] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:26:12] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:26:14] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:26:27] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.295 second response time [01:27:22] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:27:33] [02puppet] 07Universal-Omega synchronize pull request 03#2319: stunnel: remove ovlon - 13https://github.com/miraheze/puppet/pull/2319 [01:27:36] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostnam [01:27:36] t_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1089, in query return self.resolve(qname, rdtype, rdclass, tcp, source, File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifet [01:27:36] ile "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.408825397491455 seconds [01:27:38] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:28:00] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.949 second response time [01:28:02] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [01:28:03] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.069 second response time [01:28:03] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 4.875 second response time [01:28:30] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:28:30] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.251 second response time [01:28:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/70dc93130ac2...77c31516f240 [01:28:33] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 18 backends are healthy [01:28:33] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:28:34] [02miraheze/puppet] 07paladox 0377c3151 - varnish: Make phab.miraheze.wiki use phab121 [01:28:41] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 18991 bytes in 0.402 second response time [01:28:42] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [01:29:00] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:29:07] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 4.369 second response time [01:29:39] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.084 second response time [01:30:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/d0809575cef9 [01:30:07] [02miraheze/puppet] 07paladox 03d080957 - Add phab.miraheze.wiki to csp [01:30:07] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:30:08] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [01:30:10] [02puppet] 07paladox opened pull request 03#2320: Add phab.miraheze.wiki to csp - 13https://github.com/miraheze/puppet/pull/2320 [01:30:16] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:30:29] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.688 second response time [01:30:43] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.496 second response time [01:30:46] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [01:30:49] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:30:53] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:31:10] RECOVERY - phab2 phd on phab2 is OK: PROCS OK: 1 process with args 'phd' [01:31:11] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.40, 1.90, 1.86 [01:31:36] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 6.62, 5.54, 5.60 [01:32:16] RECOVERY - cloud12 SMART on cloud12 is OK: OK: [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean [01:32:17] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:32:22] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:32:22] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [01:32:25] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:28] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:32:34] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [01:32:38] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:48] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:32:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/77c31516f240...0974496038ad [01:32:58] [02miraheze/puppet] 07paladox 030974496 - Add phab.miraheze.wiki to csp (#2320) [01:32:59] [02puppet] 07paladox closed pull request 03#2320: Add phab.miraheze.wiki to csp - 13https://github.com/miraheze/puppet/pull/2320 [01:33:01] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [01:33:02] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [01:33:11] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 3.34, 2.54, 2.10 [01:33:25] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:34:00] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:34:13] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.509 second response time [01:34:22] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.243 second response time [01:34:31] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.097 second response time [01:34:32] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.604 second response time [01:34:36] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.432 second response time [01:34:36] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 1.772 second response time [01:34:43] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.312 second response time [01:34:50] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [01:34:54] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:34:58] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.330 second response time [01:35:04] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:35:31] PROBLEM - phab2 phab.miraheze.wiki HTTPS on phab2 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [01:35:57] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.373 second response time [01:36:20] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw111 mw122 [01:36:23] PROBLEM - phab2 phabricator.miraheze.org HTTPS on phab2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4189 bytes in 2.022 second response time [01:36:30] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [01:36:48] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.722 second response time [01:36:50] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.014 second response time [01:36:52] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/91d954767f84...d76569ef6251 [01:36:53] [02miraheze/dns] 07paladox 03d76569e - Update miraheze.wiki [01:36:56] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.863 second response time [01:37:01] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.348 second response time [01:37:30] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.746 second response time [01:37:32] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.671 second response time [01:37:34] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.163 second response time [01:38:07] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.573 second response time [01:38:26] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.393 second response time [01:38:29] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.318 second response time [01:38:31] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.556 second response time [01:38:41] [02puppet] 07Universal-Omega opened pull request 03#2321: prometheus: remove ovlon - 13https://github.com/miraheze/puppet/pull/2321 [01:38:45] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.543 second response time [01:38:58] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:39:00] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:39:02] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:39:23] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:39:37] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.22, 5.64, 5.49 [01:40:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/2c1478851022 [01:40:05] [02miraheze/puppet] 07paladox 032c14788 - Allow phab.miraheze.wiki to proxy through varnish [01:40:07] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [01:40:08] [02puppet] 07paladox opened pull request 03#2322: Allow phab.miraheze.wiki to proxy through varnish - 13https://github.com/miraheze/puppet/pull/2322 [01:40:21] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [01:40:28] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 18 backends are healthy [01:40:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2c1478851022...d3377d431d9e [01:40:43] [02miraheze/puppet] 07paladox 03d3377d4 - Update nginx.pp [01:40:45] [02puppet] 07paladox synchronize pull request 03#2322: Allow phab.miraheze.wiki to proxy through varnish - 13https://github.com/miraheze/puppet/pull/2322 [01:41:03] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [01:41:08] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.161 second response time [01:41:16] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:41:21] [02puppet] 07paladox closed pull request 03#2322: Allow phab.miraheze.wiki to proxy through varnish - 13https://github.com/miraheze/puppet/pull/2322 [01:41:22] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:41:23] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [01:41:24] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [01:41:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/0974496038ad...879628490192 [01:41:27] [02miraheze/puppet] 07paladox 038796284 - Allow phab.miraheze.wiki to proxy through varnish (#2322) [01:41:54] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:41:58] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.52, 1.55, 1.98 [01:42:27] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.020 second response time [01:42:41] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 9.822 second response time [01:42:42] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.462 second response time [01:42:43] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.955 second response time [01:42:43] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.373 second response time [01:42:50] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [01:42:52] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:43:07] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.270 second response time [01:43:08] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.371 second response time [01:43:13] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.963 second response time [01:43:13] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:43:22] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 3.327 second response time [01:43:29] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 9 backends are down. mw11 mw13 mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [01:43:35] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:43:37] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.63, 6.00, 5.70 [01:44:13] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.962 second response time [01:44:28] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.258 second response time [01:44:30] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:44:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/879628490192...6f7b25859d26 [01:44:34] [02miraheze/puppet] 07paladox 036f7b258 - Fix [01:44:44] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 4 backends are down. mw101 mw102 mw112 mw121 [01:44:53] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.066 second response time [01:44:58] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:45:05] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:45:06] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:45:20] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:45:21] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:45:27] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [01:45:36] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.34, 4.87, 5.67 [01:45:37] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.37, 6.14, 5.79 [01:45:46] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:45:59] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [01:46:17] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [01:46:21] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw101 [01:47:09] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [01:47:22] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [01:47:27] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:47:51] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.258 second response time [01:47:59] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:48:17] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:48:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6f7b25859d26...7058abf2b5f1 [01:48:23] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/nagios [01:48:23] ns/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - diamowiki.ga reverse DNS resolves to cp20.miraheze.org - CNAME OK [01:48:24] [02miraheze/puppet] 07paladox 037058abf - Update csp.yaml [01:48:28] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.518 second response time [01:48:36] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:48:37] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:48:40] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:48:50] [02puppet] 07paladox closed pull request 03#2321: prometheus: remove ovlon - 13https://github.com/miraheze/puppet/pull/2321 [01:48:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/7058abf2b5f1...234d48656b80 [01:48:53] [02miraheze/puppet] 07Universal-Omega 03234d486 - prometheus: remove ovlon (#2321) [01:49:00] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 0.58, 1.02, 1.85 [01:49:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/234d48656b80...e013a0fb1006 [01:49:03] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:49:04] [02miraheze/puppet] 07Universal-Omega 03e013a0f - check_reverse_dns: use `dns.resolver.Resolver.resolve()` (#2211) [01:49:06] [02puppet] 07paladox closed pull request 03#2211: check_reverse_dns: use `dns.resolver.Resolver.resolve()` - 13https://github.com/miraheze/puppet/pull/2211 [01:49:07] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:49:19] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [01:49:33] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.925 second response time [01:49:46] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.989 second response time [01:49:46] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:49:47] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.406 second response time [01:49:47] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 5.902 second response time [01:49:55] [02puppet] 07Universal-Omega opened pull request 03#2323: varnish: remove ovlon backends - 13https://github.com/miraheze/puppet/pull/2323 [01:50:18] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:26] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [01:50:39] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.362 second response time [01:50:40] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.518 second response time [01:50:43] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [01:50:44] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [01:51:00] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 0.72, 0.89, 1.70 [01:51:05] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:11] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.98, 1.81, 2.00 [01:51:17] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:36] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 6.22, 5.74, 5.78 [01:51:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e013a0fb1006...c1dc84faad4e [01:51:40] [02miraheze/puppet] 07paladox 03c1dc84f - Add phab.miraheze.wiki to image csp [01:51:50] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.54, 1.81, 1.86 [01:52:20] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:21] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [01:52:27] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:49] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:57] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:53:37] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.96, 5.85, 5.86 [01:53:40] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:53:47] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:53:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c1dc84faad4e...e09f02a78c8f [01:53:58] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:53:59] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:53:59] [02miraheze/puppet] 07paladox 03e09f02a - Add phab.miraheze.wiki to fonts csp [01:54:10] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:54:11] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:54:29] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:55:04] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:55:04] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:55:11] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.51, 2.63, 2.25 [01:55:16] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.988 second response time [01:55:37] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.19, 5.43, 5.67 [01:55:41] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.661 second response time [01:55:59] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.571 second response time [01:56:26] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/1c219360cc83 [01:56:27] [02miraheze/dns] 07paladox 031c21936 - Make grafana and icinga use new cache proxies [01:56:29] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [01:56:31] [02dns] 07paladox opened pull request 03#247: Make grafana and icinga use new cache proxies - 13https://github.com/miraheze/dns/pull/247 [01:56:34] [02dns] 07paladox edited pull request 03#247: Make grafana and icinga use cache proxies - 13https://github.com/miraheze/dns/pull/247 [01:56:45] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.653 second response time [01:56:54] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.936 second response time [01:56:58] [02dns] 07paladox closed pull request 03#247: Make grafana and icinga use cache proxies - 13https://github.com/miraheze/dns/pull/247 [01:56:59] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/d76569ef6251...5dfe214d33cb [01:57:01] [02miraheze/dns] 07paladox 035dfe214 - Make grafana and icinga use new cache proxies (#247) [01:57:11] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.546 second response time [01:57:17] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.398 second response time [01:57:31] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.350 second response time [01:57:34] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [01:57:36] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [01:57:36] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.59, 6.70, 6.11 [01:57:45] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 8.667 second response time [01:57:48] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.420 second response time [01:57:55] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.220 second response time [01:58:05] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw102 [01:58:19] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw112 [01:58:56] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.075 second response time [01:59:35] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.720 second response time [02:00:00] [02puppet] 07Universal-Omega opened pull request 03#2324: logsalmsg: remove mon2 support - 13https://github.com/miraheze/puppet/pull/2324 [02:00:25] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/cc9e6390fbc5 [02:00:56] [02miraheze/puppet] 07paladox 03cc9e639 - Put webamil behind varnish [02:00:57] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [02:00:59] [02puppet] 07paladox opened pull request 03#2325: Put webamil behind varnish - 13https://github.com/miraheze/puppet/pull/2325 [02:01:05] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.013 second response time [02:01:11] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:01:37] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.37, 4.27, 5.08 [02:01:39] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:01:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cc9e6390fbc5...eaa30dc0b40d [02:01:46] [02miraheze/puppet] 07paladox 03eaa30dc - Update stunnel.conf [02:01:48] [02puppet] 07paladox synchronize pull request 03#2325: Put webamil behind varnish - 13https://github.com/miraheze/puppet/pull/2325 [02:01:51] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.489 second response time [02:01:53] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:01:55] [02puppet] 07paladox edited pull request 03#2325: Put webmail behind varnish - 13https://github.com/miraheze/puppet/pull/2325 [02:02:00] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [02:02:03] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.172 second response time [02:02:05] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [02:02:07] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [02:02:09] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:02:15] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.556 second response time [02:02:17] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 18 backends are healthy [02:03:08] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.524 second response time [02:03:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/eaa30dc0b40d...3351a1c82475 [02:03:13] [02miraheze/puppet] 07paladox 033351a1c - Update roundcubemail.pp [02:03:15] [02puppet] 07paladox synchronize pull request 03#2325: Put webmail behind varnish - 13https://github.com/miraheze/puppet/pull/2325 [02:03:15] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [02:03:16] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.313 second response time [02:03:17] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:03:36] [02puppet] 07paladox closed pull request 03#2325: Put webmail behind varnish - 13https://github.com/miraheze/puppet/pull/2325 [02:03:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/e09f02a78c8f...4340f0231c85 [02:03:39] [02miraheze/puppet] 07paladox 034340f02 - Put webmail behind varnish (#2325) [02:03:40] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [02:03:42] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [02:03:46] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.50, 1.81, 2.00 [02:03:55] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:04:07] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [02:04:10] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:04:19] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.470 second response time [02:04:30] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.362 second response time [02:04:31] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.022 second response time [02:04:32] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.059 second response time [02:04:50] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/5dfe214d33cb...253327523ae8 [02:04:52] [02miraheze/dns] 07paladox 032533275 - Put webmail behind cache proxies [02:05:37] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.35, 4.73, 5.12 [02:05:53] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.301 second response time [02:06:00] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [02:06:03] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: connect to address localhost and port 8099: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:06:14] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.050 second response time [02:06:14] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.227 second response time [02:06:16] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.305 second response time [02:06:18] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: connect to address localhost and port 8099: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:06:20] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 13 backends are down. mw8 mw9 mw10 mw11 mw12 mw13 mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [02:06:30] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: connect to address localhost and port 8096: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:06:40] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.219 second response time [02:06:43] [02puppet] 07Universal-Omega opened pull request 03#2326: deploy-mediawiki: remove ovlon - 13https://github.com/miraheze/puppet/pull/2326 [02:07:00] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: connect to address localhost and port 8097: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:07:08] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: connect to address localhost and port 8097: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:07:09] RECOVERY - mail121 webmail.miraheze.org HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 200 OK - 6152 bytes in 0.137 second response time [02:07:11] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 0.95, 1.74, 2.00 [02:07:13] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: connect to address localhost and port 8096: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:07:18] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: connect to address localhost and port 8097: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:07:37] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.06, 4.31, 4.92 [02:07:48] PROBLEM - cp31 Stunnel Http for mon2 on cp31 is CRITICAL: connect to address localhost and port 8201: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:07:50] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: connect to address localhost and port 8092: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:07:52] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: connect to address localhost and port 8093: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:07:55] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: connect to address localhost and port 8087: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:07:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 0.88, 2.90, 3.93 [02:08:02] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [02:08:04] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: connect to address localhost and port 8088: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 0.83, 2.85, 4.00 [02:08:07] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: connect to address localhost and port 8087: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:07] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [02:08:08] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: connect to address localhost and port 8086: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:10] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: connect to address localhost and port 8093: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:13] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 13 backends are down. mw8 mw9 mw10 mw11 mw12 mw13 mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [02:08:15] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: connect to address localhost and port 8085: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:16] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: connect to address localhost and port 8098: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:20] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: connect to address localhost and port 8093: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:24] PROBLEM - cp21 Stunnel Http for mon2 on cp21 is CRITICAL: connect to address localhost and port 8201: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:26] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: connect to address localhost and port 8098: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:32] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: connect to address localhost and port 8098: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:32] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: connect to address localhost and port 8087: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:32] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: connect to address localhost and port 8085: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:39] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: connect to address localhost and port 8087: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:41] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: connect to address localhost and port 8088: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:47] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: connect to address localhost and port 8086: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:49] PROBLEM - cp21 Stunnel Http for mon111 on cp21 is CRITICAL: connect to address localhost and port 8300: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:50] PROBLEM - cp30 Stunnel Http for mon2 on cp30 is CRITICAL: connect to address localhost and port 8201: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:50] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: connect to address localhost and port 8088: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:55] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: connect to address localhost and port 8086: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:08:57] PROBLEM - cp20 Stunnel Http for mon2 on cp20 is CRITICAL: connect to address localhost and port 8201: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:09:05] PROBLEM - cp31 Stunnel Http for mon111 on cp31 is CRITICAL: connect to address localhost and port 8300: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:09:10] PROBLEM - cp30 Stunnel Http for mon111 on cp30 is CRITICAL: connect to address localhost and port 8300: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:09:16] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: connect to address localhost and port 8092: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:09:18] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: connect to address localhost and port 8094: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:09:21] PROBLEM - cp20 Stunnel Http for mon111 on cp20 is CRITICAL: connect to address localhost and port 8300: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:09:22] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [02:09:24] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: connect to address localhost and port 8086: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:09:24] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: connect to address localhost and port 8085: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:09:25] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: connect to address localhost and port 8092: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:09:26] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: connect to address localhost and port 8093: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:09:26] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: connect to address localhost and port 8092: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:09:26] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[stunnel4] [02:09:29] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: connect to address localhost and port 8088: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:09:34] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 45% [02:09:45] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: connect to address localhost and port 8085: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:09:59] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[stunnel4] [02:10:07] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 43% [02:10:19] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[stunnel4] [02:11:07] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[stunnel4] [02:11:11] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 0.55, 1.07, 1.65 [02:11:16] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 0.59, 3.92, 7.74 [02:11:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 1.10, 4.05, 7.77 [02:11:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 0.65, 2.13, 3.83 [02:11:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 0.42, 3.23, 7.14 [02:11:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.44, 1.52, 3.12 [02:12:04] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 0.28, 1.42, 3.15 [02:12:07] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 89% [02:12:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 0.79, 3.41, 7.29 [02:12:39] [02puppet] 07Universal-Omega synchronize pull request 03#2326: deploy-mediawiki: remove ovlon - 13https://github.com/miraheze/puppet/pull/2326 [02:12:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 0.81, 3.49, 7.50 [02:13:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 0.68, 3.13, 7.18 [02:13:34] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [02:13:36] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 0.29, 2.90, 5.52 [02:13:42] [02puppet] 07Universal-Omega synchronize pull request 03#2326: deploy-mediawiki: remove ovlon - 13https://github.com/miraheze/puppet/pull/2326 [02:13:55] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 0.33, 2.30, 6.33 [02:14:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4340f0231c85...f4db31654635 [02:14:05] [02miraheze/puppet] 07paladox 03f4db316 - Fix [02:14:07] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 6% [02:14:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 0.79, 2.55, 6.50 [02:14:46] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.024 second response time [02:14:48] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 0.89, 2.62, 6.69 [02:14:49] RECOVERY - cp21 Stunnel Http for mon111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 31581 bytes in 0.038 second response time [02:14:50] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.021 second response time [02:14:50] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.026 second response time [02:14:51] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.325 second response time [02:14:54] RECOVERY - cp31 Stunnel Http for mon111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 31581 bytes in 0.364 second response time [02:14:55] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.009 second response time [02:14:55] RECOVERY - cp20 Stunnel Http for mon2 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35416 bytes in 0.021 second response time [02:14:58] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.025 second response time [02:15:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 1.04, 2.45, 6.44 [02:15:03] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.018 second response time [02:15:07] [02puppet] 07Universal-Omega opened pull request 03#2327: site: remove ovlon MediaWiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2327 [02:15:09] RECOVERY - cp30 Stunnel Http for mon111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 31581 bytes in 0.339 second response time [02:15:09] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.336 second response time [02:15:11] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.306 second response time [02:15:13] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.022 second response time [02:15:15] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.028 second response time [02:15:17] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.012 second response time [02:15:17] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 0.85, 2.13, 6.12 [02:15:20] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.314 second response time [02:15:21] RECOVERY - cp20 Stunnel Http for mon111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 31573 bytes in 0.041 second response time [02:15:21] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.006 second response time [02:15:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 0.74, 2.30, 6.20 [02:15:22] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.306 second response time [02:15:23] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.323 second response time [02:15:24] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.006 second response time [02:15:25] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.312 second response time [02:15:25] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [02:15:26] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.007 second response time [02:15:30] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [02:15:32] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.350 second response time [02:15:34] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 35% [02:15:36] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 2.27, 2.45, 5.02 [02:15:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.13, 1.50, 3.17 [02:15:39] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.320 second response time [02:15:39] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 19 backends are healthy [02:15:41] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.366 second response time [02:15:42] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.313 second response time [02:15:44] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.155 second response time [02:15:46] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.188 second response time [02:15:46] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.315 second response time [02:15:48] RECOVERY - cp31 Stunnel Http for mon2 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 35370 bytes in 0.325 second response time [02:15:48] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.320 second response time [02:15:50] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.327 second response time [02:15:58] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [02:15:59] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 8% [02:16:00] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 19 backends are healthy [02:16:01] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.303 second response time [02:16:01] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.407 second response time [02:16:04] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.009 second response time [02:16:06] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.005 second response time [02:16:08] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.010 second response time [02:16:09] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 19 backends are healthy [02:16:09] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.458 second response time [02:16:10] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.019 second response time [02:16:11] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.063 second response time [02:16:13] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.774 second response time [02:16:17] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.318 second response time [02:16:18] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.315 second response time [02:16:21] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.614 second response time [02:16:22] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.592 second response time [02:16:22] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [02:16:22] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 19 backends are healthy [02:16:24] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.316 second response time [02:16:25] RECOVERY - cp21 Stunnel Http for mon2 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35370 bytes in 0.014 second response time [02:16:28] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.371 second response time [02:16:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/56d56164d64d [02:16:32] [02miraheze/puppet] 07paladox 0356d5616 - Make matomo use mon111 [02:16:32] RECOVERY - cp30 Stunnel Http for mon2 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 35370 bytes in 0.315 second response time [02:16:32] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.006 second response time [02:16:33] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.005 second response time [02:16:33] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.319 second response time [02:16:34] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [02:16:35] [02puppet] 07paladox opened pull request 03#2328: Make matomo use mon111 - 13https://github.com/miraheze/puppet/pull/2328 [02:16:38] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.476 second response time [02:16:39] [02puppet] 07paladox closed pull request 03#2328: Make matomo use mon111 - 13https://github.com/miraheze/puppet/pull/2328 [02:16:41] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [02:16:42] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [02:16:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f4db31654635...f7dab6a17c25 [02:16:45] [02miraheze/puppet] 07paladox 03f7dab6a - Make matomo use mon111 (#2328) [02:17:07] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [02:17:22] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 4% [02:18:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f7dab6a17c25...9340fc17c6f4 [02:18:06] [02miraheze/puppet] 07paladox 039340fc1 - Make matomo use db111 (db) [02:18:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 9.84, 6.21, 7.04 [02:18:34] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:18:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.52, 6.63, 7.35 [02:19:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.63, 7.07, 7.37 [02:19:15] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.80, 1.50, 1.56 [02:19:15] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:19:18] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.00, 7.71, 7.46 [02:19:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.95, 7.53, 7.46 [02:19:31] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:19:36] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 6.10, 4.54, 5.30 [02:19:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.01, 3.62, 3.66 [02:19:37] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.70, 5.20, 4.71 [02:19:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.55, 7.87, 7.31 [02:19:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.65, 3.14, 3.10 [02:20:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.90, 3.07, 3.11 [02:20:33] [02puppet] 07Universal-Omega opened pull request 03#2329: hieradata: delete ovlon mediawiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2329 [02:20:39] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 8.769 second response time [02:21:21] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.810 second response time [02:21:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.73, 4.30, 5.11 [02:22:04] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.64, 2.08, 1.92 [02:22:18] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.44, 3.87, 3.40 [02:22:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9340fc17c6f4...d8f0074b6b03 [02:22:58] [02miraheze/puppet] 07paladox 03d8f0074 - Make matomo use ldap111 [02:23:34] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.65, 1.53, 1.54 [02:24:00] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.46, 3.94, 4.87 [02:24:02] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.75, 5.58, 5.04 [02:24:32] PROBLEM - mon111 Current Load on mon111 is CRITICAL: CRITICAL - load average: 14.33, 7.87, 4.17 [02:24:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.49, 3.28 [02:24:38] [02puppet] 07Universal-Omega opened pull request 03#2330: hieradata common: use puppet111 by default - 13https://github.com/miraheze/puppet/pull/2330 [02:24:58] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:25:02] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:25:09] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:25:12] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:25:14] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [02:25:19] CosmicAlpha: https://github.com/miraheze/MirahezeDebug/pull/6 could probably be merged now [02:25:20] [url] Switch servers to SCSVG by Universal-Omega · Pull Request #6 · miraheze/MirahezeDebug · GitHub | github.com [02:25:30] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:37] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:41] [02MirahezeDebug] 07Universal-Omega closed pull request 03#6: Switch servers to SCSVG - 13https://github.com/miraheze/MirahezeDebug/pull/6 [02:25:43] [02miraheze/MirahezeDebug] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeDebug/compare/2d158d7a6cf7...8fcb8eeca7bb [02:25:44] [02miraheze/MirahezeDebug] 07Universal-Omega 038fcb8ee - Switch servers to SCSVG (#6) [02:25:46] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:25:46] [02miraheze/MirahezeDebug] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [02:25:47] [02MirahezeDebug] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/MirahezeDebug [02:25:50] ty [02:25:54] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:25:55] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:26:02] no problem [02:26:34] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 6.47, 4.61, 2.72 [02:26:39] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:26:40] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.30, 1.91, 1.92 [02:27:49] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23573 bytes in 9.658 second response time [02:28:16] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.016 second response time [02:28:23] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:28:31] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 2.83, 3.93, 2.69 [02:28:36] looks like we're getting close, only 4 open subtasks left [02:28:39] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:28:44] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:28:45] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:28:51] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:28:51] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:28:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.15, 5.39, 4.07 [02:28:58] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:29:10] !log [paladox@mwtask111] starting deploy of {'l10n': True, 'force': True} to scsvg [02:30:02] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.13, 2.24, 1.87 [02:30:17] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.80, 5.10, 4.99 [02:30:23] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.162 second response time [02:30:24] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.058 second response time [02:30:33] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [02:30:40] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.08, 1.96, 1.93 [02:30:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:31:03] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw9 mw12 [02:31:49] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.825 second response time [02:32:01] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.92, 1.94, 1.79 [02:32:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.18, 6.32, 5.48 [02:32:08] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23756 bytes in 6.257 second response time [02:32:23] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 110s [02:32:31] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.156 second response time [02:32:38] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:32:38] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:32:39] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:44] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.022 second response time [02:32:49] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.329 second response time [02:32:55] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.017 second response time [02:32:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:33:02] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [02:33:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/cd419bd1be2e [02:33:07] [02miraheze/mw-config] 07Universal-Omega 03cd419bd - remove ovlon [02:33:08] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [02:33:10] [02mw-config] 07Universal-Omega opened pull request 03#4383: remove ovlon - 13https://github.com/miraheze/mw-config/pull/4383 [02:33:31] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:34:03] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 61s [02:34:07] miraheze/mw-config - Universal-Omega the build passed. [02:34:14] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw111 mw112 [02:34:17] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.282 second response time [02:34:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cd419bd1be2e...8491789e4777 [02:34:31] [02miraheze/mw-config] 07Universal-Omega 038491789 - Update LocalSettings.php [02:34:33] [02mw-config] 07Universal-Omega synchronize pull request 03#4383: remove ovlon - 13https://github.com/miraheze/mw-config/pull/4383 [02:34:50] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:34:51] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:35:24] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [02:35:29] miraheze/mw-config - Universal-Omega the build passed. [02:35:34] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.265 second response time [02:35:34] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.386 second response time [02:35:41] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.837 second response time [02:36:04] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 3.62, 2.18, 1.88 [02:36:10] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:36:13] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 19 backends are healthy [02:36:39] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:36:49] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.021 second response time [02:36:49] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.882 second response time [02:36:50] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.202 second response time [02:36:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:36:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8491789e4777...f683a9919ac1 [02:36:53] [02miraheze/mw-config] 07Universal-Omega 03f683a99 - Update Database.php [02:36:54] [02mw-config] 07Universal-Omega synchronize pull request 03#4383: remove ovlon - 13https://github.com/miraheze/mw-config/pull/4383 [02:37:06] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:37:07] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:37:14] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:37:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f683a9919ac1...a1c4a4a20ad4 [02:37:49] [02miraheze/mw-config] 07Universal-Omega 03a1c4a4a - Update LocalSettings.php [02:37:50] [02mw-config] 07Universal-Omega synchronize pull request 03#4383: remove ovlon - 13https://github.com/miraheze/mw-config/pull/4383 [02:37:51] miraheze/mw-config - Universal-Omega the build passed. [02:37:51] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 147s [02:38:12] !log [paladox@mwtask111] finished deploy of {'l10n': True, 'force': True} to scsvg - SUCCESS in 541s [02:38:19] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:34] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:34] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [02:38:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.65, 3.75 [02:38:43] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:38:50] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.028 second response time [02:38:51] miraheze/mw-config - Universal-Omega the build passed. [02:38:56] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw122 [02:39:06] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.165 second response time [02:39:11] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:39:15] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.019 second response time [02:39:16] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:39:24] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.072 second response time [02:39:24] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23756 bytes in 2.115 second response time [02:39:45] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:40:04] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.54, 1.83, 1.81 [02:40:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:40:40] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.252 second response time [02:40:41] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 1.641 second response time [02:40:58] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.624 second response time [02:41:07] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 152s [02:41:55] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.612 second response time [02:42:04] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.914 second response time [02:42:09] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [02:42:24] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.405 second response time [02:42:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.34, 3.80, 3.93 [02:43:02] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:43:04] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.392 second response time [02:43:25] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.903 second response time [02:43:25] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.112 second response time [02:43:27] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 78s [02:43:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:43:50] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.45, 5.25, 5.03 [02:43:55] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.706 second response time [02:43:56] [02mw-config] 07Universal-Omega commented on pull request 03#4378: [22:53] Revert "DC-Switch: Stop uploads" - 13https://github.com/miraheze/mw-config/pull/4378#issuecomment-1013592424 [02:43:58] [02mw-config] 07Universal-Omega closed pull request 03#4378: [22:53] Revert "DC-Switch: Stop uploads" - 13https://github.com/miraheze/mw-config/pull/4378 [02:44:01] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4353-RhinosF1-patch-2 [02:44:02] [02mw-config] 07Universal-Omega deleted branch 03revert-4353-RhinosF1-patch-2 - 13https://github.com/miraheze/mw-config [02:44:07] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw121 [02:44:31] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.79, 1.89, 2.00 [02:44:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.81, 3.52, 3.60 [02:44:42] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [02:44:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.45, 4.40, 4.10 [02:45:22] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:45:25] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:45:25] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:45:29] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.511 second response time [02:45:45] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.49, 4.59, 4.81 [02:46:02] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 19 backends are healthy [02:46:03] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.14, 1.49, 1.67 [02:46:27] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 104s [02:46:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.40, 3.55 [02:46:40] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 19 backends are healthy [02:46:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.93, 3.97 [02:46:47] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:15] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:47:42] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:47:55] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:48:01] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw112 [02:48:20] what [02:48:32] > its not relaying to discord [02:48:38] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.22, 3.82, 3.91 [02:48:50] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.230 second response time [02:48:56] what isn't, the messages from here? [02:49:05] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:49:06] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:49:17] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:49:23] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:25] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.313 second response time [02:49:26] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.023 second response time [02:49:29] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.214 second response time [02:49:40] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:50:01] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw101 mw122 [02:50:10] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [02:50:29] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.49, 2.06, 2.01 [02:50:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 3.51, 3.53 [02:50:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.50, 3.33, 3.71 [02:50:52] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.604 second response time [02:51:26] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [02:51:38] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.81, 5.20, 5.00 [02:51:45] > https://cdn.discordapp.com/attachments/808001911868489748/931742420993970186/unknown.png [02:51:48] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.336 second response time [02:51:50] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [02:51:51] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:51:58] > it didnt relay todiscord [02:52:02] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.07, 1.93, 1.80 [02:52:04] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:52:09] those don't relay. [02:52:15] intentionally. [02:52:24] to not spam the relay discord side. [02:52:26] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:52:31] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:52:32] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:52:48] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:52:57] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 19 backends are healthy [02:53:04] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 97s [02:54:01] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 19 backends are healthy [02:54:06] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:54:41] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw112 [02:55:12] !log [@mwtask111] starting deploy of {'config': True} to scsvg [02:55:14] !log [@mwtask111] DEPLOY ABORTED: Canary check failed for mw101 [02:55:17] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:56:01] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.537 second response time [02:56:03] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.76, 1.95, 1.85 [02:56:19] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.310 second response time [02:56:23] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.013 second response time [02:56:26] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [02:56:31] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.328 second response time [02:56:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.27, 4.06, 3.87 [02:56:45] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.189 second response time [02:56:53] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.319 second response time [02:56:54] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw122 [02:57:16] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.336 second response time [02:57:17] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.702 second response time [02:57:32] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.496 second response time [02:57:38] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.670 second response time [02:57:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.10, 4.98, 5.79 [02:57:41] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.507 second response time [02:57:47] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:57:47] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 8.544 second response time [02:57:52] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:57:56] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [02:58:07] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [02:58:09] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.910 second response time [02:58:10] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw102 mw112 [02:58:12] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.03, 2.13, 1.94 [02:58:13] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.351 second response time [02:58:29] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [02:58:31] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.018 second response time [02:58:43] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 19 backends are healthy [02:58:49] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.878 second response time [02:59:04] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 19 backends are healthy [02:59:33] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.973 second response time [03:00:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.16, 3.70, 3.76 [03:00:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.00, 3.76, 3.81 [03:00:58] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:00:59] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:01:19] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.235 second response time [03:01:22] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:01:38] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.60, 4.30, 4.77 [03:01:43] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:01:46] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:01:56] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:02:01] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:02:04] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 20515 bytes in 7.732 second response time [03:02:11] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.016 second response time [03:02:31] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:02:38] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 4 backends are down. mw101 mw102 mw111 mw112 [03:02:39] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.73, 1.54, 1.93 [03:03:01] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [03:03:02] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [03:03:14] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:03:27] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.373 second response time [03:03:33] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.323 second response time [03:03:35] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.211 second response time [03:03:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.09, 3.31, 3.98 [03:03:40] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.645 second response time [03:03:51] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 2.39, 4.43, 7.37 [03:04:12] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 69s [03:04:20] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.017 second response time [03:04:36] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.51, 2.61, 3.30 [03:04:42] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.821 second response time [03:04:43] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.966 second response time [03:04:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.92, 2.70, 3.36 [03:04:46] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 9.482 second response time [03:04:53] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.331 second response time [03:05:00] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 19 backends are healthy [03:05:00] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.896 second response time [03:05:01] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.552 second response time [03:05:01] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.703 second response time [03:05:25] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.246 second response time [03:05:41] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 6.28, 5.32, 5.58 [03:05:45] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 19 backends are healthy [03:05:51] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.15, 6.57, 7.80 [03:06:01] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 19 backends are healthy [03:06:14] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.27, 1.77, 1.86 [03:06:32] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:06:34] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 19 backends are healthy [03:07:08] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:07:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.86, 4.36, 4.22 [03:07:38] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:07:55] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:07:58] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:08:03] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:08:07] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:08:08] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.227 second response time [03:08:14] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:08:15] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.826 second response time [03:08:16] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.458 second response time [03:08:17] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 4.054 second response time [03:08:30] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.754 second response time [03:08:31] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [03:08:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.43, 3.50 [03:08:41] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.89, 3.83, 3.72 [03:08:46] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.102 second response time [03:09:07] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:09:24] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:09:39] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.54, 5.58, 5.09 [03:09:51] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:09:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:09:57] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:09:59] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.491 second response time [03:10:06] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:10:23] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 112s [03:10:39] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.28, 4.34, 3.91 [03:11:01] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.816 second response time [03:11:36] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.23, 5.88, 5.79 [03:11:38] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.585 second response time [03:12:05] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.178 second response time [03:12:17] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.829 second response time [03:12:22] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.784 second response time [03:12:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:12:32] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:13:19] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.999 second response time [03:13:36] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 6.62, 6.03, 5.85 [03:13:38] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:13:52] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 3.467 second response time [03:14:07] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:14:20] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 0.92, 1.42, 1.68 [03:14:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.16, 3.77, 3.58 [03:14:34] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:14:36] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.42, 1.49, 1.70 [03:14:37] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:14:38] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:02] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:24] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:15:31] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:15:37] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.83, 5.62, 5.36 [03:15:40] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.914 second response time [03:16:00] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw101 mw112 [03:16:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.63, 3.54 [03:16:24] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw122 [03:18:15] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.218 second response time [03:18:20] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.35, 1.86, 1.79 [03:18:25] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [03:18:31] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [03:18:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.05, 3.95, 3.96 [03:18:34] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:18:34] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.017 second response time [03:18:34] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.72, 1.69, 1.74 [03:18:35] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:18:36] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.312 second response time [03:18:45] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.310 second response time [03:19:00] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:19:31] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [03:19:53] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:20:01] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 19 backends are healthy [03:20:18] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.809 second response time [03:20:19] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.74, 1.94, 1.84 [03:20:21] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:20:29] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.931 second response time [03:20:33] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.22, 1.85, 1.79 [03:20:51] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw122 [03:21:08] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 1.316 second response time [03:21:27] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 19 backends are healthy [03:21:37] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.45, 4.87, 5.09 [03:21:38] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.322 second response time [03:21:52] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.014 second response time [03:22:03] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.316 second response time [03:22:06] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.016 second response time [03:22:44] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:22:57] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.974 second response time [03:22:59] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:23:02] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:23:13] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.276 second response time [03:23:14] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:34] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:23:58] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.776 second response time [03:23:59] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.447 second response time [03:24:11] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.968 second response time [03:24:13] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:24:21] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 19 backends are healthy [03:24:26] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:24:39] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.921 second response time [03:24:49] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:24:50] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 1.457 second response time [03:24:51] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:24:56] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:25:00] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.866 second response time [03:25:01] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:25:09] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:25:16] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.971 second response time [03:25:17] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:26:19] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.03, 1.47, 1.68 [03:26:31] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.821 second response time [03:26:34] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.41, 1.85, 1.85 [03:26:57] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 19 backends are healthy [03:27:40] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:27:44] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.82, 5.80, 5.36 [03:28:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.30, 3.73, 3.62 [03:28:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 3.65, 3.73 [03:28:37] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:28:38] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:28:45] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:29:46] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.33, 5.31, 5.25 [03:30:26] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 3.22, 1.96, 1.81 [03:30:43] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.791 second response time [03:30:47] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.488 second response time [03:31:03] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.548 second response time [03:31:13] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.015 second response time [03:31:31] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 4.603 second response time [03:32:01] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [03:32:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.71, 3.67, 3.78 [03:32:25] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.41, 1.62, 1.70 [03:32:29] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.58, 1.54, 1.70 [03:33:14] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.926 second response time [03:33:21] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:33:39] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:33:54] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:33:58] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:34:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.31, 3.72, 3.66 [03:34:34] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:35:05] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:35:09] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.954 second response time [03:35:21] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw112 [03:35:36] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.69, 5.35, 5.93 [03:35:46] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 4.661 second response time [03:35:58] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.880 second response time [03:35:58] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.388 second response time [03:36:01] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 19 backends are healthy [03:36:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.64, 4.16, 3.94 [03:36:28] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.05, 1.78, 1.75 [03:37:06] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.013 second response time [03:37:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/101201d24e7f [03:37:17] [02miraheze/puppet] 07paladox 03101201d - php: experimental change [03:37:19] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [03:37:20] [02puppet] 07paladox opened pull request 03#2331: php: experimental change - 13https://github.com/miraheze/puppet/pull/2331 [03:37:26] [02puppet] 07paladox commented on pull request 03#2331: php: experimental change - 13https://github.com/miraheze/puppet/pull/2331#issuecomment-1013600820 [03:37:28] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:37:35] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:37:36] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 6.71, 5.51, 5.89 [03:37:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 2.61, 5.05, 7.68 [03:37:43] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:37:49] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.35, 3.91, 3.88 [03:39:07] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.935 second response time [03:39:23] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:39:36] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.68, 5.52, 5.85 [03:39:38] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 3.187 second response time [03:40:17] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.253 second response time [03:40:21] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.76, 4.20, 3.98 [03:40:29] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.31, 2.07, 1.84 [03:41:36] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 6.70, 5.69, 5.85 [03:41:38] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.23, 4.74, 5.00 [03:41:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 8.55, 6.00, 7.41 [03:41:52] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.967 second response time [03:41:57] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.381 second response time [03:42:01] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw102 mw111 [03:42:03] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.413 second response time [03:42:04] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.956 second response time [03:42:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.66, 3.95, 3.91 [03:42:23] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw111 mw112 mw122 [03:42:28] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.90, 2.00, 1.84 [03:42:48] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.021 second response time [03:43:08] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 19 backends are healthy [03:43:24] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.337 second response time [03:43:40] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.90, 5.64, 5.81 [03:44:00] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:44:11] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:44:11] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:44:18] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.805 second response time [03:44:27] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.12, 1.86, 1.87 [03:44:27] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 3.07, 2.39, 2.00 [03:44:29] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:44:46] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.836 second response time [03:45:07] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.703 second response time [03:45:36] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 6.15, 5.73, 5.82 [03:45:40] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.05, 7.77, 7.95 [03:45:52] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw101 [03:46:17] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:46:19] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:46:42] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.257 second response time [03:47:11] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:47:28] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:47:38] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.30, 5.52, 5.73 [03:47:41] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:47:49] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:48:25] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.50, 1.24, 1.63 [03:48:53] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.024 second response time [03:48:59] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw101 [03:49:33] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:49:37] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 6.91, 5.98, 5.87 [03:49:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.12, 3.34, 4.00 [03:49:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 2.59, 4.78, 6.66 [03:50:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.98, 2.70, 3.24 [03:50:04] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.313 second response time [03:50:07] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.339 second response time [03:50:12] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.021 second response time [03:50:26] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:50:28] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:50:28] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.35, 1.86, 1.91 [03:50:38] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:50:40] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 2.923 second response time [03:50:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 1.76, 3.99, 7.39 [03:50:56] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 19 backends are healthy [03:51:32] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.312 second response time [03:52:07] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.874 second response time [03:52:11] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.716 second response time [03:52:13] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.32, 2.66, 3.29 [03:52:55] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.312 second response time [03:53:36] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.78, 5.68, 5.83 [03:53:43] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.021 second response time [03:53:56] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 4.779 second response time [03:54:33] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.439 second response time [03:54:33] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.786 second response time [03:54:36] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.321 second response time [03:54:42] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [03:54:43] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.308 second response time [03:54:48] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 3.19, 3.61, 6.46 [03:54:49] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.330 second response time [03:54:53] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 4 backends are down. mw102 mw112 mw121 mw122 [03:54:56] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.248 second response time [03:55:02] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.653 second response time [03:55:23] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.235 second response time [03:55:29] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.021 second response time [03:55:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.78, 2.40, 3.38 [03:55:51] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [03:55:51] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.017 second response time [03:56:09] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.315 second response time [03:56:13] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.317 second response time [03:56:31] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.021 second response time [03:56:33] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:56:34] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.538 second response time [03:56:34] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:57:27] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:57:38] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:57:56] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:57:58] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.325 second response time [03:57:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 2.76, 5.00, 7.79 [03:58:27] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:58:28] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:58:51] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:59:01] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:59:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 3.82, 4.74, 7.65 [03:59:10] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:59:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 3.03, 4.78, 7.79 [03:59:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 26.71, 12.74, 8.50 [03:59:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 4.75, 4.97, 7.76 [03:59:43] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:00:13] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:00:15] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:00:42] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:00:57] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:01:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a5a55ce5bc61 [04:01:12] [02miraheze/puppet] 07paladox 03a5a55ce - varnish: use round_robin [04:01:14] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [04:01:15] [02puppet] 07paladox opened pull request 03#2332: varnish: use round_robin - 13https://github.com/miraheze/puppet/pull/2332 [04:01:33] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.997 second response time [04:01:37] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 2.84, 3.95, 4.97 [04:01:41] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.185 second response time [04:01:55] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 3.69, 4.12, 6.78 [04:02:10] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:02:23] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 0.94, 1.41, 1.65 [04:02:28] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:02:32] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.216 second response time [04:02:34] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:02:41] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.020 second response time [04:02:41] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.042 second response time [04:03:12] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.454 second response time [04:03:13] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.082 second response time [04:03:17] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.051 second response time [04:03:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 3.69, 3.94, 6.73 [04:03:51] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 19 backends are healthy [04:04:01] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 19 backends are healthy [04:04:14] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.795 second response time [04:04:30] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.895 second response time [04:04:37] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.415 second response time [04:04:42] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.479 second response time [04:04:43] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.220 second response time [04:05:29] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:05:34] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:05:40] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:05:45] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:05:46] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.356 second response time [04:06:07] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.025 second response time [04:06:11] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.413 second response time [04:06:36] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.316 second response time [04:06:41] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.956 second response time [04:06:48] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.023 second response time [04:06:55] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [04:07:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 3.46, 4.53, 6.47 [04:07:07] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.018 second response time [04:07:31] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:07:31] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.508 second response time [04:07:32] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:07:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 20.24, 11.05, 8.85 [04:07:36] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.48, 5.81, 5.39 [04:07:39] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:07:51] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw101 mw112 [04:08:00] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw101 mw112 [04:08:12] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.743 second response time [04:08:41] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:08:51] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.128 second response time [04:08:55] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:08:55] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:09:03] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.648 second response time [04:09:06] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:09:14] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:09:19] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.330 second response time [04:09:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.51, 8.37, 7.64 [04:09:34] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:09:40] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.30, 5.46, 5.31 [04:09:41] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:09:43] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:10:00] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 19 backends are healthy [04:10:07] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:10:21] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 19 backends are healthy [04:11:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.90, 8.13, 7.38 [04:11:07] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:11:15] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.728 second response time [04:11:17] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:11:32] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:11:40] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.230 second response time [04:11:43] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.331 second response time [04:11:43] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.344 second response time [04:11:45] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:11:47] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.019 second response time [04:11:50] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.170 second response time [04:11:50] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.885 second response time [04:11:56] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:12:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.99, 3.78, 3.42 [04:12:41] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [04:12:44] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.330 second response time [04:12:47] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.248 second response time [04:12:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.99, 7.30, 6.13 [04:13:10] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.018 second response time [04:13:37] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.60, 4.43, 4.93 [04:13:51] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.304 second response time [04:13:56] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 19 backends are healthy [04:14:14] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.415 second response time [04:14:15] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.920 second response time [04:14:22] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 19 backends are healthy [04:14:22] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 8.046 second response time [04:14:46] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:14:49] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:15:34] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.402 second response time [04:15:40] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.435 second response time [04:16:06] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:16:06] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:16:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.85, 3.53 [04:16:13] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:16:16] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.927 second response time [04:16:18] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.762 second response time [04:16:32] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:14] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:37] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:17:37] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.60, 5.39, 4.53 [04:17:57] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw101 [04:18:02] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:18:59] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:19:39] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.86, 5.25, 4.60 [04:19:57] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.231 second response time [04:20:07] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:20:11] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.321 second response time [04:20:12] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.313 second response time [04:20:12] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:20:18] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [04:20:43] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:20:46] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 4.834 second response time [04:20:54] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:21:04] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:21:20] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.932 second response time [04:21:33] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.320 second response time [04:21:41] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.026 second response time [04:21:56] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 19 backends are healthy [04:21:58] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.629 second response time [04:22:10] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.377 second response time [04:22:42] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw112 mw121 [04:22:49] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 4.86, 7.05, 7.06 [04:22:54] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.306 second response time [04:23:00] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.610 second response time [04:23:07] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw112 [04:23:37] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.45, 4.87, 4.60 [04:24:54] !log temporarily disable InstantCommons on mw101 [04:25:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.51, 3.56, 3.06 [04:25:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 5.48, 6.71, 7.69 [04:25:10] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.021 second response time [04:25:15] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.021 second response time [04:25:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.96, 5.25, 4.92 [04:26:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:26:19] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [04:26:22] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [04:26:31] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 19 backends are healthy [04:26:48] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.50, 5.94, 6.59 [04:27:01] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:27:04] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:27:05] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:27:05] [02puppet] 07paladox closed pull request 03#2331: php: experimental change - 13https://github.com/miraheze/puppet/pull/2331 [04:27:07] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [04:27:09] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [04:27:12] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:27:17] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:27:18] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.023 second response time [04:27:22] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw112 [04:27:27] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.013 second response time [04:27:36] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.93, 4.87, 4.83 [04:27:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.79, 3.66 [04:28:18] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:29:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.71, 3.23 [04:29:07] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.086 second response time [04:29:11] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 8.971 second response time [04:29:30] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:29:31] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:30:21] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.509 second response time [04:30:26] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:30:27] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw112 mw122 [04:30:48] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:30:54] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:31:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.64, 5.20, 6.69 [04:31:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 3.86, 3.34 [04:31:25] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.018 second response time [04:31:35] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:31:50] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:31:53] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw121 [04:33:00] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 19 backends are healthy [04:33:14] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:33:14] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 19 backends are healthy [04:33:17] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:35:21] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:35:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.92, 3.85 [04:36:58] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw112 [04:36:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.72, 3.42, 3.33 [04:37:09] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw112 [04:38:12] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:38:20] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:38:29] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:38:47] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.258 second response time [04:38:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.79, 2.92, 3.16 [04:39:09] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [04:39:38] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.47, 5.12, 4.87 [04:40:32] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:41:01] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.918 second response time [04:41:24] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.369 second response time [04:41:25] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.832 second response time [04:41:29] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.321 second response time [04:41:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.31, 2.72, 3.33 [04:41:37] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.94, 4.97, 4.84 [04:41:54] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.644 second response time [04:42:07] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.236 second response time [04:42:22] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.41, 1.53, 1.33 [04:42:28] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.482 second response time [04:42:30] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.425 second response time [04:42:42] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:42:46] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:42:50] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 19 backends are healthy [04:43:11] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.388 second response time [04:44:01] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 19 backends are healthy [04:44:17] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.057 second response time [04:44:22] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.70, 1.61, 1.39 [04:44:35] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.317 second response time [04:44:35] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.788 second response time [04:44:56] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 19 backends are healthy [04:45:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.69, 7.63, 6.58 [04:45:17] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.675 second response time [04:45:24] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:45:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 3.78, 3.61 [04:45:37] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.27, 5.12, 4.93 [04:45:48] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.317 second response time [04:45:48] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 19 backends are healthy [04:46:18] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:46:35] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:46:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.30, 8.20, 6.43 [04:46:58] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:47:37] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.56, 4.99, 4.91 [04:47:40] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:49] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:48:53] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw112 mw122 [04:48:58] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:49:00] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:49:34] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.274 second response time [04:49:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.08, 3.99, 3.79 [04:49:46] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:49:48] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.335 second response time [04:49:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.50, 3.28 [04:49:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.80, 5.30, 4.78 [04:50:04] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [04:50:12] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:15] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:50:16] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:50:17] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:50:23] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:50:39] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:50:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 19 backends are healthy [04:50:53] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 20431 bytes in 0.318 second response time [04:50:59] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.313 second response time [04:51:07] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:09] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:51:09] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.096 second response time [04:51:17] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.922 second response time [04:51:17] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:51:28] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.223 second response time [04:51:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.81, 4.22, 3.91 [04:51:42] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:51:44] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.601 second response time [04:52:00] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 19 backends are healthy [04:52:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.00, 7.31, 5.72 [04:52:17] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.021 second response time [04:53:44] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 6.07, 5.83, 5.12 [04:53:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.92, 4.12, 3.59 [04:53:53] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:54:10] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:54:45] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:54:52] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:55:30] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.027 second response time [04:55:32] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:55:33] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:55:37] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.61, 5.39, 5.08 [04:55:38] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.34, 5.38, 5.04 [04:55:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.79, 3.52 [04:55:46] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:55:49] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:55:51] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.306 second response time [04:56:10] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:56:19] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.336 second response time [04:56:39] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 9.965 second response time [04:56:39] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.919 second response time [04:56:42] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.280 second response time [04:56:45] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 3 backends are down. mw102 mw111 mw122 [04:56:55] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.417 second response time [04:57:22] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.566 second response time [04:57:39] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 6.92, 5.92, 5.27 [04:57:48] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.163 second response time [04:58:05] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [04:58:19] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.510 second response time [04:58:30] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.31, 1.95, 1.61 [04:58:37] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.240 second response time [04:58:41] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 19 backends are healthy [04:58:55] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.252 second response time [04:59:06] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 4.622 second response time [04:59:43] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.964 second response time [04:59:44] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.705 second response time [04:59:44] [02mw-config] 07AgentIsai opened pull request 03#4384: Remove migration sitenotice - 13https://github.com/miraheze/mw-config/pull/4384 [04:59:46] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:59:56] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.585 second response time [05:00:01] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.891 second response time [05:00:05] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 19 backends are healthy [05:00:08] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.730 second response time [05:00:23] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:00:41] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.624 second response time [05:00:42] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [05:00:47] miraheze/mw-config - AgentIsai the build passed. [05:00:51] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:01:38] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.60, 4.93, 5.02 [05:01:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.96, 3.97, 3.64 [05:01:59] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:01:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:02:13] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:02:38] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 115s [05:03:13] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.841 second response time [05:03:22] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.762 second response time [05:03:25] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:03:30] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 8.240 second response time [05:04:30] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:04:35] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:04:42] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:04:43] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.647 second response time [05:05:05] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [05:05:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:05:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.98, 4.41, 3.50 [05:05:33] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.185 second response time [05:06:20] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.233 second response time [05:06:28] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 4.396 second response time [05:06:39] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 94s [05:06:43] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.560 second response time [05:06:46] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.470 second response time [05:07:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:07:01] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.573 second response time [05:07:09] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:07:39] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:07:48] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:07:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.05, 6.90, 5.79 [05:08:03] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:08:08] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:08:29] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:08:39] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:08:41] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:08:44] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:08:45] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:08:49] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:09:03] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:09:15] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:09:23] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:09:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:10:17] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 9.461 second response time [05:10:18] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.354 second response time [05:10:25] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.921 second response time [05:10:32] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.158 second response time [05:10:36] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.891 second response time [05:10:38] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bb06b2e0d432...31c2ca58c1d1 [05:10:39] [02miraheze/mw-config] 07paladox 0331c2ca5 - Update GlobalCache.php [05:10:39] !log [@mw11] starting deploy of {'config': True} to ovlon [05:10:43] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 3s [05:10:45] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.61, 2.00, 1.94 [05:10:45] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.894 second response time [05:10:51] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.533 second response time [05:10:53] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [05:11:37] miraheze/mw-config - paladox the build passed. [05:12:34] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 100s [05:12:43] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.932 second response time [05:12:51] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.481 second response time [05:13:01] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.360 second response time [05:13:10] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.707 second response time [05:13:28] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 4.573 second response time [05:13:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:13:37] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.22, 5.95, 5.73 [05:13:44] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:13:50] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:13:58] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:14:25] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:14:28] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 8.033 second response time [05:14:42] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:14:42] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:14:48] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.027 second response time [05:14:48] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:14:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:14:56] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:14:58] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:15:10] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:15:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.69, 3.51, 3.53 [05:15:22] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:15:27] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.070 second response time [05:16:03] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.757 second response time [05:16:04] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.578 second response time [05:16:05] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.963 second response time [05:16:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:16:42] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.43, 1.41, 1.67 [05:17:37] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.28, 6.12, 5.85 [05:18:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:18:13] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:23] !log [@test101] starting deploy of {'config': True} to skip [05:18:24] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [05:18:51] PROBLEM - phab2 phd on phab2 is CRITICAL: PROCS CRITICAL: 0 processes with args 'phd' [05:18:52] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:19:05] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 9.895 second response time [05:19:09] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.00, 3.21, 3.40 [05:19:37] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.46, 5.96, 5.84 [05:19:41] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.32, 3.31, 3.88 [05:20:03] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.353 second response time [05:20:30] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:20:47] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:21:00] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.746 second response time [05:21:14] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:21:18] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.216 second response time [05:21:24] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.322 second response time [05:21:34] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [05:21:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.23, 3.71, 3.96 [05:21:50] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:22:34] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:22:45] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:22:46] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:22:51] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:22:53] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.69, 1.74, 1.34 [05:22:58] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:23:00] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:23:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.31, 3.42 [05:23:13] RECOVERY - phab2 phd on phab2 is OK: PROCS OK: 1 process with args 'phd' [05:23:14] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:23:32] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:23:35] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:23:36] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:23:53] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:24:03] !log [@test3] starting deploy of {'config': True} to skip [05:24:04] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:24:04] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [05:24:45] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/updateSpecialPages.php (END - exit=0) [05:25:00] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.789 second response time [05:25:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.14, 3.17, 3.35 [05:25:06] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.195 second response time [05:25:28] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:25:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.91, 3.99 [05:25:37] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.29, 5.67, 5.70 [05:25:40] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:25:47] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:26:02] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 4.290 second response time [05:26:22] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:27:10] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.019 second response time [05:27:11] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.019 second response time [05:27:33] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.325 second response time [05:27:34] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.96, 4.52, 4.17 [05:27:37] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.10, 5.45, 5.62 [05:27:39] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 1.150 second response time [05:28:06] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 9.754 second response time [05:28:14] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:28:35] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.835 second response time [05:29:05] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.49, 1.79, 1.68 [05:29:11] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.83, 1.68, 1.52 [05:30:06] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:30:36] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:30:48] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:30:49] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.973 second response time [05:31:05] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.26, 1.88, 1.72 [05:31:40] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.184 second response time [05:31:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.85, 3.57, 3.85 [05:31:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.59, 3.92 [05:32:00] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:32:01] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.465 second response time [05:32:19] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.908 second response time [05:32:25] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.973 second response time [05:32:37] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.213 second response time [05:32:48] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:33:10] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 0.85, 1.54, 1.62 [05:33:33] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.530 second response time [05:33:48] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.26, 5.58, 5.58 [05:34:05] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.125 second response time [05:34:06] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 2.395 second response time [05:34:24] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.080 second response time [05:34:54] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.205 second response time [05:34:58] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 4.956 second response time [05:35:27] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.196 second response time [05:35:37] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.515 second response time [05:35:42] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.014 second response time [05:35:43] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.31, 5.44, 5.53 [05:36:01] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.520 second response time [05:36:23] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:36:46] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.527 second response time [05:38:02] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:39:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.70, 3.83 [05:40:29] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.833 second response time [05:40:38] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:40:44] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.177 second response time [05:40:51] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.502 second response time [05:41:00] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 1.626 second response time [05:41:12] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:41:37] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.38, 5.58, 5.51 [05:41:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.37, 2.82, 3.35 [05:41:43] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:41:59] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:42:00] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.328 second response time [05:42:20] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:42:45] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 8.853 second response time [05:43:15] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.910 second response time [05:43:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.75, 3.46, 3.72 [05:43:41] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.300 second response time [05:43:53] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:43:58] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.288 second response time [05:44:18] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.331 second response time [05:45:08] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.15, 1.71, 1.60 [05:45:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 3.83, 3.83 [05:47:08] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.64, 1.65, 1.59 [05:47:35] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 3.43, 3.38 [05:47:53] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [05:49:13] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:49:28] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:49:29] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:49:34] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:49:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.97, 3.89 [05:49:46] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:49:59] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:50:19] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 9.703 second response time [05:50:19] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.888 second response time [05:50:29] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.987 second response time [05:51:08] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:51:17] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.045 second response time [05:51:20] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:51:22] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:51:26] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.127 second response time [05:51:27] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.471 second response time [05:51:30] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:51:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.57, 3.49 [05:51:36] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.661 second response time [05:51:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.98, 3.90 [05:51:56] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.830 second response time [05:53:12] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 4.682 second response time [05:53:27] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.703 second response time [05:53:28] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.205 second response time [05:53:32] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.019 second response time [05:53:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.05, 4.10, 3.69 [05:53:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.21, 3.96, 3.92 [05:53:41] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.48, 5.80, 5.80 [05:53:47] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.107 second response time [05:53:55] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.191 second response time [05:54:44] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:54:54] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:55:34] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.50, 3.78, 3.62 [05:55:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.99, 4.37, 4.07 [05:55:37] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.06, 6.50, 6.05 [05:55:38] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.943 second response time [05:57:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.86, 4.41, 3.86 [05:57:53] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:58:10] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:58:11] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:58:12] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:58:13] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:58:59] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 2.375 second response time [05:59:00] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.588 second response time [06:00:09] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:00:23] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:01:38] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:01:59] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:02:02] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:02:27] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.985 second response time [06:04:25] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.590 second response time [06:04:36] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.325 second response time [06:04:37] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.744 second response time [06:04:38] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.481 second response time [06:05:13] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:05:51] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:05:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.19, 3.50, 3.32 [06:06:19] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:06:49] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:07:04] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:25] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.74, 1.74, 1.65 [06:07:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.08, 3.30, 3.26 [06:09:14] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.777 second response time [06:09:28] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.752 second response time [06:09:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.38, 3.63, 3.83 [06:10:14] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.494 second response time [06:10:32] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.018 second response time [06:10:55] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.482 second response time [06:11:26] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 0.90, 1.38, 1.53 [06:11:37] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.83, 5.32, 5.82 [06:12:56] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:13:01] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:13:05] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:13:05] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:13:09] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:13:36] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:13:47] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:13:47] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:13:55] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:14:00] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:14:04] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:15:36] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.876 second response time [06:15:38] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 7.27, 5.94, 5.92 [06:15:48] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.928 second response time [06:15:49] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 5.229 second response time [06:15:57] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.123 second response time [06:16:02] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 5.490 second response time [06:16:14] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.574 second response time [06:16:44] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:17:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.78, 3.60, 3.33 [06:17:16] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.89, 1.80, 1.47 [06:18:32] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.946 second response time [06:18:35] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.320 second response time [06:19:00] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.948 second response time [06:19:06] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.82, 3.19, 3.20 [06:19:10] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.055 second response time [06:19:12] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.326 second response time [06:19:37] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.07, 5.70, 5.84 [06:21:13] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.70, 2.03, 1.63 [06:21:26] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.85, 2.60, 3.24 [06:21:38] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.87, 6.60, 6.18 [06:23:05] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [06:23:06] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.240 second response time [06:23:06] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.247 second response time [06:23:23] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:23:33] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.024 second response time [06:24:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.76, 3.48, 3.29 [06:25:29] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:25:33] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.798 second response time [06:25:41] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:25:43] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:25:51] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.550 second response time [06:25:51] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.754 second response time [06:25:57] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 2.580 second response time [06:25:59] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.495 second response time [06:26:44] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:26:57] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:27:03] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:27:25] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.41, 3.93, 3.56 [06:27:31] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.65, 2.00, 1.65 [06:27:41] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.39, 5.61, 5.85 [06:29:33] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.68, 1.89, 1.65 [06:29:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.55, 3.88 [06:31:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.50, 4.09, 4.04 [06:32:00] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:32:23] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:32:31] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:32:39] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:32:45] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:32:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.39, 3.95, 3.55 [06:33:34] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.12, 1.70, 1.65 [06:33:58] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.596 second response time [06:34:02] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.096 second response time [06:34:06] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.722 second response time [06:34:27] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.393 second response time [06:34:35] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.951 second response time [06:34:43] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.681 second response time [06:34:49] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 8.720 second response time [06:34:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.81, 3.50, 3.43 [06:35:37] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.36, 4.17, 5.03 [06:35:48] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.800 second response time [06:36:05] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.919 second response time [06:36:19] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 9.974 second response time [06:37:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.35, 3.62, 3.72 [06:37:27] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.573 second response time [06:37:27] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.496 second response time [06:37:35] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.14, 1.92, 1.75 [06:38:19] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:38:28] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:39:05] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.71, 1.92, 1.93 [06:39:34] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.43, 1.77, 1.71 [06:40:17] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:40:28] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:40:34] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.732 second response time [06:40:34] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.279 second response time [06:40:50] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:06] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.11, 1.95, 1.94 [06:41:34] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 0.83, 1.50, 1.63 [06:43:05] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.96, 1.97, 1.94 [06:44:32] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:44:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.81, 3.29, 3.39 [06:44:57] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:45:01] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:45:01] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:45:01] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:45:08] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:46:29] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 2.937 second response time [06:46:49] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 9.855 second response time [06:47:03] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.811 second response time [06:47:08] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.745 second response time [06:47:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.21, 2.04, 1.98 [06:47:14] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.562 second response time [06:47:40] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.29, 1.94, 1.75 [06:47:41] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 7.32, 5.55, 5.10 [06:49:16] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.422 second response time [06:49:43] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.35, 5.39, 5.10 [06:51:14] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:51:31] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:51:37] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:51:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.69, 3.40, 3.38 [06:51:41] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:51:44] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.69, 2.00, 1.84 [06:53:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 4.02, 3.76 [06:54:56] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:55:00] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.117 second response time [06:55:02] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.618 second response time [06:55:03] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.824 second response time [06:55:13] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.315 second response time [06:55:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.80, 3.71 [06:55:37] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.43, 4.68, 4.92 [06:55:44] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:46] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:55:47] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:55:49] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.05, 1.44, 1.65 [06:55:58] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:57:54] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 9.031 second response time [06:59:20] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.333 second response time [06:59:31] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:59:34] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:59:36] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:59:36] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.283 second response time [06:59:40] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.322 second response time [06:59:58] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.009 second response time [07:00:00] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.187 second response time [07:01:12] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.29, 4.20, 3.88 [07:01:14] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.890 second response time [07:01:37] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.108 second response time [07:01:40] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.458 second response time [07:01:42] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.277 second response time [07:01:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.35, 3.89, 3.66 [07:02:00] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:02:04] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 5.153 second response time [07:02:04] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:02:04] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.550 second response time [07:02:09] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.985 second response time [07:02:10] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:02:15] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.127 second response time [07:02:17] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:03:03] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.19, 1.66, 1.95 [07:03:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.88, 3.69 [07:03:56] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.310 second response time [07:04:02] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.016 second response time [07:04:11] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.619 second response time [07:04:15] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.326 second response time [07:05:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.87, 3.84 [07:05:36] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:06:04] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:06:12] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:06:13] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:06:18] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:07:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.13, 4.52, 4.07 [07:07:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.22, 4.42, 3.88 [07:09:02] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.34, 1.89, 1.92 [07:09:17] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:09:25] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:09:48] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:09:51] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:09:59] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:10:00] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:10:08] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:11:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.53, 1.79, 1.88 [07:11:17] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:11:17] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:11:32] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:11:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.53, 3.82, 3.81 [07:13:34] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 4.524 second response time [07:13:37] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.988 second response time [07:13:41] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.907 second response time [07:13:51] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.690 second response time [07:13:55] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.757 second response time [07:13:58] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.709 second response time [07:14:08] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 3.996 second response time [07:14:20] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.875 second response time [07:14:29] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:14:34] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.623 second response time [07:14:41] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.156 second response time [07:14:42] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.594 second response time [07:14:43] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.525 second response time [07:15:29] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.884 second response time [07:15:31] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.120 second response time [07:16:06] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.342 second response time [07:17:06] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.43, 1.46, 1.69 [07:19:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:19:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.83, 3.99 [07:21:36] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.12, 2.76, 3.31 [07:23:33] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 4.80, 5.16, 4.40 [07:25:30] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 3.99, 4.69, 4.32 [07:27:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.35, 4.11, 3.96 [07:27:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 3.81, 3.60 [07:29:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb [07:29:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [07:30:17] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:30:44] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:30:45] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:30:46] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:31:49] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:33:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.41, 3.43, 3.58 [07:35:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.73, 3.66 [07:37:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.59, 3.63 [07:41:58] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:42:02] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:42:37] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:43:04] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:43:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.80, 3.67 [07:43:59] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:45:07] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 9.710 second response time [07:45:09] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.997 second response time [07:45:20] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.111 second response time [07:45:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.82, 3.47, 3.57 [07:45:40] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.932 second response time [07:45:46] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.188 second response time [07:46:00] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.300 second response time [07:46:10] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.563 second response time [07:46:11] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 7.645 second response time [07:46:41] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.118 second response time [07:46:49] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.054 second response time [07:49:41] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:49:44] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:49:47] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:50:06] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:50:10] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:51:08] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:51:36] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.315 second response time [07:51:39] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:45] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.029 second response time [07:53:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:53:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:55:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.77, 3.62, 3.90 [07:55:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.75, 2.85, 3.30 [07:56:15] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 6.73, 5.63, 4.89 [07:57:10] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2607:5300:201:3100::929a/cpweb [07:58:18] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 5.75, 5.63, 4.98 [07:59:43] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.765 second response time [08:00:08] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [08:00:08] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.557 second response time [08:00:12] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.785 second response time [08:02:28] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 6.85, 5.81, 5.16 [08:02:55] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.029 second response time [08:03:04] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.193 second response time [08:03:15] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:05:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.72, 4.58, 4.07 [08:05:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.50, 3.82, 3.50 [08:07:19] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [08:07:54] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:08:22] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:09:20] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:09:31] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:09:36] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:09:49] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:10:53] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:10:58] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:11:21] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:11:23] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:11:25] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:11:31] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:11:43] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:11:47] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:13:31] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.479 second response time [08:13:43] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.392 second response time [08:13:44] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.322 second response time [08:14:12] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 4.274 second response time [08:14:51] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.627 second response time [08:15:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.79, 3.79, 3.74 [08:15:37] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.075 second response time [08:15:39] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.105 second response time [08:15:49] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.834 second response time [08:15:59] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.639 second response time [08:16:02] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.398 second response time [08:16:03] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.779 second response time [08:16:38] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:17:43] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.780 second response time [08:18:41] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.577 second response time [08:20:04] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:20:06] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:20:12] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:21:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.21, 3.44 [08:22:06] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:22:12] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 1.90, 3.90, 5.56 [08:22:23] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:22:26] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:22:32] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:23:03] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:23:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.15, 3.02, 3.33 [08:23:24] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:23:43] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:23:50] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:24:01] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:24:38] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:26:07] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 2.38, 3.17, 4.89 [08:27:58] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:28:08] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:29:42] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:29:52] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:29:54] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.790 second response time [08:30:03] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 8.164 second response time [08:30:06] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.913 second response time [08:30:14] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 8.745 second response time [08:30:31] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.057 second response time [08:30:41] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:30:41] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.934 second response time [08:30:44] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:30:51] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.152 second response time [08:30:53] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 8.256 second response time [08:30:56] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:31:10] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:31:12] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.36, 2.64, 1.55 [08:31:27] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.642 second response time [08:31:36] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:31:42] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.584 second response time [08:31:50] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.766 second response time [08:31:54] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.826 second response time [08:31:58] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.312 second response time [08:31:58] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.33, 2.04, 1.67 [08:32:02] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.504 second response time [08:32:19] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.314 second response time [08:32:31] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.313 second response time [08:32:40] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.311 second response time [08:32:55] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.172 second response time [08:32:57] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 0.899 second response time [08:33:03] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.478 second response time [08:33:12] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.50, 1.86, 1.39 [08:33:13] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 6.479 second response time [08:33:43] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.566 second response time [08:33:57] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.40, 1.82, 1.64 [08:34:47] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.584 second response time [08:34:50] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.430 second response time [08:35:10] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.84, 1.45, 1.29 [08:39:55] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.10, 1.47, 1.55 [08:40:29] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:40:57] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:41:00] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:41:26] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:41:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.21, 3.31, 3.94 [08:41:58] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:41:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.50, 3.41 [08:43:28] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:43:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.65, 4.30, 4.23 [08:43:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.10, 3.74, 3.52 [08:45:07] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.120 second response time [08:45:07] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.678 second response time [08:46:08] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.560 second response time [08:46:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.56, 3.49, 3.90 [08:46:48] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.878 second response time [08:47:41] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23572 bytes in 5.545 second response time [08:47:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.87, 3.62 [08:48:38] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.45, 4.07, 4.05 [08:50:40] [02mw-config] 07Reception123 closed pull request 03#4384: Remove migration sitenotice - 13https://github.com/miraheze/mw-config/pull/4384 [08:50:42] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/31c2ca58c1d1...573393068bfc [08:50:43] [02miraheze/mw-config] 07AgentIsai 035733930 - Remove migration sitenotice (#4384) [08:51:39] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:51:47] miraheze/mw-config - Reception123 the build passed. [08:51:55] !log [@mwtask111] starting deploy of {'config': True} to scsvg [08:52:27] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 31s [08:52:34] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.69, 3.90 [08:53:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:53:06] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:53:40] !log [@test3] starting deploy of {'config': True} to skip [08:53:41] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [08:53:54] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 3.52, 2.36, 1.83 [08:53:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:54:16] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2607:5300:201:3100::5ebc/cpweb [08:54:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:55:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:55:21] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:55:27] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:55:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.76, 3.98 [08:56:12] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:56:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.41, 3.80, 3.88 [08:57:15] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.105 second response time [08:57:23] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.137 second response time [08:57:26] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.635 second response time [08:57:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.70, 4.09, 4.08 [08:59:54] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.30, 1.89, 1.82 [09:00:03] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:00:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 4.00, 3.70 [09:00:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [09:00:44] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [09:02:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.68, 3.80, 3.65 [09:02:05] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.627 second response time [09:04:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:04:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 3.84, 3.94 [09:04:44] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:05:52] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.01, 1.39, 1.62 [09:07:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.04, 3.26, 3.81 [09:08:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.16, 2.81, 3.27 [09:10:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.48, 3.69 [09:10:48] !log [@mw11] starting deploy of {'config': True} to ovlon [09:10:51] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 3s [09:11:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:11:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:12:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.12, 3.33, 3.61 [09:15:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.49, 3.62 [09:16:01] when i try to open with logged in account, it's showing > [b280cc35f1992e69913d5c6a] 2022-01-15 09:10:04: "Wikimedia\Rdbms\DBQueryError" ধরনের মারাত্মক ব্যতিক্রম [09:16:29] but it was good with logged out or incognito mode [09:17:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.54, 3.63 [09:17:50] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb [09:17:53] Reception123: ^ [09:18:04] !log [@test101] starting deploy of {'config': True} to skip [09:18:05] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [09:18:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:18:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:19:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [09:19:51] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:20:08] https://ibb.co/4t47n6n [09:20:09] [url] Screenshot-6 — ImgBB | ibb.co [09:20:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.84, 2.75, 3.27 [09:21:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:25:18] Guest42: will look [09:27:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.93, 3.04, 3.34 [09:30:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 3.79, 3.43 [09:32:59] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [09:33:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [09:33:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.61, 3.48 [09:34:16] [02dns] 07JohnFLewis closed pull request 03#233: [22:58] mediawiki: failover deployment & mwtask - 13https://github.com/miraheze/dns/pull/233 [09:34:17] [02miraheze/dns] 07JohnFLewis pushed 034 commits to 03master [+8/-1/±5] 13https://github.com/miraheze/dns/compare/253327523ae8...d97139a82e52 [09:34:19] [02miraheze/dns] 07RhinosF1 0308ed140 - mediawiki: failover deployment & mwtask [09:34:20] [02miraheze/dns] 07RhinosF1 03399402e - Update miraheze.org [09:34:22] [02miraheze/dns] 07RhinosF1 03565282d - Merge branch 'master' into patch-25 [09:34:23] [02miraheze/dns] 07JohnFLewis 03d97139a - Merge pull request #233 from RhinosF1/patch-25 [09:34:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.85, 3.86, 3.55 [09:34:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:35:06] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:35:07] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/d97139a82e52...4a88dd4f0a23 [09:35:09] [02miraheze/dns] 07JohnFLewis 034a88dd4 - Move graylog over [09:37:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.80, 3.18, 3.33 [09:40:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.71, 2.80, 3.22 [09:41:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.74, 3.36, 3.41 [09:43:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.59, 3.15, 3.33 [09:47:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.21, 3.68, 3.54 [09:49:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 4.04, 3.70 [09:55:04] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.58, 1.75, 1.53 [09:55:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.39, 4.38, 3.71 [09:55:07] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [09:56:11] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:56:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.82, 3.84, 3.33 [09:56:56] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:56:58] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:57:06] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:57:29] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:59:03] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.29, 1.84, 1.60 [09:59:07] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:59:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.87, 3.84 [10:00:30] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.429 second response time [10:01:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.71, 4.00, 3.83 [10:01:06] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 4.505 second response time [10:01:07] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.464 second response time [10:01:13] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.515 second response time [10:01:34] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.531 second response time [10:02:23] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.34, 3.11, 3.20 [10:02:59] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.77, 4.27, 3.92 [10:03:02] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.18, 1.59, 1.56 [10:03:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 3.71, 3.75 [10:04:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.28, 3.87, 3.82 [10:06:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.31, 3.25 [10:08:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.37, 3.34, 3.27 [10:08:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.77, 4.30, 3.99 [10:09:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.78, 3.79 [10:12:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.59, 3.50, 3.76 [10:14:17] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 5.17, 4.57, 3.84 [10:14:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.26, 3.87, 3.84 [10:15:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 4.00, 3.88 [10:16:17] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 2.89, 3.98, 3.72 [10:16:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.98, 3.55, 3.73 [10:17:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.72, 3.80 [10:20:04] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.87, 1.89, 1.64 [10:20:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.23, 4.26, 3.58 [10:20:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.81, 3.77 [10:21:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 4.39, 4.06 [10:22:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.83, 3.50 [10:22:22] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [10:22:26] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [10:22:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.81, 3.51, 3.67 [10:23:41] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:24:05] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 0.76, 1.43, 1.52 [10:24:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.31, 3.23, 3.31 [10:25:04] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:25:08] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:26:06] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:26:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.03, 3.86, 3.73 [10:27:15] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.529 second response time [10:27:21] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.005 second response time [10:27:55] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.393 second response time [10:28:12] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.323 second response time [10:28:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.62, 3.44 [10:28:38] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.731 second response time [10:30:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:32:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.63, 3.43, 3.43 [10:34:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:34:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [10:36:28] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:38:07] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.23, 1.68, 1.52 [10:38:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.86, 3.09, 3.28 [10:40:10] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.66, 1.74, 1.56 [10:44:10] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.24, 1.85, 1.64 [10:44:56] PROBLEM - phab2 phd on phab2 is CRITICAL: PROCS CRITICAL: 0 processes with args 'phd' [10:46:09] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.29, 1.71, 1.63 [10:48:07] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.65, 4.01, 3.61 [10:48:09] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.35, 1.63, 1.61 [10:48:35] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.94, 3.98 [10:49:13] RECOVERY - phab2 phd on phab2 is OK: PROCS OK: 1 process with args 'phd' [10:49:27] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:49:35] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:49:45] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:49:45] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:50:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.19, 3.66, 3.54 [10:51:06] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:51:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [10:53:04] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.464 second response time [10:53:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:53:29] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.893 second response time [10:53:40] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.574 second response time [10:53:49] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.446 second response time [10:53:50] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.410 second response time [10:54:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.41, 2.68, 3.18 [10:57:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.57, 3.94 [10:58:29] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.04, 1.93, 1.70 [10:59:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 3.83, 3.98 [10:59:46] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:59:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [11:00:00] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:04] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:00:08] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:00:30] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:00:33] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.57, 1.77, 1.67 [11:01:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [11:02:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.23, 5.99, 4.37 [11:02:32] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.05, 1.51, 1.59 [11:06:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.00, 4.40, 3.97 [11:06:38] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.110 second response time [11:06:54] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.646 second response time [11:08:00] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.384 second response time [11:08:22] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.427 second response time [11:08:32] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.582 second response time [11:08:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.63, 3.75, 3.80 [11:09:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:09:41] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:16:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.53, 3.24, 3.83 [11:18:52] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.55, 1.74, 1.60 [11:20:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [11:20:52] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.04, 1.50, 1.53 [11:21:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [11:22:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.24, 2.66, 3.38 [11:22:31] PROBLEM - phab2 phd on phab2 is CRITICAL: PROCS CRITICAL: 0 processes with args 'phd' [11:23:42] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:23:54] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:24:00] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:24:07] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:24:39] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.37, 2.98, 3.30 [11:24:39] RECOVERY - phab2 phd on phab2 is OK: PROCS OK: 1 process with args 'phd' [11:24:55] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:26:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:27:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:28:13] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.017 second response time [11:28:16] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.391 second response time [11:28:21] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.001 second response time [11:29:01] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 2.023 second response time [11:30:01] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.914 second response time [11:31:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.84, 2.96, 3.86 [11:34:25] !log create missing 6 tables on new wikis [11:34:43] Reception123: ^^ [11:35:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [11:35:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:35:59] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [11:37:33] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.29, 3.17 [11:37:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.72, 3.86 [11:37:58] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.84, 1.73, 1.54 [11:39:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.73, 3.39, 3.22 [11:41:07] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:41:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.65, 3.82 [11:41:59] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.49, 1.99, 1.68 [11:43:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.30, 3.36, 3.24 [11:43:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.94, 3.90 [11:44:03] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.84, 1.90, 1.69 [11:45:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.78, 3.79, 3.85 [11:48:02] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 0.81, 1.45, 1.56 [11:49:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [11:49:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.81, 4.45, 4.06 [11:49:51] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:49:54] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:56] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:50:48] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:50:50] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:50:53] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:51:11] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:13] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:51:25] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 3.94, 3.54 [11:51:50] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.280 second response time [11:51:52] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.613 second response time [11:51:56] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.281 second response time [11:52:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.26, 3.00 [11:52:33] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:52:35] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:52:37] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:52:54] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:00] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:53:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.74, 3.01, 2.94 [11:54:08] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:08] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:54:29] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.620 second response time [11:54:32] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.744 second response time [11:54:32] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.601 second response time [11:55:00] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 9.438 second response time [11:55:05] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.421 second response time [11:55:07] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.474 second response time [11:55:08] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.313 second response time [11:55:22] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.724 second response time [11:55:24] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.390 second response time [11:55:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.90, 3.95, 3.93 [11:56:15] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 4.415 second response time [11:56:16] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.150 second response time [11:57:11] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.578 second response time [11:59:08] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:59:09] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:59:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 4.02, 3.95 [12:00:10] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.39, 2.11, 1.75 [12:01:07] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.110 second response time [12:01:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.98, 3.77, 3.72 [12:03:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.43, 4.35, 3.94 [12:05:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.09, 3.55, 3.23 [12:05:07] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [12:05:24] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.06, 3.80, 3.78 [12:06:11] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.14, 1.96, 1.85 [12:06:15] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:06:54] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:06:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.17, 3.07, 3.10 [12:08:02] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:09:22] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.24, 2.88, 3.40 [12:10:23] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.635 second response time [12:10:38] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:11:11] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 8.082 second response time [12:12:08] PROBLEM - phab2 phd on phab2 is CRITICAL: PROCS CRITICAL: 0 processes with args 'phd' [12:12:11] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.42, 1.55, 1.70 [12:12:14] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.075 second response time [12:12:42] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.353 second response time [12:14:20] RECOVERY - phab2 phd on phab2 is OK: PROCS OK: 1 process with args 'phd' [12:14:42] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:16:40] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:16:49] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:17:04] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:17:16] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:17:18] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:17:21] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:17:37] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:56] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.926 second response time [12:19:20] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.994 second response time [12:19:25] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.311 second response time [12:25:25] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.958 second response time [12:25:31] PROBLEM - phab2 phd on phab2 is CRITICAL: PROCS CRITICAL: 0 processes with args 'phd' [12:25:35] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.207 second response time [12:25:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.16, 4.00 [12:26:03] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 4.011 second response time [12:27:45] RECOVERY - phab2 phd on phab2 is OK: PROCS OK: 1 process with args 'phd' [12:29:42] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.327 second response time [12:29:52] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.983 second response time [12:29:55] people getting closed wiki emails right now is known problem? [12:30:08] suppose tied w/ RC reload [12:31:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:31:22] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.20, 1.88, 1.69 [12:31:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:32:54] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:32:55] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 6 backends are down. mw8 mw9 mw10 mw11 mw12 mw13 [12:33:01] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 6 backends are down. mw8 mw9 mw10 mw11 mw12 mw13 [12:33:17] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:21] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:23] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:29] PROBLEM - cp30 Stunnel Http for mon2 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:30] PROBLEM - cp31 Stunnel Http for mon2 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:31] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:31] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:32] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:33] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 6 backends are down. mw8 mw9 mw10 mw11 mw12 mw13 [12:33:33] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 6 backends are down. mw8 mw9 mw10 mw11 mw12 mw13 [12:33:43] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:43] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:44] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:47] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:48] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:50] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.30, 1.62, 1.62 [12:33:51] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:51] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:55] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:33:58] PROBLEM - cp21 Stunnel Http for mon2 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:34:10] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:34:19] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:34:19] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:34:36] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:34:37] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:34:43] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:34:45] PROBLEM - cp20 Stunnel Http for mon2 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:35:05] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:35:17] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:35:38] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:36:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [12:36:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [12:36:50] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.29, 3.80, 3.82 [12:38:23] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/2aae6833750b [12:38:40] [02miraheze/puppet] 07paladox 032aae683 - Make blog.miraheze.org use phab121 [12:38:41] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [12:38:43] [02puppet] 07paladox opened pull request 03#2333: Make blog.miraheze.org use phab121 - 13https://github.com/miraheze/puppet/pull/2333 [12:38:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:39:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.88, 3.41, 3.67 [12:39:30] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:39:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2aae6833750b...afaf9020e454 [12:39:33] [02miraheze/puppet] 07paladox 03afaf902 - Update default.vcl [12:39:34] [02puppet] 07paladox synchronize pull request 03#2333: Make blog.miraheze.org use phab121 - 13https://github.com/miraheze/puppet/pull/2333 [12:39:51] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:39:52] [02puppet] 07paladox closed pull request 03#2333: Make blog.miraheze.org use phab121 - 13https://github.com/miraheze/puppet/pull/2333 [12:39:53] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [12:39:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d8f0074b6b03...4a40bf374e09 [12:39:56] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:39:56] [02miraheze/puppet] 07paladox 034a40bf3 - Make blog.miraheze.org use phab121 (#2333) [12:39:58] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [12:40:11] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:42:50] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.69, 1.74, 1.65 [12:43:09] !log stopped manageInactiveWikis and reverting all wiki closures for today [12:43:13] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [12:44:11] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.592 second response time [12:44:18] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.352 second response time [12:44:28] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.526 second response time [12:44:42] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:50] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.971 second response time [12:44:51] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.02, 1.54, 1.59 [12:44:55] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:45:01] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:45:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.70, 3.68 [12:45:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:45:48] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.162 second response time [12:46:18] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:46:47] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:47:34] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.95, 3.67, 3.30 [12:49:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.78, 3.73 [12:49:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [12:49:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.55, 3.24, 3.19 [12:50:13] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:50:24] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:50:36] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:43] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:50:53] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.039 second response time [12:51:13] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:51:16] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:51:24] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.511 second response time [12:51:26] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.593 second response time [12:51:30] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:51:35] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.151 second response time [12:51:43] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.202 second response time [12:51:49] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:51:49] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:52:01] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.00, 4.65, 4.06 [12:56:39] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 6.566 second response time [12:58:22] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.682 second response time [12:58:27] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.714 second response time [12:58:45] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.734 second response time [12:59:10] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.609 second response time [12:59:22] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.764 second response time [13:00:28] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.701 second response time [13:00:36] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.030 second response time [13:00:53] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 9.549 second response time [13:00:54] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.436 second response time [13:01:44] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.43, 1.91, 1.63 [13:02:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.67, 3.68, 3.51 [13:04:14] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.85, 1.86, 1.65 [13:05:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.64, 3.37 [13:05:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.33, 3.82, 3.58 [13:06:13] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.23, 1.63, 1.59 [13:06:56] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:07:02] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:07:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.06, 3.40, 3.31 [13:07:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.47, 3.57, 3.51 [13:09:00] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.310 second response time [13:09:11] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.429 second response time [13:09:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.16, 4.12, 3.71 [13:11:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.85, 3.44, 3.37 [13:12:20] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:12:24] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:12:43] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:12:43] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:12:55] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.92, 3.20, 3.29 [13:15:00] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.538 second response time [13:15:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.70, 4.00, 3.93 [13:16:37] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.761 second response time [13:16:45] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.521 second response time [13:17:02] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.001 second response time [13:17:04] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.074 second response time [13:18:35] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:18:42] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:18:45] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:19:02] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:19:42] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:19:51] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:21:11] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.915 second response time [13:21:14] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 6.869 second response time [13:21:34] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.441 second response time [13:22:04] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.895 second response time [13:22:14] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 9.351 second response time [13:22:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 3.82, 3.82 [13:25:21] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:25:22] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:26:30] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:26:30] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:26:44] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:26:50] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:27:44] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:27:52] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.355 second response time [13:27:55] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.922 second response time [13:27:56] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:38] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:29:09] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 5.513 second response time [13:29:09] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.288 second response time [13:29:24] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.234 second response time [13:31:21] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:31:24] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.198 second response time [13:31:30] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.298 second response time [13:32:19] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.003 second response time [13:32:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.14, 3.29, 3.71 [13:33:05] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.753 second response time [13:33:15] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 3.932 second response time [13:33:39] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:34:06] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.447 second response time [13:35:36] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:38:08] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [13:38:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.24, 3.25, 3.86 [13:39:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:40:05] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:40:25] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:40:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.18, 3.75, 3.98 [13:41:15] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:41:18] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:41:31] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:42:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.66, 3.93 [13:44:18] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:44:54] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:45:02] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.69, 3.64, 3.60 [13:46:00] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 3.14, 2.21, 1.70 [13:46:09] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:46:11] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.242 second response time [13:47:08] [02puppet] 07paladox closed pull request 03#2323: varnish: remove ovlon backends - 13https://github.com/miraheze/puppet/pull/2323 [13:47:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4a40bf374e09...5d72c280c40c [13:47:11] [02miraheze/puppet] 07Universal-Omega 035d72c28 - varnish: remove ovlon backends (#2323) [13:47:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.54, 4.55, 4.23 [13:47:40] [02puppet] 07paladox closed pull request 03#2319: stunnel: remove ovlon - 13https://github.com/miraheze/puppet/pull/2319 [13:47:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/5d72c280c40c...4e1148d88fdc [13:47:43] [02miraheze/puppet] 07Universal-Omega 034e1148d - stunnel: remove ovlon (#2319) [13:48:38] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is UNKNOWN: NRPE: Command 'check_stunnel_mw13' not defined [13:48:46] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is UNKNOWN: NRPE: Command 'check_stunnel_mw8' not defined [13:48:51] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is UNKNOWN: NRPE: Command 'check_stunnel_mw12' not defined [13:48:52] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is UNKNOWN: NRPE: Command 'check_stunnel_mw11' not defined [13:48:55] PROBLEM - cp31 Stunnel Http for mon2 on cp31 is UNKNOWN: NRPE: Command 'check_stunnel_mon2' not defined [13:48:58] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is UNKNOWN: NRPE: Command 'check_stunnel_mw10' not defined [13:48:59] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is UNKNOWN: NRPE: Command 'check_stunnel_mw10' not defined [13:49:03] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is UNKNOWN: NRPE: Command 'check_stunnel_mw13' not defined [13:49:05] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:49:05] PROBLEM - cp30 Stunnel Http for mon2 on cp30 is UNKNOWN: NRPE: Command 'check_stunnel_mon2' not defined [13:49:10] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is UNKNOWN: NRPE: Command 'check_stunnel_mw11' not defined [13:49:13] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is UNKNOWN: NRPE: Command 'check_stunnel_mw8' not defined [13:49:14] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is UNKNOWN: NRPE: Command 'check_stunnel_mw12' not defined [13:49:18] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.048 second response time [13:49:21] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is UNKNOWN: NRPE: Command 'check_stunnel_mw9' not defined [13:49:29] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is UNKNOWN: NRPE: Command 'check_stunnel_mw10' not defined [13:49:30] PROBLEM - cp21 Stunnel Http for mon2 on cp21 is UNKNOWN: NRPE: Command 'check_stunnel_mon2' not defined [13:49:33] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is UNKNOWN: NRPE: Command 'check_stunnel_mw11' not defined [13:49:33] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is UNKNOWN: NRPE: Command 'check_stunnel_mw13' not defined [13:49:42] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:49:48] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 11 backends are healthy [13:50:02] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is UNKNOWN: NRPE: Command 'check_stunnel_mw9' not defined [13:50:31] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is UNKNOWN: NRPE: Command 'check_stunnel_mw8' not defined [13:50:31] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is UNKNOWN: NRPE: Command 'check_stunnel_mw9' not defined [13:50:35] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is UNKNOWN: NRPE: Command 'check_stunnel_mw9' not defined [13:50:40] PROBLEM - cp20 Stunnel Http for mon2 on cp20 is UNKNOWN: NRPE: Command 'check_stunnel_mon2' not defined [13:50:40] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is UNKNOWN: NRPE: Command 'check_stunnel_mw12' not defined [13:50:41] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is UNKNOWN: NRPE: Command 'check_stunnel_mw10' not defined [13:50:43] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.543 second response time [13:50:44] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.165 second response time [13:50:47] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is UNKNOWN: NRPE: Command 'check_stunnel_mw8' not defined [13:50:49] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is UNKNOWN: NRPE: Command 'check_stunnel_mw12' not defined [13:50:50] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 8.541 second response time [13:50:55] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is UNKNOWN: NRPE: Command 'check_stunnel_mw11' not defined [13:50:56] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.226 second response time [13:51:01] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is UNKNOWN: NRPE: Command 'check_stunnel_mw13' not defined [13:51:01] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.695 second response time [13:51:12] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.313 second response time [13:51:17] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 6.318 second response time [13:51:23] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 11 backends are healthy [13:51:24] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 11 backends are healthy [13:51:33] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.139 second response time [13:51:38] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.232 second response time [13:51:50] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.209 second response time [13:51:51] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 11 backends are healthy [13:52:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.40, 3.55, 3.70 [13:52:29] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.18, 1.92, 1.79 [13:52:57] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.03, 5.39, 4.67 [13:54:58] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.16, 5.57, 4.82 [13:55:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.73, 3.58, 3.92 [13:56:30] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.182 second response time [13:56:56] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.37, 5.14, 4.75 [13:58:37] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:43] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:58:50] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.34, 1.54, 1.66 [13:59:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.75, 4.78, 4.68 [13:59:14] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:59:51] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.73, 4.10, 4.00 [14:00:18] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:02:43] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:02:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.54, 3.03, 3.62 [14:03:01] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:20] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:03:39] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:03:41] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:03:50] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:04:03] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:04:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.40, 2.83, 3.35 [14:04:47] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:05:13] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:05:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 4.64, 6.08, 7.97 [14:05:32] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.018 second response time [14:05:33] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:05:37] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:05:42] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:49] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw101 mw102 mw122 [14:05:56] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw102 mw111 mw122 [14:06:10] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:11] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:12] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.012 second response time [14:06:16] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw111 mw121 mw122 [14:06:26] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 8.529 second response time [14:06:31] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw122 [14:06:36] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:07:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.41, 2.73, 3.34 [14:07:11] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.823 second response time [14:07:12] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.319 second response time [14:07:12] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.307 second response time [14:07:27] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.477 second response time [14:08:02] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.615 second response time [14:08:34] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:08:54] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.330 second response time [14:09:13] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 4.27, 4.95, 7.53 [14:09:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 3.47, 5.47, 7.76 [14:09:28] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.322 second response time [14:10:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 3.83, 5.24, 8.00 [14:10:21] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.006 second response time [14:10:22] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.282 second response time [14:10:29] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.018 second response time [14:10:37] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.028 second response time [14:10:40] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.317 second response time [14:11:11] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:11:26] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.294 second response time [14:12:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 3.94, 5.09, 7.58 [14:12:08] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.015 second response time [14:12:33] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:12:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 3.81, 4.76, 7.78 [14:13:02] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 4.21, 4.44, 6.43 [14:13:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 2.69, 3.80, 6.48 [14:13:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 3.07, 3.92, 6.59 [14:13:14] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.343 second response time [14:13:28] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.306 second response time [14:13:41] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [14:13:48] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:52] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:14:02] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 4.281 second response time [14:14:04] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [14:14:31] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.011 second response time [14:14:32] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.250 second response time [14:14:32] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:28] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:15:39] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:39] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:16:15] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 4.07, 4.00, 6.52 [14:16:18] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.273 second response time [14:16:30] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.270 second response time [14:17:09] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.016 second response time [14:17:37] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.262 second response time [14:17:44] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.364 second response time [14:17:44] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 9.458 second response time [14:17:46] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.315 second response time [14:17:49] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.017 second response time [14:17:55] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.016 second response time [14:17:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.52, 4.41, 6.45 [14:17:59] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.242 second response time [14:18:06] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:18:12] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:18:22] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.929 second response time [14:18:38] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.525 second response time [14:18:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.88, 4.40, 6.56 [14:19:35] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.766 second response time [14:19:51] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.021 second response time [14:19:56] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.328 second response time [14:19:58] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:20:00] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.610 second response time [14:20:09] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.419 second response time [14:20:16] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.547 second response time [14:20:40] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:20:51] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.329 second response time [14:21:10] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.021 second response time [14:21:31] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 11 backends are healthy [14:21:37] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 11 backends are healthy [14:22:03] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.079 second response time [14:22:07] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 11 backends are healthy [14:22:16] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 11 backends are healthy [14:22:38] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.966 second response time [14:22:51] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.86, 7.76, 6.85 [14:22:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.83, 8.76, 7.83 [14:23:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.54, 7.37, 6.66 [14:23:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.78, 6.84, 6.41 [14:23:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.67, 8.55, 7.55 [14:24:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.39, 6.81, 6.61 [14:24:51] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:26:15] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.77, 1.82, 1.52 [14:26:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:27:35] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.480 second response time [14:27:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.42, 3.67, 3.08 [14:28:31] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.66, 2.37, 1.77 [14:28:32] !log adjusted power profiles for cloud10, 11 and 12 to static maximum performance [14:29:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:30:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.01, 4.49, 3.29 [14:30:29] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:30:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.43, 2.67 [14:30:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [14:31:17] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:31:37] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:31:39] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:31:40] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:31:40] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:45] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:32:14] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:32:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.96, 3.25 [14:32:28] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:32:30] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.396 second response time [14:32:30] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 0.86, 1.78, 1.68 [14:32:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.52, 3.19, 2.69 [14:33:25] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.409 second response time [14:33:50] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.567 second response time [14:33:53] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.877 second response time [14:34:28] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.620 second response time [14:34:34] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.43, 1.62, 1.63 [14:34:38] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.448 second response time [14:35:50] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.354 second response time [14:35:52] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.881 second response time [14:35:58] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.890 second response time [14:36:38] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:19] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:37:25] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:37:35] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:38:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.71, 3.30, 3.19 [14:38:48] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 8.572 second response time [14:39:10] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:39:21] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.476 second response time [14:39:27] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.920 second response time [14:39:42] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.313 second response time [14:39:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.88, 3.94, 3.69 [14:41:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.08, 4.33, 3.86 [14:42:34] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:42:36] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:42:43] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:42:52] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:43:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:43:29] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:44:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.62, 3.83, 3.48 [14:44:38] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 8.071 second response time [14:44:39] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.704 second response time [14:44:47] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:44:49] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.812 second response time [14:44:56] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.779 second response time [14:44:56] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:45:13] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:45:18] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:45:29] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.018 second response time [14:45:31] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:46:24] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:46:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.80, 3.68, 3.08 [14:46:37] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:46:37] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.78, 1.72, 1.57 [14:46:47] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:54] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.316 second response time [14:46:57] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:47:02] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.164 second response time [14:47:15] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.145 second response time [14:47:23] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.057 second response time [14:48:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.66, 4.25, 3.71 [14:48:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.45, 3.06 [14:48:58] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:49:39] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.371 second response time [14:50:17] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.91, 3.21, 3.02 [14:50:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.04, 3.88, 3.64 [14:50:38] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.25, 1.61, 1.57 [14:51:02] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:02] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:51:16] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:51:24] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:15] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.92, 4.99, 4.31 [14:53:38] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.611 second response time [14:53:39] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 9.733 second response time [14:53:53] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.770 second response time [14:54:05] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.656 second response time [14:54:38] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.67, 4.84, 4.35 [14:54:40] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:52] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:54:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.36, 3.12, 3.40 [14:55:28] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.967 second response time [14:55:34] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.563 second response time [14:55:47] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.015 second response time [14:55:49] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.629 second response time [14:56:01] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.515 second response time [14:56:34] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:56:35] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:56:44] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:58:30] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:58:33] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.96, 3.71 [15:01:01] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.941 second response time [15:01:04] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 6.969 second response time [15:01:36] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.659 second response time [15:01:44] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.514 second response time [15:01:52] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.158 second response time [15:01:57] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.65, 3.28 [15:01:59] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 6.051 second response time [15:02:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.65, 4.03, 3.79 [15:02:03] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.519 second response time [15:02:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:03:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.10, 3.46, 3.26 [15:06:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.34, 3.41, 3.61 [15:06:19] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:06:28] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:06:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:06:42] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:44] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:06:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.64, 3.46, 3.93 [15:08:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.91, 3.35, 3.27 [15:08:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 3.87, 3.76 [15:08:40] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 2.39, 2.01, 1.73 [15:08:42] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.536 second response time [15:08:52] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.860 second response time [15:09:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.37, 3.97, 4.04 [15:09:06] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.941 second response time [15:09:09] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.462 second response time [15:10:50] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.78, 1.92, 1.74 [15:12:51] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.28, 1.65, 1.66 [15:13:36] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:13:38] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:13:39] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:13:42] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:05] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:14:14] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:14:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/04165f4cfabc [15:14:43] [02miraheze/puppet] 07paladox 0304165f4 - mediawiki: Reduce fastcgi_read_timeout to 140 [15:14:45] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [15:14:46] [02puppet] 07paladox opened pull request 03#2334: mediawiki: Reduce fastcgi_read_timeout to 140 - 13https://github.com/miraheze/puppet/pull/2334 [15:14:50] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:55] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:15:00] [02puppet] 07paladox closed pull request 03#2334: mediawiki: Reduce fastcgi_read_timeout to 140 - 13https://github.com/miraheze/puppet/pull/2334 [15:15:01] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [15:15:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4e1148d88fdc...34a8094731df [15:15:04] [02miraheze/puppet] 07paladox 0334a8094 - mediawiki: Reduce fastcgi_read_timeout to 140 (#2334) [15:15:05] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [15:16:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.54, 3.99, 3.94 [15:17:10] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 9.807 second response time [15:17:15] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.777 second response time [15:17:56] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.839 second response time [15:18:01] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.465 second response time [15:18:01] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:18:04] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.422 second response time [15:18:07] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.624 second response time [15:18:26] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.033 second response time [15:18:35] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.466 second response time [15:18:53] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:19:57] !log increase cores by one on db* [15:19:58] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:20:01] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.062 second response time [15:20:22] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:20:27] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 808 bytes in 0.262 second response time [15:20:33] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.67, 3.80 [15:20:39] !log restarting db* to pick up change (things are slow as it is) [15:20:52] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.325 second response time [15:21:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.78, 3.65, 3.98 [15:21:57] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.016 second response time [15:22:11] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 1.10, 0.28, 0.10 [15:22:18] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.324 second response time [15:22:25] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.280 second response time [15:22:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.17, 2.64, 3.40 [15:22:53] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 6.60, 3.35, 2.22 [15:22:55] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:23:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:23:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:23:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:23:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 3.23, 5.16, 7.77 [15:25:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.73, 7.21, 8.19 [15:26:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.14, 3.77, 3.78 [15:28:33] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 11.20, 5.67, 4.24 [15:29:07] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:29:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:29:48] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:29:49] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:29:53] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:29:59] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:09] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:11] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.88, 5.79, 4.14 [15:30:19] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:25] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:33] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:41] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:44] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:47] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:59] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:01] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:31:08] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:27] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:31:36] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:49] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:31:58] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:32:13] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.667 second response time [15:32:53] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/rebuildrecentchanges.php (END - exit=2) [15:32:57] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/rebuildrecentchanges.php (START) [15:33:22] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:33:23] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/rebuildtextindex.php (END - exit=2) [15:33:30] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/rebuildtextindex.php (START) [15:34:04] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.175 second response time [15:34:05] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.044 second response time [15:34:20] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.425 second response time [15:34:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:34:34] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.197 second response time [15:34:41] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.624 second response time [15:34:49] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.478 second response time [15:34:49] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.486 second response time [15:35:14] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.446 second response time [15:35:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:37:23] [02puppet] 07RhinosF1 opened pull request 03#2335: jobs::cron: disable misbehaving scripts - 13https://github.com/miraheze/puppet/pull/2335 [15:37:45] paladox: merge ^ pls [15:38:07] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:11] [02puppet] 07paladox closed pull request 03#2335: jobs::cron: disable misbehaving scripts - 13https://github.com/miraheze/puppet/pull/2335 [15:38:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/34a8094731df...afb1769ad243 [15:38:14] [02miraheze/puppet] 07RhinosF1 03afb1769 - jobs::cron: disable misbehaving scripts (#2335) [15:38:20] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:38:41] Thanks paladox [15:38:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:38:58] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:08] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/rebuildrecentchanges.php (END - exit=2) [15:39:11] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:16] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:18] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/rebuildtextindex.php (END - exit=2) [15:39:20] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:31] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/rebuildtextindex.php (START) [15:39:44] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:45] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:47] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:48] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:53] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:40:04] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:40:22] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/rebuildrecentchanges.php (START) [15:41:16] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.475 second response time [15:41:22] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:41:25] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 8.881 second response time [15:41:30] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:45] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:41:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:42:29] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.583 second response time [15:42:44] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.441 second response time [15:43:08] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.892 second response time [15:43:29] !log raise core to 5 on mw101 [15:43:33] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.417 second response time [15:43:35] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 8.483 second response time [15:43:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:43:47] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.870 second response time [15:44:04] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.220 second response time [15:44:28] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.953 second response time [15:44:47] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.337 second response time [15:45:09] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 4.56, 1.38, 0.48 [15:45:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:45:37] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.702 second response time [15:46:15] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.032 second response time [15:46:19] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.224 second response time [15:46:19] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.263 second response time [15:46:20] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 3.302 second response time [15:46:24] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.845 second response time [15:46:27] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.531 second response time [15:46:28] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 3.528 second response time [15:46:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:46:53] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.894 second response time [15:47:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:47:38] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.978 second response time [15:47:47] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.349 second response time [15:47:55] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.093 second response time [15:48:03] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.551 second response time [15:48:14] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.410 second response time [15:48:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:48:40] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.410 second response time [15:52:43] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:52:43] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:53:18] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:53:25] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:53:26] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:53:26] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:53:45] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.021 second response time [15:53:58] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:54:37] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:54:46] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.013 second response time [15:54:48] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.512 second response time [15:55:03] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:55:12] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:55:21] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.344 second response time [15:55:21] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:55:28] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 2.518 second response time [15:55:44] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:55:48] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:55:53] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.778 second response time [15:56:05] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.667 second response time [15:56:09] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:56:21] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.05, 8.28, 4.66 [15:56:29] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:56:40] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:04] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.671 second response time [15:57:06] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:57:16] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:18] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.630 second response time [15:57:38] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.552 second response time [15:57:39] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:57:39] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 9.992 second response time [15:57:50] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:57:50] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:49] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.210 second response time [15:59:17] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/4a88dd4f0a23...8e40a6ca0b53 [15:59:19] [02miraheze/dns] 07paladox 038e40a6c - Make grafana-new use mon2 [15:59:21] PROBLEM - mail2 Puppet on mail2 is CRITICAL: connect to address 2001:41d0:800:1bbd::7 port 5666: No route to hostconnect to host 2001:41d0:800:1bbd::7 port 5666: No route to host [15:59:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High 000000004 mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:00:16] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.801 second response time [16:01:15] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.325 second response time [16:01:40] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.314 second response time [16:01:41] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:01:44] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.903 second response time [16:02:09] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.288 second response time [16:02:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.71, 3.24, 3.88 [16:02:23] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 9.359 second response time [16:02:31] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.501 second response time [16:02:35] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.354 second response time [16:02:38] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:02:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.15, 8.39, 6.04 [16:02:43] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:02:49] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.891 second response time [16:02:54] PROBLEM - graylog121 APT on graylog121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:03:03] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:03:12] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.548 second response time [16:04:43] PROBLEM - linkwiki.cf - LetsEncrypt on sslhost is CRITICAL: connect to address linkwiki.cf and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:06:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.50, 3.59, 3.84 [16:08:01] PROBLEM - linkwiki.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for linkwiki.cf could not be found [16:08:04] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:08:13] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 3.631 second response time [16:08:13] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 7.234 second response time [16:08:27] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:08:33] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:08:39] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.313 second response time [16:08:46] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.224 second response time [16:08:57] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.347 second response time [16:09:23] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.814 second response time [16:09:33] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 4.31, 1.08, 0.36 [16:09:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 3.13, 0.82, 0.28 [16:10:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 2.04, 0.54, 0.18 [16:10:07] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.185 second response time [16:10:27] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.730 second response time [16:10:27] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.616 second response time [16:11:04] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.974 second response time [16:11:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.10, 2.19, 0.77 [16:13:03] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:13:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.59, 3.78 [16:13:18] !log switched all mw* cores to 5 [16:13:30] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.471 second response time [16:14:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:14:30] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.190 second response time [16:14:39] PROBLEM - mon111 Check correctness of the icinga configuration on mon111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:16:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.25, 4.37, 4.10 [16:17:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.21, 8.81, 4.54 [16:18:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.18, 8.08, 4.19 [16:18:36] RECOVERY - mon111 Check correctness of the icinga configuration on mon111 is OK: Icinga configuration is correct [16:20:45] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:20:51] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.48, 8.09, 5.03 [16:22:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.93, 9.02, 5.45 [16:24:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.59, 8.35, 5.67 [16:25:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.94, 9.61, 6.38 [16:26:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.91, 9.47, 6.49 [16:29:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.81, 9.96, 7.47 [16:29:54] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.99, 9.43, 8.56 [16:32:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.09, 8.67, 7.10 [16:34:40] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.28, 8.34, 7.01 [16:36:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.57, 10.50, 8.25 [16:36:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.82, 10.48, 8.20 [16:37:11] PROBLEM - mon111 Check correctness of the icinga configuration on mon111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:38:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.62, 9.51, 8.85 [16:39:18] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.58, 9.27, 8.12 [16:39:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.91, 9.23, 8.25 [16:41:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.65, 8.91, 8.72 [16:42:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.60, 9.53, 8.43 [16:43:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.06, 8.62, 7.71 [16:46:03] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.80, 9.77, 9.07 [16:47:30] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.77, 7.91, 7.61 [16:55:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.95, 10.44, 9.51 [16:59:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:00:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.47, 9.39, 9.48 [17:01:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.91, 10.59, 9.68 [17:04:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.70, 9.74, 9.53 [17:07:06] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.46, 9.30, 8.73 [17:07:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.19, 9.78, 9.62 [17:08:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.49, 10.31, 9.47 [17:10:40] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.44, 9.02, 9.38 [17:13:12] RECOVERY - mon111 Check correctness of the icinga configuration on mon111 is OK: Icinga configuration is correct [17:13:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.58, 10.03, 9.17 [17:15:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.20, 9.47, 9.84 [17:17:23] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.90, 10.81, 10.02 [17:17:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.86, 9.66, 9.48 [17:18:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.04, 9.76, 9.76 [17:18:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.30, 9.84, 9.91 [17:20:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.88, 9.05, 9.03 [17:23:45] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:24:57] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.96, 9.55, 9.87 [17:26:11] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.71, 9.39, 9.48 [17:30:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.32, 9.69, 9.78 [17:31:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.08, 8.35, 8.96 [17:34:02] PROBLEM - mon111 Check correctness of the icinga configuration on mon111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:35:43] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.377 second response time [17:35:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:36:18] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.27, 10.24, 9.93 [17:41:02] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.35, 10.17, 9.59 [17:41:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.56, 9.36, 9.62 [17:42:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.49, 9.89, 9.24 [17:44:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.37, 9.74, 9.53 [17:45:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.25, 10.09, 9.88 [17:46:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.18, 9.86, 9.38 [17:53:55] RECOVERY - mon111 Check correctness of the icinga configuration on mon111 is OK: Icinga configuration is correct [17:54:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.12, 9.69, 9.48 [17:56:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.78, 3.97 [17:56:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.56, 10.26, 9.73 [17:59:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.03, 9.30, 9.41 [18:01:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.40, 9.29, 9.49 [18:04:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 9.78, 10.06, 9.78 [18:08:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.93, 9.49, 9.61 [18:09:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 4.15, 4.08 [18:14:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.87, 3.33, 3.73 [18:14:53] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:16:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.44, 9.66, 9.52 [18:16:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:19:23] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.64, 4.49, 4.15 [18:21:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.99, 9.48, 9.95 [18:22:11] !log exporting searchindex for metawiki from old db cluster [18:24:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:49] !log killed rebuildtext maintenance script on mwtask111 [18:25:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:49] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:51] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.71, 10.43, 9.64 [18:25:55] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:25:58] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:26:47] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.19, 7.11, 6.43 [18:26:59] PROBLEM - mon111 Check correctness of the icinga configuration on mon111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:27:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.62, 10.80, 10.33 [18:29:21] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 6.025 second response time [18:29:31] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.023 second response time [18:29:31] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.889 second response time [18:29:33] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.239 second response time [18:29:40] !log dumping searchindex on all wikis on old cluster [18:29:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:30:30] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.32, 6.55, 6.37 [18:31:57] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:34:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:34:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.41, 9.64, 9.61 [18:36:29] RECOVERY - mon111 Check correctness of the icinga configuration on mon111 is OK: Icinga configuration is correct [18:39:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.20, 9.38, 9.48 [18:40:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.67, 9.93, 9.29 [18:40:50] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.20, 1.99, 1.69 [18:42:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.16, 9.11, 9.82 [18:43:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.41, 9.05, 9.32 [18:43:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.39, 8.98, 9.10 [18:44:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [18:46:04] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [18:46:24] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.96, 9.30, 9.32 [18:47:01] !log importing searchindex on db111 and db121 [18:47:30] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:47:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:48:20] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.17, 1.78, 1.48 [18:49:34] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:50:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a1c4a4a20ad4...4bbe91b5343b [18:50:31] [02miraheze/mw-config] 07Universal-Omega 034bbe91b - Update Defines.php [18:50:32] [02mw-config] 07Universal-Omega synchronize pull request 03#4383: remove ovlon - 13https://github.com/miraheze/mw-config/pull/4383 [18:50:43] !log importing searchindex on db101 [18:50:56] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.408 second response time [18:51:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:51:28] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.55, 1.13, 1.27 [18:51:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4bbe91b5343b...7b803fdbf058 [18:51:41] [02miraheze/mw-config] 07Universal-Omega 037b803fd - Update Defines.php [18:51:43] [02mw-config] 07Universal-Omega synchronize pull request 03#4383: remove ovlon - 13https://github.com/miraheze/mw-config/pull/4383 [18:53:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.70, 10.05, 9.42 [18:53:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.68, 9.27, 9.95 [18:54:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:54:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.38, 9.43, 9.52 [18:55:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.50, 9.80, 9.42 [19:00:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.95, 8.49, 9.61 [19:00:57] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 11.18, 9.17, 7.28 [19:01:08] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:01:57] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:02:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.36, 7.33, 8.39 [19:02:20] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:02:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.92, 6.79, 8.21 [19:02:33] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:02:33] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:02:39] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [19:02:41] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.248 second response time [19:02:41] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.344 second response time [19:02:44] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:02:53] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:02:53] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:02:54] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:02:55] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.07, 6.63, 8.10 [19:03:07] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:07] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:03:07] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:03:12] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:03:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 3.87, 6.34, 8.30 [19:03:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.47, 7.17, 9.50 [19:04:01] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:02] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw112 [19:04:26] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:04:27] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:33] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:04:58] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.85, 1.74, 1.99 [19:05:07] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:05:16] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw111 mw121 [19:05:17] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:05:17] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:05:29] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 4 backends are down. mw101 mw111 mw121 mw122 [19:05:47] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:05:59] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.089 second response time [19:06:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.65, 3.56, 3.98 [19:06:20] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [19:06:26] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.028 second response time [19:06:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.66, 6.67, 8.39 [19:06:30] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.351 second response time [19:06:35] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.835 second response time [19:06:38] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.539 second response time [19:06:49] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.235 second response time [19:06:50] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.131 second response time [19:06:51] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 9.500 second response time [19:06:51] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 4.106 second response time [19:06:56] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 6.563 second response time [19:07:09] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 3.078 second response time [19:07:12] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.459 second response time [19:07:14] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.020 second response time [19:07:15] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 11 backends are healthy [19:07:21] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.361 second response time [19:07:22] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.427 second response time [19:07:24] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.469 second response time [19:07:49] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.740 second response time [19:08:06] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 11 backends are healthy [19:09:11] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.023 second response time [19:09:11] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.07, 0.80, 1.53 [19:09:13] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.310 second response time [19:09:15] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.322 second response time [19:09:29] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [19:09:30] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.319 second response time [19:09:40] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 11 backends are healthy [19:10:40] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 5.674 second response time [19:12:06] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:12:06] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:12:11] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.75, 6.93, 8.33 [19:12:14] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.781 second response time [19:14:08] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:14:24] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:14:26] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:16:25] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 4.16, 3.99 [19:18:39] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 10.42, 8.39, 6.64 [19:18:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.57, 8.51, 8.56 [19:18:52] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.424 second response time [19:19:00] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.025 second response time [19:19:07] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.755 second response time [19:19:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.02, 3.86, 3.92 [19:19:35] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.397 second response time [19:19:38] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.837 second response time [19:19:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:21:47] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.93, 7.95, 6.81 [19:21:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.80, 9.71, 9.03 [19:22:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.48, 4.19, 4.05 [19:22:56] PROBLEM - mon111 Check correctness of the icinga configuration on mon111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:23:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.92, 9.05, 8.79 [19:24:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.14, 8.68, 8.77 [19:25:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.14, 8.91, 8.22 [19:25:45] RECOVERY - mon111 Check correctness of the icinga configuration on mon111 is OK: Icinga configuration is correct [19:26:15] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:26:51] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.22, 6.35, 6.54 [19:26:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.17, 7.75, 8.40 [19:27:04] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.83, 3.66, 3.87 [19:27:14] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:27:16] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.78, 7.98, 7.95 [19:27:41] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:27:43] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:27:45] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:27:52] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:27:57] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:11] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.83, 7.65, 8.27 [19:29:08] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:30:32] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:30:58] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:31:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.78, 4.08, 3.97 [19:31:09] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:31:26] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:31:39] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:32:00] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.222 second response time [19:32:04] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.372 second response time [19:32:05] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.861 second response time [19:32:11] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.487 second response time [19:32:21] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 9.088 second response time [19:32:30] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:32:36] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:46] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:33:31] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.442 second response time [19:33:38] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.874 second response time [19:33:38] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.672 second response time [19:33:45] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.269 second response time [19:33:51] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.390 second response time [19:34:15] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:34:30] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.023 second response time [19:34:33] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.416 second response time [19:34:45] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.268 second response time [19:34:55] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.080 second response time [19:35:17] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 5.875 second response time [19:36:10] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:36:18] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:36:57] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:36:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.66, 3.83 [19:37:04] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:37:07] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:37:57] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:37:59] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:38:08] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 3.320 second response time [19:38:17] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.310 second response time [19:38:17] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:38:21] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.023 second response time [19:38:25] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:38:27] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:38:33] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.045 second response time [19:38:39] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:38:40] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:38:58] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:38:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.88, 3.98, 3.92 [19:39:00] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:39:04] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.755 second response time [19:39:10] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.800 second response time [19:39:13] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:39:16] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.401 second response time [19:39:50] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:39:51] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:39:54] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:40:00] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.390 second response time [19:40:08] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:40:19] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:40:21] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:40:25] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.793 second response time [19:40:26] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.796 second response time [19:40:33] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.584 second response time [19:40:33] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 2.659 second response time [19:40:37] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.834 second response time [19:40:48] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.971 second response time [19:41:02] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.982 second response time [19:41:11] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.809 second response time [19:41:21] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.225 second response time [19:41:54] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.911 second response time [19:41:56] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.523 second response time [19:42:00] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.638 second response time [19:42:07] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 9.820 second response time [19:42:12] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.883 second response time [19:42:16] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.339 second response time [19:42:18] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.527 second response time [19:42:18] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.787 second response time [19:43:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.25, 6.93, 6.38 [19:44:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.67, 8.67, 8.03 [19:44:14] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.26, 6.44, 7.66 [19:44:43] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.29, 9.54, 8.27 [19:45:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.61, 8.78, 8.04 [19:45:38] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.42, 7.45, 6.64 [19:45:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:46:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.36, 8.81, 7.95 [19:47:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.58, 8.37, 8.02 [19:47:46] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:47:52] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.37, 6.95, 6.57 [19:48:03] !log killing rebuildrecentchanges on mwtask111 [19:48:54] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.32, 5.10, 6.76 [19:49:36] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:49:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:50] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 9.08, 8.65, 7.31 [19:49:55] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:50:19] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:50:21] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:51:12] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.43, 9.69, 8.67 [19:51:36] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.58, 9.67, 8.47 [19:51:48] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:58] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:02] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.882 second response time [19:52:03] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.377 second response time [19:52:21] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.248 second response time [19:52:26] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 8.658 second response time [19:53:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.94, 9.39, 8.68 [19:53:52] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.57, 7.17, 7.08 [19:54:03] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:11] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:54:53] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.33, 8.00, 8.22 [19:55:28] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.034 second response time [19:55:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.74, 9.85, 9.05 [19:55:49] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.78, 6.05, 6.68 [19:56:05] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:56:31] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:56:43] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:56:55] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:57:20] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:01] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.468 second response time [19:58:39] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.592 second response time [19:59:37] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.472 second response time [20:00:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.79, 8.17, 8.38 [20:00:40] PROBLEM - mon111 Check correctness of the icinga configuration on mon111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:01:58] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.31, 6.77, 6.86 [20:03:30] RECOVERY - mon111 Check correctness of the icinga configuration on mon111 is OK: Icinga configuration is correct [20:03:35] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.07, 6.76, 6.25 [20:03:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.36, 9.24, 8.54 [20:03:48] !log importing recentchanges onto db101 and db111 wikis [20:04:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.87, 9.95, 9.38 [20:04:22] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.85, 6.23, 6.66 [20:04:47] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.544 second response time [20:04:52] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:04:53] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:04:57] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:04:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.26, 9.69, 8.53 [20:05:28] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:05:32] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.35, 6.54, 6.23 [20:05:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.15, 9.64, 8.89 [20:05:39] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:05:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.50, 9.88, 8.98 [20:05:50] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:06:03] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:06:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.10, 9.37, 8.88 [20:06:28] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.537 second response time [20:06:32] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [20:06:46] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.927 second response time [20:06:47] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:06:50] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.752 second response time [20:06:51] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.329 second response time [20:06:55] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.311 second response time [20:07:04] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:07:05] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:07:10] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.232 second response time [20:07:19] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:07:23] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:07:31] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.754 second response time [20:07:34] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 5.902 second response time [20:07:38] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.921 second response time [20:07:38] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:08:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.01, 9.13, 8.85 [20:08:14] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:08:36] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:08:46] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.265 second response time [20:08:53] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:08:57] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 6.581 second response time [20:09:06] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.325 second response time [20:09:10] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:09:14] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:09:43] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.492 second response time [20:10:06] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:10:08] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:10:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.23, 9.48, 9.01 [20:10:09] !log importing recentchanges on db121 [20:10:15] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.682 second response time [20:11:07] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.442 second response time [20:11:10] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.397 second response time [20:11:21] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [20:11:32] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 8.212 second response time [20:11:40] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:11:40] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:11:43] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.333 second response time [20:11:44] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.499 second response time [20:11:48] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:12:14] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:12:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:12:17] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.315 second response time [20:12:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.68, 9.55, 9.12 [20:12:23] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.914 second response time [20:12:25] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.812 second response time [20:12:48] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.672 second response time [20:13:17] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.627 second response time [20:13:49] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.571 second response time [20:13:57] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:13:58] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 4.326 second response time [20:14:00] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.029 second response time [20:14:00] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.558 second response time [20:14:09] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.195 second response time [20:14:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.03, 9.78, 9.87 [20:14:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.38, 10.54, 9.53 [20:14:33] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.519 second response time [20:17:47] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:18:40] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:19:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.80, 9.29, 9.68 [20:19:39] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.38, 11.37, 10.46 [20:19:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [20:20:06] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:20:14] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:20:21] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:20:24] [02miraheze/mw-config] 07Universal-Omega pushed 033 commits to 03Universal-Omega-patch-3 [+0/-0/±4] 13https://github.com/miraheze/mw-config/compare/7b803fdbf058...3c774ffa5691 [20:20:26] [02miraheze/mw-config] 07Universal-Omega 033c774ff - Merge branch 'master' into Universal-Omega-patch-3 [20:20:27] [02mw-config] 07Universal-Omega synchronize pull request 03#4383: remove ovlon - 13https://github.com/miraheze/mw-config/pull/4383 [20:20:44] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:20:54] !log recentchanges and searchindex all fully imported, running initSiteStats [20:21:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.36, 9.34, 9.75 [20:21:39] miraheze/mw-config - Universal-Omega the build passed. [20:21:54] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.022 second response time [20:21:58] [02puppet] 07RhinosF1 opened pull request 03#2336: Revert "jobs::cron: disable misbehaving scripts" - 13https://github.com/miraheze/puppet/pull/2336 [20:22:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:17] JohnLewis: ^ [20:22:38] [02puppet] 07JohnFLewis closed pull request 03#2336: Revert "jobs::cron: disable misbehaving scripts" - 13https://github.com/miraheze/puppet/pull/2336 [20:22:40] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/afb1769ad243...0fc8ca0f37d4 [20:22:41] [02miraheze/puppet] 07RhinosF1 030fc8ca0 - Revert "jobs::cron: disable misbehaving scripts (#2335)" (#2336) [20:23:07] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.885 second response time [20:23:16] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.246 second response time [20:23:17] Ty [20:23:41] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.907 second response time [20:23:55] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 2.519 second response time [20:24:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.31, 9.95, 9.89 [20:26:01] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.196 second response time [20:27:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.69, 10.00, 9.94 [20:28:07] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:28:07] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:28:29] PROBLEM - mon111 Check correctness of the icinga configuration on mon111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:29:23] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:29:38] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:29:49] [02mw-config] 07Universal-Omega closed pull request 03#4383: remove ovlon - 13https://github.com/miraheze/mw-config/pull/4383 [20:29:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/mw-config/compare/573393068bfc...65f22323d690 [20:29:52] [02miraheze/mw-config] 07Universal-Omega 0365f2232 - remove ovlon (#4383) [20:29:54] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [20:29:55] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [20:30:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.82, 9.61, 9.74 [20:30:46] miraheze/mw-config - Universal-Omega the build passed. [20:31:35] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [20:31:41] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.897 second response time [20:32:02] RECOVERY - mon111 Check correctness of the icinga configuration on mon111 is OK: Icinga configuration is correct [20:32:06] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:32:15] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.35, 9.79, 9.44 [20:32:19] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:32:42] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.179 second response time [20:33:09] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.748 second response time [20:33:18] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:33:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/a3187ae87107 [20:33:50] [02miraheze/mw-config] 07Universal-Omega 03a3187ae - Remove mwtask1 [20:33:51] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [20:33:53] [02mw-config] 07Universal-Omega opened pull request 03#4385: Remove mwtask1 - 13https://github.com/miraheze/mw-config/pull/4385 [20:34:13] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a3187ae87107...b9c47b67e86c [20:34:15] [02miraheze/mw-config] 07github-actions 03b9c47b6 - CI: lint code to MediaWiki standards [20:34:16] [02mw-config] 07github-actions[bot] synchronize pull request 03#4385: Remove mwtask1 - 13https://github.com/miraheze/mw-config/pull/4385 [20:34:45] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:34:47] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:34:55] miraheze/mw-config - Universal-Omega the build passed. [20:36:53] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:38:26] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.859 second response time [20:39:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b9c47b67e86c...c7f4149cbf1d [20:39:59] [02miraheze/mw-config] 07Universal-Omega 03c7f4149 - Update LocalSettings.php [20:40:00] [02mw-config] 07Universal-Omega synchronize pull request 03#4385: Remove mwtask1 - 13https://github.com/miraheze/mw-config/pull/4385 [20:40:54] miraheze/mw-config - Universal-Omega the build passed. [20:41:02] [02mw-config] 07Universal-Omega closed pull request 03#4385: Remove mwtask1 - 13https://github.com/miraheze/mw-config/pull/4385 [20:41:03] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.759 second response time [20:41:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/65f22323d690...369792af30cf [20:41:05] [02miraheze/mw-config] 07Universal-Omega 03369792a - Remove mwtask1 (#4385) [20:41:06] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [20:41:08] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [20:41:09] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.439 second response time [20:42:11] miraheze/mw-config - Universal-Omega the build passed. [20:42:55] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:42:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:43:06] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:43:17] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [20:43:45] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.019 second response time [20:43:47] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 4 backends are down. mw101 mw102 mw121 mw122 [20:43:58] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:44:07] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:44:09] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:44:09] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:44:16] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 5 backends are down. mw102 mw111 mw112 mw121 mw122 [20:44:24] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.015 second response time [20:44:25] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:44:37] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:41] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.855 second response time [20:44:50] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.201 second response time [20:45:02] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:45:04] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.340 second response time [20:45:04] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw112 [20:45:13] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.083 second response time [20:45:54] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 11 backends are healthy [20:46:20] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw122 [20:47:04] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 2.718 second response time [20:47:08] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 11 backends are healthy [20:47:42] !log [@test101] starting deploy of {'config': True} to skip [20:47:43] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [20:48:13] PROBLEM - cp20 Stunnel Http for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:48:16] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.728 second response time [20:48:17] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 11 backends are healthy [20:48:25] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:48:25] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:48:43] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.220 second response time [20:49:04] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:49:24] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:49:32] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:24] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.017 second response time [20:50:29] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [20:50:30] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.338 second response time [20:50:33] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.018 second response time [20:50:55] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.733 second response time [20:51:02] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.667 second response time [20:51:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:51:31] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 795 bytes in 0.251 second response time [20:51:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/369792af30cf...23eb0bc31d4b [20:51:38] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.838 second response time [20:51:39] [02miraheze/mw-config] 07Universal-Omega 0323eb0bc - fix [20:52:01] CosmicAlpha: deployment failrd [20:52:18] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.307 second response time [20:52:29] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 11 backends are healthy [20:52:43] miraheze/mw-config - Universal-Omega the build passed. [20:52:58] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 29 minutes ago with 0 failures [20:53:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:53:27] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.57, 2.00, 1.14 [20:53:30] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to skip [20:53:31] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 0s [20:53:33] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:39] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 22230 bytes in 0.358 second response time [20:53:39] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 5.700 second response time [20:54:08] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:54:19] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:54:24] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:54:39] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:54:41] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:54:42] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:54:58] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:55:00] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:55:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:55:05] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.191 second response time [20:55:24] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.116 second response time [20:55:34] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.39, 1.41, 1.03 [20:55:36] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.276 second response time [20:55:48] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:55:57] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:56:20] !log [@mwtask111] starting deploy of {'config': True} to scsvg [20:56:34] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.525 second response time [20:56:43] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.277 second response time [20:57:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:57:03] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 3.180 second response time [20:57:09] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.110 second response time [20:57:45] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 84s [20:58:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:58:33] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 2.941 second response time [20:58:41] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.148 second response time [20:58:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:59:17] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.018 second response time [21:02:33] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:03:20] PROBLEM - mon111 Check correctness of the icinga configuration on mon111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:04:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.78, 3.57, 3.94 [21:07:00] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.708 second response time [21:07:17] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.982 second response time [21:07:53] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.322 second response time [21:08:56] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 5.669 second response time [21:09:02] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.077 second response time [21:09:07] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.234 second response time [21:09:08] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:09:14] RECOVERY - mon111 Check correctness of the icinga configuration on mon111 is OK: Icinga configuration is correct [21:09:18] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:09:31] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:09:36] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:09:57] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:10:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 3.85, 3.90 [21:11:46] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.625 second response time [21:11:47] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.839 second response time [21:12:20] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.222 second response time [21:12:22] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.638 second response time [21:12:41] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.021 second response time [21:13:15] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:13:17] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:13:22] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:14:07] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:14:15] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:15:32] PROBLEM - mon111 Check correctness of the icinga configuration on mon111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:15:34] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:46] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.314 second response time [21:18:15] RECOVERY - mon111 Check correctness of the icinga configuration on mon111 is OK: Icinga configuration is correct [21:18:17] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 8.227 second response time [21:18:43] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.540 second response time [21:18:47] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 5.168 second response time [21:20:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:20:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.88, 3.61, 3.79 [21:20:54] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:21:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.13, 3.42, 3.85 [21:22:39] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.182 second response time [21:24:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.37, 3.98, 4.01 [21:25:04] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.370 second response time [21:26:19] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [21:26:25] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:26:32] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:26:51] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:27:18] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:27:42] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:28:03] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.242 second response time [21:28:05] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:28:18] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:28:22] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:28:53] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:29:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/3fbbe1167a9f [21:29:29] [02miraheze/mw-config] 07Universal-Omega 033fbbe11 - Use CACHE_ACCEL for ParserCache [21:29:31] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [21:29:32] [02mw-config] 07Universal-Omega opened pull request 03#4386: Use CACHE_ACCEL for ParserCache - 13https://github.com/miraheze/mw-config/pull/4386 [21:29:57] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 9.897 second response time [21:30:01] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.725 second response time [21:30:10] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.170 second response time [21:30:13] RhinosF1: ^ [21:30:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.85, 3.67, 3.91 [21:30:26] miraheze/mw-config - Universal-Omega the build passed. [21:33:01] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:33:02] PROBLEM - cp20 Stunnel Http for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:33:12] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:33:14] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.83, 5.03, 4.34 [21:34:11] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.876 second response time [21:34:23] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 3.348 second response time [21:34:32] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.165 second response time [21:34:34] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.314 second response time [21:34:56] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:35:07] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.887 second response time [21:35:25] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:35:28] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.526 second response time [21:35:36] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.525 second response time [21:35:48] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.020 second response time [21:36:14] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.692 second response time [21:36:18] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:36:18] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.182 second response time [21:36:20] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:36:22] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:36:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.27, 4.70, 4.11 [21:37:00] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.049 second response time [21:37:39] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.442 second response time [21:38:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:38:42] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.746 second response time [21:38:46] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.683 second response time [21:38:47] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.343 second response time [21:39:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:41:59] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.77, 9.16, 10.00 [21:43:05] PROBLEM - mw101 JobRunner Service on mw101 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [21:43:10] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:43:12] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:43:14] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:43:31] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:43:39] CosmicAlpha: not sure on impact [21:43:49] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:44:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.78, 3.91 [21:44:38] RhinosF1: has minor impact, but I'm trying a couple things to improve it. [21:44:54] CosmicAlpha: it can't be worse [21:45:01] You could try locally on 50% first [21:45:31] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.111 second response time [21:45:32] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.375 second response time [21:45:33] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.438 second response time [21:45:43] RhinosF1: I am trying on mw101. It helps, but memcached helps far more so am going to try something in a minute. [21:45:57] CosmicAlpha: kk [21:46:04] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.497 second response time [21:47:54] PROBLEM - graylog121 Disk Space on graylog121 is WARNING: DISK WARNING - free space: / 911 MB (10% inode=86%); [21:48:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/998850f76d77 [21:48:59] [02miraheze/mw-config] 07Universal-Omega 03998850f - use memcached [21:49:01] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [21:49:12] [02mw-config] 07Universal-Omega opened pull request 03#4387: use memcached - 13https://github.com/miraheze/mw-config/pull/4387 [21:49:32] [02mw-config] 07Universal-Omega closed pull request 03#4387: use memcached - 13https://github.com/miraheze/mw-config/pull/4387 [21:49:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3fbbe1167a9f...d79fa12a5ce6 [21:49:35] [02miraheze/mw-config] 07Universal-Omega 03d79fa12 - use memcached (#4387) [21:49:36] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [21:49:38] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [21:49:39] [02mw-config] 07Universal-Omega synchronize pull request 03#4386: Use CACHE_ACCEL for ParserCache - 13https://github.com/miraheze/mw-config/pull/4386 [21:49:51] [02mw-config] 07Universal-Omega edited pull request 03#4386: Use memcached for ParserCache - 13https://github.com/miraheze/mw-config/pull/4386 [21:50:12] miraheze/mw-config - Universal-Omega the build passed. [21:50:36] [02mw-config] 07Universal-Omega edited pull request 03#4386: Use memcached for ParserCache - 13https://github.com/miraheze/mw-config/pull/4386 [21:50:37] miraheze/mw-config - Universal-Omega the build passed. [21:50:49] RhinosF1: ^ [21:51:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.43, 9.07, 10.00 [21:51:17] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.247 second response time [21:51:24] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:26] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:27] CosmicAlpha: why 5 days [21:53:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.43, 10.33, 10.33 [21:53:40] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.456 second response time [21:53:43] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.868 second response time [21:53:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.63, 4.16, 3.97 [21:53:51] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 3.081 second response time [21:53:57] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.189 second response time [21:54:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.64, 9.28, 9.97 [21:54:25] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:54:48] [02mw-config] 07Universal-Omega synchronize pull request 03#4386: Use memcached for ParserCache - 13https://github.com/miraheze/mw-config/pull/4386 [21:54:50] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d79fa12a5ce6...c7f047a1621e [21:54:51] [02miraheze/mw-config] 07Universal-Omega 03c7f047a - Update GlobalCache.php [21:55:04] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:55:05] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:09] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:55:15] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:55:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:55:32] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:55:46] miraheze/mw-config - Universal-Omega the build passed. [21:56:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.32, 9.57, 9.97 [21:56:28] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 3 minutes ago with 1 failures [21:56:38] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.382 second response time [21:57:22] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.552 second response time [21:57:24] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 6.133 second response time [21:57:31] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.966 second response time [21:57:42] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:58:12] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.501 second response time [21:59:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.31, 8.94, 9.66 [21:59:31] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:59:42] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:59:56] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:00:21] PROBLEM - cp20 Stunnel Http for mw101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:02:53] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.122 second response time [22:03:03] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.534 second response time [22:03:11] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 4.683 second response time [22:03:13] RECOVERY - mw101 JobRunner Service on mw101 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [22:03:38] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.030 second response time [22:04:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:04:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:04:30] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.562 second response time [22:05:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.15, 10.09, 9.62 [22:05:33] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.70, 9.36, 9.54 [22:07:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.06, 9.37, 9.53 [22:08:09] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:08:13] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:08:37] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:09:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [22:09:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb [22:09:56] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:09:59] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:10:04] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.82, 10.25, 9.85 [22:12:02] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.218 second response time [22:13:39] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:13:41] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:09] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:41] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:45] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:16:54] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:19:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.58, 3.97 [22:22:24] PROBLEM - www.pornwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.pornwiki.org' expires in 15 day(s) (Mon 31 Jan 2022 22:18:29 GMT +0000). [22:22:38] PROBLEM - graylog121 Disk Space on graylog121 is CRITICAL: DISK CRITICAL - free space: / 509 MB (5% inode=86%); [22:23:46] PROBLEM - files.pornwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'files.pornwiki.org' expires in 15 day(s) (Mon 31 Jan 2022 22:19:53 GMT +0000). [22:24:11] PROBLEM - en.pornwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.pornwiki.org' expires in 15 day(s) (Mon 31 Jan 2022 22:21:00 GMT +0000). [22:24:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.42, 3.60, 3.84 [22:26:12] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.983 second response time [22:26:41] PROBLEM - pornwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pornwiki.org' expires in 15 day(s) (Mon 31 Jan 2022 22:17:30 GMT +0000). [22:27:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/2bef3c0f0238 [22:27:57] [02miraheze/puppet] 07paladox 032bef3c0 - mariadb: tweek config settings [22:27:58] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [22:28:00] [02puppet] 07paladox opened pull request 03#2337: mariadb: tweek config settings - 13https://github.com/miraheze/puppet/pull/2337 [22:28:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2bef3c0f0238...398bdd3b305f [22:28:37] [02miraheze/puppet] 07paladox 03398bdd3 - Update config.pp [22:28:38] [02puppet] 07paladox synchronize pull request 03#2337: mariadb: tweek config settings - 13https://github.com/miraheze/puppet/pull/2337 [22:29:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/398bdd3b305f...181c2efe9ba2 [22:29:18] [02miraheze/puppet] 07paladox 03181c2ef - Update db101.yaml [22:29:20] [02puppet] 07paladox synchronize pull request 03#2337: mariadb: tweek config settings - 13https://github.com/miraheze/puppet/pull/2337 [22:29:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/181c2efe9ba2...31e623f70ddd [22:29:33] [02miraheze/puppet] 07paladox 0331e623f - Update db111.yaml [22:29:35] [02puppet] 07paladox synchronize pull request 03#2337: mariadb: tweek config settings - 13https://github.com/miraheze/puppet/pull/2337 [22:29:42] PROBLEM - wiki.meregos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.meregos.com' expires in 15 day(s) (Mon 31 Jan 2022 22:24:31 GMT +0000). [22:29:44] !sre I haven't been able to auto-sign in on any Miraheze wiki using a custom domain :( [22:29:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/31e623f70ddd...49d6e0e072b7 [22:29:45] PROBLEM - www.sdiy.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sdiy.info' expires in 15 day(s) (Mon 31 Jan 2022 22:23:27 GMT +0000). [22:29:45] [02miraheze/puppet] 07paladox 0349d6e0e - Update db121.yaml [22:29:47] [02puppet] 07paladox synchronize pull request 03#2337: mariadb: tweek config settings - 13https://github.com/miraheze/puppet/pull/2337 [22:29:50] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.288 second response time [22:30:00] (since the migration) [22:30:07] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 4.043 second response time [22:30:28] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.021 second response time [22:30:33] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.322 second response time [22:30:51] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.847 second response time [22:31:31] PROBLEM - sdiy.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sdiy.info' expires in 15 day(s) (Mon 31 Jan 2022 22:23:27 GMT +0000). [22:31:38] dmehus: not a reason to page [22:31:51] PROBLEM - wiki.worldsofweary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.worldsofweary.com' expires in 15 day(s) (Mon 31 Jan 2022 22:25:48 GMT +0000). [22:32:20] RhinosF1, well I view it is something that's broken in a systemic way affecting many users, including users with global permissions [22:32:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 3.72, 3.91 [22:33:00] dmehus: we have more important things right now. It's an issue sure but not the biggest issue right now. [22:33:04] dmehus: the only reason to page is a full site outage [22:33:11] It's an easily workaroundable issue [22:33:19] You've been told a million times [22:33:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:33:35] hey dhemus [22:33:45] Next time you ignore polices on paging, I'm going to start with mutes [22:33:46] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:33:48] RhinosF1, well, no, you've not ever said it was limited to the entire Miraheze infrastructure being down [22:34:10] CosmicAlpha, ack, thank you, I'll start a high priority task instead [22:34:16] dmehus: it's been very clear it's only allowed when there is grounds to wake the entire of SRE up and there's no ongoing response [22:34:30] Neither of them were met [22:34:44] PROBLEM - wiki.meeusen.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.meeusen.net' expires in 15 day(s) (Mon 31 Jan 2022 22:26:57 GMT +0000). [22:34:52] Well, aren't all SRE awake right now (so it's sort of moot)? [22:34:57] tbh, I take it more of a "hey guys, look at this", it's not really a page is it? [22:35:15] JohnLewis, thanks, that's how I intended it [22:35:24] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 8.077 second response time [22:35:28] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.098 second response time [22:35:38] JohnLewis: !_sre is supposed to be for alerts that require an immediate response [22:35:39] I don't even receive pings for it so it doesn't really bother me anyway, so isn't really an SRE page [22:35:40] PROBLEM - wiki.vinesh.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.vinesh.eu.org' expires in 15 day(s) (Mon 31 Jan 2022 22:28:33 GMT +0000). [22:35:55] yeah, isn't !sre a client-side stalkword? [22:35:59] It was supposed to start sending alerts to SRE's phones but PagerDuty was rubbish [22:36:03] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:36:06] dmehus: you've just pinged again [22:36:19] How many times can we tell you not to do it [22:36:26] PROBLEM - www.baharna.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.baharna.org' expires in 15 day(s) (Mon 31 Jan 2022 22:29:54 GMT +0000). [22:36:30] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.98, 6.45, 5.55 [22:36:30] PROBLEM - mw101 Puppet on mw101 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 35 minutes ago with 0 failures [22:36:33] If you're paging via an IRC stalk word, the problem is the pager, not the user [22:36:42] JohnLewis: it's mainly mw team because the only thing that uses it is the exceptions alert [22:36:45] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.826 second response time [22:36:54] [02puppet] 07paladox closed pull request 03#2337: mariadb: tweek config settings - 13https://github.com/miraheze/puppet/pull/2337 [22:36:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/puppet/compare/0fc8ca0f37d4...064dd8c3c8a5 [22:36:57] [02miraheze/puppet] 07paladox 03064dd8c - mariadb: tweek config settings (#2337) [22:36:58] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [22:37:00] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [22:37:00] JohnLewis: yes the pager is useless because good free paging software is pants [22:37:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.99, 8.99, 9.79 [22:37:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.10, 9.34, 9.91 [22:37:10] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.125 second response time [22:37:27] Free hasn't worked [22:37:32] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.331 second response time [22:37:46] RhinosF1, sorry, but that was clearly used as an example. How is it an ! and a word any different than a ping? arguably, it's more friendly than pinging, directly, multiple SRE, no? [22:37:53] I would advise alternatives method than relying on IRC to page, personally. Stalk words are useful when a response is needed, not when everyone needs waking up as if I wanted the listen to the stalk word, I'd listen to it than get a text message [22:38:12] dmehus: it still sent an alert [22:38:12] PROBLEM - www.portalsofphereon.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.portalsofphereon.com' expires in 15 day(s) (Mon 31 Jan 2022 22:31:57 GMT +0000). [22:38:16] PROBLEM - baharna.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.baharna.org' expires in 15 day(s) (Mon 31 Jan 2022 22:29:54 GMT +0000). [22:38:36] You've been told by a million users not to use stalk words outside of emergencies [22:38:52] I might try Zenduty to properly set it up [22:38:54] Do all SRE still use PagerDuty? If it's not so useful, surely it's using server resources we could deploy elsewhere, no? [22:39:10] dmehus: it's free software not hosted by us and no [22:39:15] The trial expired [22:39:21] I might try Zenduty [22:39:23] ok [22:39:32] It never worked and support weren't great [22:39:53] I mean, if it's a trial, it's not really free software; it's a severely limited free plan for a freemium cloud-based product [22:40:08] !log raise ram on db* to 15gb and restart [22:40:08] it wasn't that limited [22:40:30] They were supposed to setup an OSS plan that was pretty good [22:40:33] well I mean the "free" version (after the trial expired) [22:40:38] But they never co-operated [22:41:09] PROBLEM - db101 Current Load on db101 is CRITICAL: connect to address 2a10:6740::6:102 port 5666: Connection refusedconnect to host 2a10:6740::6:102 port 5666: Connection refused [22:41:13] PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.autocountsoft.com' expires in 15 day(s) (Mon 31 Jan 2022 22:36:14 GMT +0000). [22:41:30] PROBLEM - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.gesamtschule-nordkirchen.de' expires in 15 day(s) (Mon 31 Jan 2022 22:38:22 GMT +0000). [22:41:31] PROBLEM - portalsofphereon.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.portalsofphereon.com' expires in 15 day(s) (Mon 31 Jan 2022 22:31:57 GMT +0000). [22:41:56] The aim was like klaxon.wikimedia.org trusted humans would also be allowed to trigger alerts, possibly via IRC [22:42:00] As well [22:42:09] PROBLEM - db101 NTP time on db101 is CRITICAL: connect to address 2a10:6740::6:102 port 5666: Connection refusedconnect to host 2a10:6740::6:102 port 5666: Connection refused [22:42:09] PROBLEM - db101 Puppet on db101 is CRITICAL: connect to address 2a10:6740::6:102 port 5666: Connection refusedconnect to host 2a10:6740::6:102 port 5666: Connection refused [22:42:13] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 808 bytes in 0.083 second response time [22:42:15] But I've never massively cared because we can't trust anyone [22:42:22] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 808 bytes in 0.088 second response time [22:42:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 1.80, 7.47, 9.63 [22:42:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 2.09, 7.09, 9.12 [22:42:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 1.96, 7.37, 9.75 [22:42:30] PROBLEM - db101 MariaDB on db101 is CRITICAL: Can't connect to MySQL server on 'db101.miraheze.org' (115) [22:42:35] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 808 bytes in 0.120 second response time [22:42:35] PROBLEM - db101 PowerDNS Recursor on db101 is CRITICAL: connect to address 2a10:6740::6:102 port 5666: Connection refusedconnect to host 2a10:6740::6:102 port 5666: Connection refused [22:42:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.75, 2.54, 3.36 [22:42:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 1.63, 7.25, 9.74 [22:42:45] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:42:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.24, 2.79, 3.63 [22:42:46] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1024 bytes in 0.082 second response time [22:42:47] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 808 bytes in 0.075 second response time [22:42:50] paladox: you can't restart DBs with no notice [22:42:57] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is WARNING: WARNING - NGINX Error Rate is 51% [22:43:03] SRE, persistent 503s on wiki.mxlinuxusers.de wiki [22:43:12] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [22:43:13] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:43:15] I can't access the wiki at all despite reloading and reloading [22:43:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [22:43:18] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:43:20] PROBLEM - db101 Disk Space on db101 is CRITICAL: connect to address 2a10:6740::6:102 port 5666: No route to hostconnect to host 2a10:6740::6:102 port 5666: No route to host [22:43:22] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1026 bytes in 6.075 second response time [22:43:32] dmehus: what do you think icinga is saying [22:43:37] And me pinging paladox [22:43:41] PROBLEM - christipedia.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.christipedia.nl' expires in 15 day(s) (Mon 31 Jan 2022 22:39:36 GMT +0000). [22:43:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 0.98, 5.00, 7.94 [22:43:47] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 1.13, 4.77, 7.77 [22:43:48] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [22:43:48] 22:40:08 !log raise ram on db* to 15gb and restart [22:44:03] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [22:44:03] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is WARNING: WARNING - NGINX Error Rate is 41% [22:44:08] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.201 second response time [22:44:11] RECOVERY - db101 NTP time on db101 is OK: NTP OK: Offset 0.06534707546 secs [22:44:12] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [22:44:17] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.277 second response time [22:44:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.13, 2.50, 3.81 [22:44:26] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 2.42, 5.29, 8.20 [22:44:27] RECOVERY - db101 MariaDB on db101 is OK: Uptime: 64 Threads: 180 Questions: 48868 Slow queries: 3564 Opens: 1964 Open tables: 1958 Queries per second avg: 763.562 [22:44:36] RECOVERY - db101 PowerDNS Recursor on db101 is OK: DNS OK: 0.104 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [22:44:38] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 2.951 second response time [22:44:42] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:44:43] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.40, 2.22, 3.31 [22:44:43] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.746 second response time [22:44:44] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.462 second response time [22:44:53] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 14% [22:45:03] PROBLEM - johanloopmans.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.johanloopmans.nl' expires in 15 day(s) (Mon 31 Jan 2022 22:42:56 GMT +0000). [22:45:09] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.21, 1.32, 0.46 [22:45:09] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 11 backends are healthy [22:45:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:45:12] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 11 backends are healthy [22:45:14] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22201 bytes in 0.443 second response time [22:45:17] RECOVERY - db101 Disk Space on db101 is OK: DISK OK - free space: / 393305 MB (62% inode=98%); [22:45:19] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.282 second response time [22:45:47] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 11 backends are healthy [22:45:59] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 23% [22:46:03] PROBLEM - www.johanloopmans.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.johanloopmans.nl' expires in 15 day(s) (Mon 31 Jan 2022 22:42:56 GMT +0000). [22:46:07] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 11 backends are healthy [22:46:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 3.20, 3.91 [22:46:20] PROBLEM - db111 ferm_active on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [22:46:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.45, 6.13, 8.49 [22:46:27] PROBLEM - db111 conntrack_table_size on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [22:46:31] PROBLEM - db111 APT on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [22:46:45] PROBLEM - db111 MariaDB on db111 is CRITICAL: Can't connect to MySQL server on 'db111.miraheze.org' (115) [22:46:45] PROBLEM - db111 NTP time on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [22:47:19] PROBLEM - www.rothwell-leeds.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.rothwell-leeds.co.uk' expires in 15 day(s) (Mon 31 Jan 2022 22:44:11 GMT +0000). [22:47:20] PROBLEM - db111 Current Load on db111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:48:05] PROBLEM - www.christipedia.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.christipedia.nl' expires in 15 day(s) (Mon 31 Jan 2022 22:39:36 GMT +0000). [22:48:23] RECOVERY - db111 ferm_active on db111 is OK: OK ferm input default policy is set [22:48:32] RECOVERY - db111 APT on db111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:48:32] RECOVERY - db111 conntrack_table_size on db111 is OK: OK: nf_conntrack is 0 % full [22:48:36] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.69, 5.48, 8.09 [22:48:45] RECOVERY - db111 MariaDB on db111 is OK: Uptime: 73 Threads: 16 Questions: 42173 Slow queries: 1278 Opens: 1305 Open tables: 1299 Queries per second avg: 577.712 [22:48:51] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset 6.791949272e-05 secs [22:49:15] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 3.10, 1.11, 0.40 [22:49:38] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) [22:49:55] PROBLEM - db121 conntrack_table_size on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: Connection refusedconnect to host 2a10:6740::6:302 port 5666: Connection refused [22:49:57] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [22:50:00] PROBLEM - db121 Puppet on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: Connection refusedconnect to host 2a10:6740::6:302 port 5666: Connection refused [22:50:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.98, 6.23, 8.25 [22:50:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.71, 3.35, 3.85 [22:50:26] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4179 bytes in 0.042 second response time [22:50:53] PROBLEM - db121 NTP time on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:53] PROBLEM - db121 ferm_active on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:54] PROBLEM - db121 PowerDNS Recursor on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:04] PROBLEM - db121 Current Load on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: No route to hostconnect to host 2a10:6740::6:302 port 5666: No route to host [22:51:37] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 28 Threads: 57 Questions: 24508 Slow queries: 1115 Opens: 1451 Open tables: 1445 Queries per second avg: 875.285 [22:51:53] RECOVERY - db121 conntrack_table_size on db121 is OK: OK: nf_conntrack is 0 % full [22:51:58] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [22:51:59] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.426 second response time [22:52:22] PROBLEM - rothwell-leeds.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.rothwell-leeds.co.uk' expires in 15 day(s) (Mon 31 Jan 2022 22:44:11 GMT +0000). [22:52:25] PROBLEM - www.erikapedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.erikapedia.com' expires in 15 day(s) (Mon 31 Jan 2022 22:50:10 GMT +0000). [22:52:26] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 18991 bytes in 0.161 second response time [22:53:00] RECOVERY - db121 NTP time on db121 is OK: NTP OK: Offset -0.0002411305904 secs [22:53:02] RECOVERY - db121 PowerDNS Recursor on db121 is OK: DNS OK: 0.087 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [22:53:04] RECOVERY - db121 ferm_active on db121 is OK: OK ferm input default policy is set [22:53:11] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.04, 1.30, 0.49 [22:53:55] PROBLEM - thestarsareright.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thestarsareright.org' expires in 15 day(s) (Mon 31 Jan 2022 22:47:28 GMT +0000). [22:57:49] PROBLEM - mon111 Check correctness of the icinga configuration on mon111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:58:37] PROBLEM - wiki.hausos.co - reverse DNS on sslhost is UNKNOWN: [22:59:14] i suppose it carn't push atm because github being ipv4 only cc RhinosF1 JohnLewis ^ [22:59:44] paladox: the proxy should be set [22:59:57] it is, but i'm not sure if it works with ssh [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:03] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:12] paladox: ffs [23:00:20] That's another thing to change [23:00:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:36] It might have to use a http based method [23:00:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.55, 4.65, 4.15 [23:02:00] How do I solve an editing issue? [23:02:13] What issue [23:02:27] I assume you same person that emailed [23:02:48] Yes I am, didnt know which way would be quickest to contact you haha. [23:03:49] So friends and I regularly edit the Majestic Song Contest wiki. It is saying that edits hadn't been made recently [23:04:04]  (Didnt mean to send that yet haha) [23:04:07] Just reopen it and ignore if you have access too [23:04:14] It was a temporary fault [23:05:13] A friend messaged earlier requesting to be a bureaucrat but the person who is the bureaucrat for our pages can not remember her login details so she can not login to uncheck the box needed to allow edits [23:05:20] PROBLEM - mon111 Check correctness of the icinga configuration on mon111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:05:43] Guest34, looking [23:05:44] dmehus: can you mark open ^ [23:05:49] Guest34: which wiki [23:05:59] Majestic Song Contest [23:06:14] * dmehus going to try majesticsongcontest.miraheze.org [23:06:47] Guest34, done [23:07:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.68, 9.49, 8.87 [23:07:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.23, 11.92, 10.27 [23:07:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.97, 5.05, 3.95 [23:08:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.08, 9.23, 8.41 [23:08:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.71, 9.68, 8.48 [23:08:10] PROBLEM - erikapedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.erikapedia.com' expires in 15 day(s) (Mon 31 Jan 2022 22:50:10 GMT +0000). [23:08:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.49, 9.53, 8.69 [23:08:17] PROBLEM - www.mcpk.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.mcpk.wiki' expires in 15 day(s) (Mon 31 Jan 2022 22:52:23 GMT +0000). [23:08:19] I realize this is low-ish priority right now, but assuming Miraheze Commons images aren't loading globally? [23:08:24] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.61, 5.32, 4.75 [23:08:25] PROBLEM - www.lab612.at - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.lab612.at' expires in 15 day(s) (Mon 31 Jan 2022 22:53:37 GMT +0000). [23:08:25] PROBLEM - iceria.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.iceria.org' expires in 15 day(s) (Mon 31 Jan 2022 23:00:20 GMT +0000). [23:08:30] PROBLEM - www.iceria.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.iceria.org' expires in 15 day(s) (Mon 31 Jan 2022 23:00:20 GMT +0000). [23:08:32] I noticed some files dodgy [23:08:36] PROBLEM - www.bluepageswiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.bluepageswiki.org' expires in 15 day(s) (Mon 31 Jan 2022 22:56:37 GMT +0000). [23:08:40] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.wikimicrofinanza.it' expires in 15 day(s) (Mon 31 Jan 2022 22:54:41 GMT +0000). [23:08:53] RhinosF1, yeah, like my image on my global user page [23:09:01] One on mine too [23:09:12] I think some might be cache issues [23:09:21] perhaps yeah [23:09:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.12, 9.26, 8.87 [23:09:53] Thank you guys, just checked and all sorted 😁 [23:10:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.92, 8.52, 8.00 [23:10:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.83, 9.08, 8.53 [23:11:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.50, 11.19, 9.30 [23:11:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.56, 9.67, 8.92 [23:14:08] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 846s [23:14:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.28, 10.72, 9.30 [23:14:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.38, 10.97, 9.63 [23:14:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:16:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.40, 10.99, 9.66 [23:16:42] PROBLEM - arru.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.arru.xyz' expires in 15 day(s) (Mon 31 Jan 2022 23:11:41 GMT +0000). [23:16:43] PROBLEM - wiki-asterix.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki-asterix.cf' expires in 15 day(s) (Mon 31 Jan 2022 23:08:28 GMT +0000). [23:17:23] PROBLEM - www.arru.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.arru.xyz' expires in 15 day(s) (Mon 31 Jan 2022 23:11:41 GMT +0000). [23:17:30] !log [@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 1047s [23:17:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:17:55] PROBLEM - www.zenbuddhism.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.zenbuddhism.info' expires in 15 day(s) (Mon 31 Jan 2022 23:06:32 GMT +0000). [23:19:26] hey my wiki won't display SVG thumbnails [23:19:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.90, 3.90 [23:19:38] RECOVERY - mon111 Check correctness of the icinga configuration on mon111 is OK: Icinga configuration is correct [23:20:27] It's a known issue due to migration I think, I'd have to use Discord search to see what the latest status is on this. [23:21:50] i know [23:21:59] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:22:02] i want everything back to normal [23:22:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.41, 9.96, 8.98 [23:22:26] It takes time. [23:22:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 4.24, 4.03 [23:22:39] well how long [23:22:41] hours [23:22:41] days [23:22:43] weeks [23:22:44] We might have to have some patience.Last I heard it was a sync issue needed to be waited for https://discord.com/channels/407504499280707585/407537962553966603/931776495809081374 [23:22:44] [url] Discord | discord.com [23:22:54] And, it was already commented, there's no current ETA I think [23:23:06] Likely another day or 2. But that shouldn't be too long. [23:23:16] Or possibly a few hours. [23:23:54] Ah yes, here's where Agent replied with no eta https://discord.com/channels/407504499280707585/903395171981459507/932032701228273674 [23:23:54] [url] Discord | discord.com [23:24:04] I don't know the exact time, but I'm just taking a crack of a guess on this one. [23:24:26] Patience is hard [23:25:06] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.135 second response time [23:25:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.77, 3.88 [23:26:13] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:26:25] I've been playing Stardew Valley 😆 [23:26:35] Such as keeping our wikis up to speed, including several of my other wikis I've set up recently, including the latest one, wiisportswiki: https://wiisports.miraheze.org/wiki [23:26:36] [url] Wii Sports Wiki | wiisports.miraheze.org [23:26:37] Years ;) [23:27:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.89, 3.91 [23:28:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.73, 8.71, 9.90 [23:29:08] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:30:13] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:30:29] DarkMatterMan4500, ah that's your wiki. That explains where my local user rights came from then ;) [23:31:01] Yeah, I gave that to you. I was actually importing some templates that I may find useful. [23:31:12] Ah, okay, cool :) [23:31:19] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:31:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.86, 3.79, 3.85 [23:31:40] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:31:44] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:33:06] [02mw-config] 07Universal-Omega closed pull request 03#4386: Use memcached for ParserCache - 13https://github.com/miraheze/mw-config/pull/4386 [23:33:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/23eb0bc31d4b...429852b61ba5 [23:33:09] [02miraheze/mw-config] 07Universal-Omega 03429852b - Use memcached for ParserCache (#4386) [23:33:11] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [23:33:12] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [23:33:42] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.500 second response time [23:34:08] miraheze/mw-config - Universal-Omega the build passed. [23:34:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.39, 3.93, 3.88 [23:35:15] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.070 second response time [23:35:51] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.91, 10.12, 9.94 [23:36:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:36:11] !log [@mwtask111] starting deploy of {'config': True} to scsvg [23:36:24] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [23:36:40] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 29s [23:36:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:37:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:39:15] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:41:25] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:42:24] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:42:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:43:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 4.00, 3.92 [23:44:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:44:13] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:44:17] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:44:21] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:44:44] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:44:50] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:45:25] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:45:25] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.77, 4.06, 3.96 [23:46:42] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.937 second response time [23:46:47] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.041 second response time [23:46:50] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.062 second response time [23:47:08] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.481 second response time [23:47:16] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.734 second response time [23:47:41] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 6.709 second response time [23:48:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/3b40b9bd208b [23:48:10] [02miraheze/puppet] 07paladox 033b40b9b - varnish: send request to mw122 (DO NOT MERGE) [23:48:11] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [23:48:13] [02puppet] 07paladox opened pull request 03#2338: varnish: send request to mw122 (DO NOT MERGE) - 13https://github.com/miraheze/puppet/pull/2338 [23:48:15] !log [@test101] starting deploy of {'config': True} to skip [23:48:16] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [23:49:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:50:48] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:19] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.801 second response time [23:51:38] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.729 second response time [23:52:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:52:28] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:52:28] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.881 second response time [23:52:59] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:53:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.69, 3.72, 3.90 [23:53:30] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.172 second response time [23:54:48] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.997 second response time [23:55:16] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.195 second response time [23:56:39] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:58] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:57:05] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:57:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.56, 4.13, 4.02 [23:58:25] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:59:25] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.616 second response time