[00:00:21] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [00:00:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/18119460b2e4...83483ac7db67 [00:00:24] [02miraheze/mw-config] 07Universal-Omega 0383483ac - Update MirahezeFunctions.php [00:00:49] !log [universalomega@test101] starting deploy of {'config': True} to all [00:00:50] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [00:01:30] miraheze/mw-config - Universal-Omega the build passed. [00:03:15] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:16] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:24] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:24] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:27] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:41] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:43] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:49] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:51] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:53] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:56] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 449 failures. Last run 2 minutes ago with 449 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:03:58] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:58] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:09] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:14] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:15] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:18] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:40] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [00:04:40] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:41] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:45] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:54] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [00:04:55] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:58] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:15] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:16] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:22] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:23] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:29] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:31] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:51] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:30] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:07:07] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw101 [00:07:56] PROBLEM - cp20 APT on cp20 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:08:47] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 8.788 second response time [00:08:58] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.023 second response time [00:08:58] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw102 [00:09:06] PROBLEM - cp31 Varnish Backends on cp31 is WARNING: No backends detected. If this is an error, see readme.txt [00:09:13] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw102 mw122 [00:09:56] RECOVERY - cp20 APT on cp20 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [00:10:58] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.28, 2.36, 1.63 [00:11:06] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [00:11:17] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.74, 1.68, 1.19 [00:11:49] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.125 second response time [00:12:17] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.155 second response time [00:12:57] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.15, 1.86, 1.53 [00:13:17] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.71, 1.29, 1.10 [00:13:24] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.110 second response time [00:14:56] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.98, 1.51, 1.44 [00:14:56] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [00:14:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.15, 3.34, 2.66 [00:15:06] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 3 backends are down. mw101 mw121 mw122 [00:15:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.08, 3.90, 3.16 [00:15:50] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.369 second response time [00:15:57] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.037 second response time [00:16:00] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:16:01] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.620 second response time [00:16:01] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:16:18] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:16:35] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.314 second response time [00:16:39] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:16:52] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.000 second response time [00:16:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.36, 2.91, 2.58 [00:16:57] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 2.307 second response time [00:17:01] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw101 mw102 mw121 [00:17:09] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 5.878 second response time [00:17:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.07, 3.50, 3.10 [00:18:08] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.313 second response time [00:18:12] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.330 second response time [00:18:55] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw102 mw112 [00:19:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.88, 3.98, 3.31 [00:19:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.47, 2.97 [00:20:34] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.085 second response time [00:20:35] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.887 second response time [00:20:42] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.653 second response time [00:21:12] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [00:21:18] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:21:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.62, 3.26 [00:22:01] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:09] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:16] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:22] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:26] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:36] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:52] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 3.256 second response time [00:23:01] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.881 second response time [00:23:01] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:07] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.316 second response time [00:23:09] wtf [00:23:21] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [00:23:27] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.44, 3.14, 3.12 [00:23:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.29, 2.93, 2.87 [00:23:49] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 1.381 second response time [00:23:49] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.331 second response time [00:23:50] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [00:23:54] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.307 second response time [00:23:56] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.322 second response time [00:24:16] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 3.051 second response time [00:24:43] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:54] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.294 second response time [00:25:04] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:25:05] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.020 second response time [00:25:31] icinga-miraheze shut up [00:25:56] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.71, 1.20, 0.83 [00:27:28] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [00:27:28] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.56, 3.33 [00:27:52] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.50, 0.93, 0.78 [00:27:59] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:11] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:11] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:13] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:17] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:29] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.951 second response time [00:28:30] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.010 second response time [00:28:42] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.387 second response time [00:28:45] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.169 second response time [00:28:49] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.984 second response time [00:28:49] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.312 second response time [00:28:52] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.029 second response time [00:28:53] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 2.028 second response time [00:29:04] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.580 second response time [00:29:05] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.721 second response time [00:29:25] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:29:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.23, 4.19, 3.57 [00:30:07] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.318 second response time [00:30:07] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.409 second response time [00:30:08] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.319 second response time [00:30:08] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 2.255 second response time [00:30:20] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.106 second response time [00:30:26] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.994 second response time [00:30:28] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 1.539 second response time [00:30:30] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.096 second response time [00:31:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.07, 3.48, 3.19 [00:31:54] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw101 mw122 [00:32:07] !log reboot mw101 [00:32:18] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:01] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [00:33:08] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:15] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.151 second response time [00:33:20] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:21] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 3.681 second response time [00:33:34] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.98, 3.84, 3.55 [00:33:47] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 6.896 second response time [00:33:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.49, 3.24, 3.14 [00:33:50] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [00:34:28] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:32] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:53] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.008 second response time [00:35:18] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:35:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.19, 3.20, 3.36 [00:35:53] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:36:11] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:36:13] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:36:31] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:36:35] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:36:40] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:36:46] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:37:01] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw102 [00:37:08] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:37:19] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.015 second response time [00:37:24] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.503 second response time [00:37:45] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.133 second response time [00:38:17] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.415 second response time [00:39:13] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.939 second response time [00:39:18] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [00:39:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/83483ac7db67...ceacbaad569c [00:39:21] [02miraheze/mw-config] 07Universal-Omega 03ceacbaa - Update MirahezeFunctions.php [00:39:26] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.031 second response time [00:39:31] !log [universalomega@test101] starting deploy of {'config': True} to all [00:39:32] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [00:40:01] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.635 second response time [00:40:09] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.319 second response time [00:40:27] miraheze/mw-config - Universal-Omega the build passed. [00:40:35] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.316 second response time [00:40:35] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.018 second response time [00:40:44] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 1.823 second response time [00:40:50] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 1.743 second response time [00:40:51] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.015 second response time [00:41:00] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.012 second response time [00:41:12] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:41:36] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:41:38] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:42:20] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:42:24] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 1.784 second response time [00:42:26] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:42:32] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [00:42:39] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.344 second response time [00:42:50] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.736 second response time [00:43:04] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.640 second response time [00:43:32] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.329 second response time [00:43:36] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:43:45] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.211 second response time [00:43:46] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 2.471 second response time [00:44:00] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.713 second response time [00:44:08] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:09] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.569 second response time [00:44:16] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:21] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.233 second response time [00:44:28] !!log restart nginx and php-fpm on mw112 [00:44:30] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.940 second response time [00:44:33] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.386 second response time [00:44:36] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.511 second response time [00:44:42] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:45] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:48] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:59] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.015 second response time [00:45:02] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:45:16] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:45:35] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:30] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:41] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:43] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.50, 6.85, 6.03 [00:46:56] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:47:01] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [00:47:11] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:47:13] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [00:47:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ceacbaad569c...2fe6c9feeb06 [00:47:16] [02miraheze/mw-config] 07Universal-Omega 032fe6c9f - Update MirahezeFunctions.php [00:47:25] !log [universalomega@test101] starting deploy of {'config': True} to all [00:47:26] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [00:47:39] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.667 second response time [00:47:57] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: connect to address 2a10:6740::6:309 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:47:57] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:47:59] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 2735 bytes in 0.325 second response time [00:48:20] miraheze/mw-config - Universal-Omega the build passed. [00:48:35] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:48:39] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.15, 6.59, 6.03 [00:48:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ae7114426af8...379dd5be9076 [00:48:42] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [00:48:42] [02miraheze/puppet] 07paladox 03379dd5b - mon111: switch to rsyslog [00:48:46] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.323 second response time [00:48:52] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 2.381 second response time [00:48:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/379dd5be9076...2df4fb8134aa [00:48:58] [02miraheze/puppet] 07paladox 032df4fb8 - matomo101: switch to rsyslog [00:49:12] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [00:49:17] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 8.244 second response time [00:49:42] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.150 second response time [00:49:53] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.839 second response time [00:50:31] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 2.652 second response time [00:50:34] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.014 second response time [00:50:36] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [00:50:57] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.341 second response time [00:51:01] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw102 mw112 [00:51:01] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.323 second response time [00:51:03] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 2.978 second response time [00:52:00] !log restart nginx and php-fpm on mw* [00:52:01] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.469 second response time [00:52:10] !log restart nginx and php-fpm on mw* [00:52:20] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:52:28] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:52:39] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:52:45] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [00:52:46] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [00:52:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2fe6c9feeb06...9e01ebeb5c33 [00:52:48] [02miraheze/mw-config] 07Universal-Omega 039e01ebe - Update MirahezeFunctions.php [00:52:55] paladox: logbot is having issues [00:52:56] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.234 second response time [00:53:01] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [00:53:01] !log [universalomega@test101] starting deploy of {'config': True} to all [00:53:02] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:53:02] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [00:53:06] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:53:06] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [00:53:33] !log [universalomega@test101] starting deploy of {'config': True} to all [00:53:34] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [00:53:50] !log test [00:53:54] miraheze/mw-config - Universal-Omega the build passed. [00:54:02] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:54:14] it keeps disconnecting [00:54:16] think it's because MW is so god damn slow [00:54:29] it cannot log because timeouts i guess [00:54:36] i cannot load MW right now [00:54:38] Looks like ^ is related to the persistent 502s [00:54:38] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:54:53] mw team need to look into why MW is so slow [00:54:59] paladox: but why does it keep disconnecting from IRC [00:55:05] paladox, I feel like MediaWiki is unnecessarily bloated software :p [00:55:24] i have no ideas, logs aren't telling me anything [00:55:36] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:45] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [00:55:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9e01ebeb5c33...3e9f7c49b7a1 [00:55:46] I've restarted nginx and php-fpm everywhere but seemingly no effect [00:55:48] [02miraheze/mw-config] 07Universal-Omega 033e9f7c4 - Update MirahezeFunctions.php [00:55:57] yeh it's not going to help [00:55:58] !log [universalomega@test101] starting deploy of {'config': True} to all [00:55:59] PROBLEM - test101 MediaWiki Rendering on test101 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2550 bytes in 0.693 second response time [00:55:59] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [00:56:02] all childs are getting used up [00:56:04] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:56:11] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:56:24] https://www.irccloud.com/pastebin/eCygnuin/ [00:56:27] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:56:33] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:56:37] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:56:44] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw111 mw121 [00:56:48] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:56:50] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:56:51] miraheze/mw-config - Universal-Omega the build passed. [00:57:01] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw101 [00:57:08] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:57:11] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.241 second response time [00:57:26] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:57:27] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [00:57:56] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.225 second response time [00:57:59] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 2677 bytes in 0.659 second response time [00:58:08] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.082 second response time [00:58:54] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.602 second response time [00:59:16] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [00:59:21] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:59:32] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:00:18] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.423 second response time [01:00:30] paladox, would it help if we potentially locked the databases for, say, 5 minutes, in order to do a complete shut down and gradual restart of all cache proxies and MediaWiki backends maybe? [01:00:38] no [01:00:47] oh [01:00:49] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.611 second response time [01:01:04] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.226 second response time [01:01:04] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.174 second response time [01:01:21] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.012 second response time [01:01:28] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 1.110 second response time [01:01:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:01:34] dmehus: its a php-fpm issue [01:01:59] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [01:02:11] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.330 second response time [01:02:22] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:02:37] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.022 second response time [01:02:42] MacFan4000, yeah I was just thinking to kill everything using the php childs then slowly bring everything back, but I guess it wouldn't do anything [01:02:44] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 1.190 second response time [01:02:45] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 2.269 second response time [01:02:50] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.024 second response time [01:02:55] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.318 second response time [01:03:03] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [01:03:05] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.537 second response time [01:03:07] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.027 second response time [01:03:07] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.251 second response time [01:03:10] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.716 second response time [01:03:13] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.253 second response time [01:03:16] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.320 second response time [01:03:23] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.167 second response time [01:03:29] !log sudo service php7.4-fpm restart on mw* [01:03:42] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.359 second response time [01:03:44] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw101 mw102 [01:03:45] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.015 second response time [01:03:48] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 3.034 second response time [01:04:05] strange, I did that too, but it didn't have much effect [01:04:38] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.331 second response time [01:04:42] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [01:05:01] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [01:05:05] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 2.754 second response time [01:05:12] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw122 [01:05:16] CosmicAlpha, seems to have helped a bit. I can at least access Special:Log and Special:Search now [01:05:38] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [01:06:14] yeah, alerts are still coming back [01:06:28] !log test [01:07:03] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:20] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:28] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:32] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:36] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:43] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:53] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:08:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:15] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.170 second response time [01:08:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.27, 3.56, 3.13 [01:08:34] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [01:08:38] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.447 second response time [01:08:50] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.992 second response time [01:09:00] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.435 second response time [01:09:05] !log sudo service php7.4-fpm restart on mw* [01:09:05] !log reboot mw122 and mw112 [01:09:12] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [01:09:18] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.013 second response time [01:09:27] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [01:09:41] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.885 second response time [01:09:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.51, 3.33, 2.79 [01:09:56] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 7.729 second response time [01:10:04] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.85, 7.93, 6.78 [01:10:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:10:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.93, 3.33 [01:10:39] MacFan4000: rebooted those 2 servers. we will see if it stays recovered this time. [01:10:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.45, 2.71 [01:11:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.96, 2.80, 2.67 [01:12:00] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.80, 6.66, 6.44 [01:12:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.39, 3.30, 3.17 [01:12:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:12:37] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.75, 2.74, 2.53 [01:12:58] were there any extensions installed recently [01:13:02] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:13:07] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:13:09] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:13:19]