[00:00:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.37, 9.48, 9.31 [00:00:53] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.39, 9.37, 8.77 [00:02:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.82, 9.34, 8.81 [00:02:56] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 565 failures. Last run 2 minutes ago with 565 failures. Failed resources (up to 3 shown): File[wiki.thunis.eu_private],File[commons.gyaanipedia.com],File[commons.gyaanipedia.com_private],File[wiki.pablojprieto.com.es] [00:03:06] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 1782 MB (18% inode=80%); [00:04:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 3.87, 3.33 [00:04:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.15, 4.62, 3.75 [00:05:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.31, 8.11, 8.41 [00:05:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.47, 9.40, 8.65 [00:06:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.63, 3.32 [00:08:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.16, 9.21, 9.11 [00:09:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.94, 9.10, 8.67 [00:11:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.08, 9.11, 8.74 [00:12:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.21, 8.59, 8.94 [00:13:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.06, 9.14, 9.11 [00:13:58] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.77, 4.02, 3.34 [00:13:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.83, 3.50 [00:15:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.72, 3.50 [00:15:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.92, 3.61, 3.27 [00:16:16] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.21, 7.92, 8.42 [00:16:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.07, 9.92, 9.35 [00:17:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.44, 3.32, 3.38 [00:17:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.64, 3.30, 3.20 [00:18:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.22, 9.05, 9.13 [00:18:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.52, 9.77, 9.39 [00:19:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.34, 10.87, 9.57 [00:20:48] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.61, 9.71, 9.42 [00:22:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.73, 3.53, 4.00 [00:22:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.42, 9.17, 9.07 [00:24:38] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.14, 9.73, 9.48 [00:24:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.24, 8.98, 9.01 [00:25:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.77, 9.95, 9.64 [00:25:37] RECOVERY - mw111 NTP time on mw111 is OK: NTP OK: Offset -0.0149089992 secs [00:26:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.12, 9.37, 9.15 [00:27:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.58, 10.55, 9.90 [00:27:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.18, 7.80, 7.14 [00:29:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.78, 9.75, 8.95 [00:29:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.09, 8.40, 7.43 [00:30:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.34, 9.98, 9.49 [00:31:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.28, 9.36, 8.89 [00:31:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.42, 8.73, 7.67 [00:32:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.71, 9.79, 9.71 [00:33:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.66, 9.42, 8.94 [00:33:11] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:34:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.42, 10.07, 9.82 [00:34:30] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.51, 2.85, 3.37 [00:35:38] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.87, 8.49, 7.82 [00:38:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.57, 9.60, 9.75 [00:39:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.85, 9.95, 10.00 [00:39:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.63, 8.53, 7.85 [00:40:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.20, 9.99, 9.88 [00:40:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.10, 10.09, 9.55 [00:41:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.93, 9.20, 9.18 [00:41:20] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.84, 8.06, 7.74 [00:41:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.45, 10.24, 10.13 [00:42:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.65, 9.28, 9.65 [00:43:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.57, 9.72, 9.37 [00:43:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.45, 9.00, 9.70 [00:44:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.87, 9.34, 9.44 [00:47:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.80, 9.62, 9.46 [00:48:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.01, 8.58, 9.12 [00:49:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.65, 9.27, 9.51 [00:50:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.58, 4.06, 3.55 [00:51:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.52, 3.22, 2.97 [00:52:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.71, 3.74, 3.50 [00:52:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.89, 9.15, 9.27 [00:55:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.71, 3.21 [00:56:55] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.27, 8.55, 8.02 [00:57:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.45, 3.17 [00:57:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:58:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.08, 9.09, 9.00 [00:58:55] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.12, 8.29, 7.98 [00:59:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.96, 9.16, 9.15 [00:59:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.58, 3.24 [00:59:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.77, 3.17, 3.37 [01:00:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.05, 9.16, 9.04 [01:01:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.39, 9.00, 9.11 [01:01:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.92, 3.00, 3.07 [01:02:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.46, 9.57, 9.20 [01:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:04:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.39, 9.07, 9.06 [01:06:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.88, 8.00, 8.47 [01:07:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.24, 8.23, 7.75 [01:09:02] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.67, 7.70, 7.62 [01:09:08] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.19, 9.81, 9.39 [01:10:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.93, 10.11, 9.42 [01:11:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.48, 8.83, 9.08 [01:11:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.40, 3.50, 3.10 [01:12:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.16, 9.24, 9.19 [01:15:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.60, 9.26, 9.86 [01:17:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.85, 10.38, 10.17 [01:17:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.23, 3.16 [01:18:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.31, 8.82, 8.48 [01:19:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.88, 7.59, 8.41 [01:19:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.88, 3.91, 3.54 [01:19:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.85, 3.25, 3.19 [01:20:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.98, 8.19, 8.30 [01:21:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.72, 4.11, 3.67 [01:24:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.26, 9.46, 9.21 [01:30:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.98, 9.50, 9.40 [01:31:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.35, 3.67, 3.80 [01:32:55] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.71, 8.67, 8.28 [01:33:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.15, 4.40, 4.03 [01:34:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.15, 9.80, 9.48 [01:34:55] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.02, 8.09, 8.12 [01:36:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.99, 9.35, 8.67 [01:36:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.82, 8.66, 8.45 [01:37:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.47, 3.60, 3.80 [01:38:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.08, 8.81, 8.55 [01:38:39] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.64, 8.28, 8.34 [01:38:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.04, 9.10, 9.28 [01:40:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.22, 8.37, 8.40 [01:42:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.16, 9.52, 9.38 [01:44:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.13, 9.37, 8.86 [01:45:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.62, 3.88, 3.81 [01:46:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.95, 8.78, 8.46 [01:47:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.75, 3.73, 3.75 [01:48:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.34, 9.28, 9.32 [01:50:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.36, 8.34, 8.37 [01:50:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.66, 9.56, 9.02 [01:52:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.16, 7.73, 8.43 [01:52:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 3.11, 7.01, 8.47 [01:53:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.16, 2.75, 3.38 [01:53:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 1.72, 6.77, 8.98 [01:53:30] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:39] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.241 second response time [01:53:40] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.011 second response time [01:53:41] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.016 second response time [01:53:51] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [01:53:56] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.240 second response time [01:53:59] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [01:54:04] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [01:54:07] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [01:54:10] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.012 second response time [01:54:29] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:54:30] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:54:30] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:54:34] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1039 bytes in 0.497 second response time [01:54:37] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is WARNING: WARNING - NGINX Error Rate is 42% [01:54:51] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw112 [01:55:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 1.47, 5.06, 8.08 [01:55:25] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.021 second response time [01:55:36] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 31.18, 43.97, 21.84 [01:55:36] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.260 second response time [01:55:39] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.329 second response time [01:55:39] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 5.17, 3.57, 2.01 [01:55:45] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:48] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [01:55:50] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.025 second response time [01:55:52] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) [01:55:58] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.018 second response time [01:56:02] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [01:56:02] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [01:56:04] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.261 second response time [01:56:24] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.329 second response time [01:56:29] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.334 second response time [01:56:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [01:56:37] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.571 second response time [01:56:37] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [01:56:42] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [01:56:57] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:57:39] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.81, 3.42, 2.13 [01:57:41] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.320 second response time [01:57:43] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [01:57:50] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.324 second response time [01:58:10] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.023 second response time [01:58:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:58:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:58:37] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 10% [01:58:51] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [01:58:52] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:00:02] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [02:00:42] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [02:01:48] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [02:11:19] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 0.00, 1.87, 7.89 [02:15:19] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.02, 0.86, 6.09 [02:17:21] Is Miraheze down right now? [02:18:01] I've been having issues with Miraheze, most being returning a 500 on all requests with a database connection error [02:20:28] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.49, 1.75, 1.15 [02:22:24] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 52004 bytes in 1.361 second response time [02:22:28] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.92, 1.49, 1.13 [02:22:43] !log [void@db121] restart mariadb [02:22:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:23:25] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 173 Threads: 29 Questions: 65978 Slow queries: 4072 Opens: 2521 Open tables: 2515 Queries per second avg: 381.375 [02:24:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:29:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:31:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.92, 7.98, 6.55 [02:31:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.84, 8.72, 7.17 [02:33:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.65, 7.73, 6.63 [02:33:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.64, 8.30, 7.21 [02:33:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:35:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.82, 8.73, 7.15 [02:36:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.81, 8.01, 6.91 [02:37:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.37, 8.94, 7.73 [02:39:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.56, 9.35, 7.78 [02:42:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.73, 8.00, 7.33 [02:42:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.87, 10.13, 8.28 [02:43:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.91, 8.63, 7.88 [02:44:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.96, 8.65, 7.97 [02:44:55] [discord] Not sure if anyone else is around to take a look at db121 logs, but I'm really not. I can give it a kick if it goes down again, but I'll soon be off for the night. [02:45:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.84, 8.38, 8.05 [02:48:44] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.07, 8.30, 7.96 [02:49:10] PROBLEM - mw122 NTP time on mw122 is WARNING: NTP WARNING: Offset 0.1558355391 secs [02:49:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.21, 9.05, 8.43 [02:55:22] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.44, 8.28, 8.11 [02:55:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.95, 9.94, 9.00 [02:57:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.13, 9.10, 8.80 [03:00:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.28, 8.96, 8.31 [03:00:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.67, 8.50, 8.09 [03:01:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.57, 9.46, 9.00 [03:01:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.81, 3.66, 3.03 [03:02:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.84, 8.56, 8.25 [03:02:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.55, 9.29, 8.43 [03:03:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.73, 3.16 [03:03:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.82, 9.26, 8.99 [03:04:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.74, 9.26, 8.53 [03:04:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.57, 9.04, 8.45 [03:05:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.02, 8.90, 8.44 [03:06:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.23, 8.92, 8.50 [03:06:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.03, 9.34, 8.62 [03:07:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 3.83, 3.29 [03:08:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.68, 9.36, 8.71 [03:09:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.83, 7.78, 8.12 [03:09:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.85, 3.36, 3.18 [03:10:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.09, 8.45, 8.40 [03:10:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.08, 9.81, 8.94 [03:12:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.89, 9.41, 8.90 [03:13:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.22, 7.59, 8.36 [03:13:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:16:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.58, 8.72, 8.55 [03:18:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.79, 9.34, 8.78 [03:20:10] RECOVERY - mw122 NTP time on mw122 is OK: NTP OK: Offset -0.03007873893 secs [03:20:10] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.98, 1.67, 1.33 [03:20:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.83, 9.08, 8.76 [03:20:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:21:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.05, 9.38, 8.74 [03:22:10] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.45, 1.59, 1.35 [03:23:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.73, 9.24, 8.78 [03:25:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:26:28] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.20, 2.15, 1.38 [03:26:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.92, 8.17, 8.50 [03:28:28] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.09, 1.75, 1.33 [03:29:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.17, 8.80, 8.28 [03:30:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.52, 7.94, 8.37 [03:30:28] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.50, 1.73, 1.35 [03:31:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.37, 9.05, 8.80 [03:31:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.05, 8.10, 8.08 [03:32:28] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.20, 1.51, 1.32 [03:32:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:33:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.67, 8.79, 8.76 [03:35:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.09, 7.77, 8.39 [03:36:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.86, 8.56, 8.49 [03:38:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.83, 8.07, 8.30 [03:42:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:44:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 3.22, 2.75 [03:46:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.71, 3.01, 2.73 [03:50:11] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.54, 9.23, 8.52 [03:50:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:52:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.03, 8.87, 8.48 [03:54:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.06, 8.34, 8.34 [03:55:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:59:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.20, 8.28, 7.70 [04:03:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.81, 8.15, 7.80 [04:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:08:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:20:28] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.88, 1.80, 1.26 [04:22:28] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.46, 1.52, 1.22 [04:27:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:32:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:51:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:54:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.39, 8.60, 7.87 [04:56:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:56:35] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.02, 8.07, 7.74 [05:22:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:27:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:50:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:55:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:26:28] RECOVERY - mw122 Disk Space on mw122 is OK: DISK OK - free space: / 2430 MB (11% inode=83%); [06:38:03] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [06:38:03] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:40:03] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/a1d4a010262d...1ab8b400e0a2 [06:40:05] [02miraheze/dns] 07Reception123 031ab8b40 - add thestories.kr zone [06:41:04] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/1ab8b400e0a2...160b08309b1a [06:41:06] [02miraheze/dns] 07Reception123 03160b083 - add obeymewiki.com zone [06:48:10] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.22, 1.62, 1.33 [06:49:49] [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+3/-0/±3] 13https://github.com/miraheze/ssl/compare/2883af045768...88d863f4c847 [06:49:51] [02miraheze/ssl] 07Reception123 0347201a6 - add wiki.ravynos.com cert [06:49:52] [02miraheze/ssl] 07Reception123 03a24590f - add thestories.kr cert [06:49:54] [02miraheze/ssl] 07Reception123 0388d863f - add www.obeymewiki.com cert [06:50:10] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.71, 1.40 [06:50:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:52:10] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.20, 1.53, 1.37 [06:54:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.40, 2.89 [06:55:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:56:52] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.72, 3.15, 2.87 [06:57:34] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=pmuniversewiki --username-prefix=w --report 1 /home/reception/Wikipedia-20220317163419.xml (START) [06:57:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:58:26] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=pmuniversewiki --username-prefix=w --report 1 /home/reception/Wikipedia-20220317163419.xml (END - exit=0) [06:58:28] PROBLEM - mw122 Disk Space on mw122 is WARNING: DISK WARNING - free space: / 2429 MB (10% inode=83%); [06:58:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:05:56] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=mappercraftwiki --search-recursively /home/reception/mappercraft --user=Miraheze_maintenance_script --comment="Imported from mappercraft.fandom.com" [07:06:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:25] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [07:09:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.63, 4.45, 3.24 [07:12:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.44, 2.98 [07:13:13] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.84, 1.75, 1.56 [07:14:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.59, 4.00, 3.23 [07:15:09] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.00, 1.50, 1.49 [07:16:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.53, 3.15 [07:17:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.16, 3.90, 3.63 [07:17:24] [discord] Ah, so it's not just on sites running DPL3 then [07:20:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.96, 2.82, 2.95 [07:21:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.86, 2.92, 3.31 [07:32:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.90, 4.11, 3.61 [07:34:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.61, 3.48 [07:38:27] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.70, 3.25, 3.38 [07:40:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:45:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:49:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.72, 1.48, 1.25 [07:51:10] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.10, 1.32, 1.21 [08:18:25] Reception123: Morning! [08:34:21] hi [08:42:03] Reception123: how was your night? [09:05:21] not bad :) [09:06:16] I see [09:11:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.52, 2.85 [09:13:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.42, 2.77, 2.65 [09:21:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:24:25] PROBLEM - mw122 NTP time on mw122 is WARNING: NTP WARNING: Offset 0.2834995091 secs [09:25:04] That's fun [09:25:12] That really shouldn't happen [09:26:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:29:46] Reception123: I just filed a UBN for mw122 [09:29:50] Cc paladox [09:33:06] Reception123: something definately happened overnight as the disk usage is an instant jump [09:34:45] RECOVERY - mw122 NTP time on mw122 is OK: NTP OK: Offset -0.02341210842 secs [09:43:53] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.34, 10.16, 7.76 [09:44:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 17.36, 11.66, 8.20 [09:45:17] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.97, 10.37, 7.31 [09:45:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:45:51] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.18, 9.77, 7.14 [09:47:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.41, 8.94, 7.18 [09:47:51] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.26, 8.38, 6.95 [09:49:17] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 4.12, 7.37, 6.82 [09:49:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.85, 9.71, 8.61 [09:50:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.20, 9.36, 8.39 [09:50:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:53:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.05, 8.44, 8.33 [10:02:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.83, 7.61, 8.22 [10:17:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.25, 1.64, 1.32 [10:19:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.95, 1.79, 1.42 [10:21:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.23, 1.57, 1.38 [10:33:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:38:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:39:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.07, 9.06, 7.98 [10:41:18] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.50, 7.97, 7.71 [10:57:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:07:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:12:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.78, 8.27, 7.55 [11:14:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.78, 7.36, 7.31 [11:18:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:28:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:35:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.20, 8.33, 7.79 [11:37:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.49, 9.03, 8.10 [11:39:16] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.99, 8.49, 8.02 [11:40:03] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 12.61, 8.38, 5.65 [11:45:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.92, 2.89, 2.44 [11:47:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.31, 2.91, 2.49 [11:51:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.92, 8.95, 8.00 [11:51:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.61, 8.71, 8.03 [11:53:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:55:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.53, 8.31, 8.04 [11:55:22] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.97, 7.78, 7.66 [11:58:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:59:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.71, 8.59, 8.27 [12:00:55] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.20, 8.39, 8.20 [12:01:13] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.67, 5.01, 6.43 [12:01:15] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.13, 8.25, 8.19 [12:05:12] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.78, 8.74, 7.85 [12:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:06:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.89, 8.80, 8.44 [12:07:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.43, 8.93, 8.04 [12:07:11] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.05, 8.91, 8.44 [12:08:42] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.36, 8.37, 8.34 [12:09:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.62, 8.46, 8.00 [12:11:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.59, 9.59, 8.90 [12:11:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:14:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:15:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.99, 9.47, 8.94 [12:17:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.70, 9.36, 8.98 [12:19:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.94, 8.86, 8.20 [12:20:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.42, 9.51, 8.64 [12:21:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.08, 9.65, 9.19 [12:22:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.60, 9.11, 8.61 [12:22:41] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.44, 1.80, 1.28 [12:23:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.98, 9.66, 9.24 [12:23:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.49, 8.47, 8.21 [12:24:26]