[00:02:10] RECOVERY - mw132 Disk Space on mw132 is OK: DISK OK - free space: / 3203MiB (13% inode=69%); [00:02:55] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.92, 10.07, 9.28 [00:03:19] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 9.31, 9.98, 8.95 [00:08:28] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.35, 3.25, 2.24 [00:09:26] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.49, 3.35, 2.73 [00:10:28] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 3.19, 3.06, 2.29 [00:11:26] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.22, 3.22, 2.75 [00:18:20] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.57, 3.30, 2.76 [00:20:20] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.13, 3.05, 2.72 [00:24:15] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.51, 3.49, 2.99 [00:24:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.27, 6.87, 6.39 [00:26:13] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.24, 3.65, 3.10 [00:26:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.80, 6.23, 6.22 [00:26:59] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.06, 10.36, 9.20 [00:27:19] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.88, 9.54, 8.71 [00:28:59] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.71, 9.51, 9.02 [00:30:06] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.20, 3.67, 3.24 [00:32:04] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.57, 3.95, 3.39 [00:33:19] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.89, 9.50, 8.97 [00:35:36] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:35:58] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.76, 3.90, 3.55 [00:36:38] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 6.08, 4.30, 3.22 [00:36:59] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 15.01, 11.79, 10.13 [00:37:34] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.000 second response time [00:37:40] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.81, 3.84, 3.05 [00:38:26] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.86, 11.79, 9.66 [00:38:55] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.69, 11.26, 9.81 [00:39:40] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:39:53] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.73, 4.19, 3.71 [00:40:01] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:40:26] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.87, 11.78, 9.94 [00:40:37] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:40:55] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.94, 11.19, 9.99 [00:41:19] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 17.74, 13.03, 10.52 [00:41:36] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.20, 10.33, 8.31 [00:41:49] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.78, 3.90, 3.68 [00:42:00] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.437 second response time [00:42:01] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 14.57, 10.69, 8.02 [00:42:36] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.608 second response time [00:42:55] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.54, 12.42, 10.59 [00:43:37] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.76, 9.30, 8.17 [00:43:43] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.811 second response time [00:44:01] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 10.57, 10.33, 8.20 [00:44:55] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.55, 10.70, 10.19 [00:45:19] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.44, 10.93, 10.31 [00:45:26] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.29, 3.93, 3.50 [00:45:36] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [00:45:44] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 7.04, 4.82, 4.04 [00:46:01] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.55, 9.87, 8.33 [00:46:55] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.38, 9.41, 9.77 [00:48:26] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.82, 10.14, 9.88 [00:48:59] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 6.90, 10.72, 11.50 [00:49:19] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 6.98, 9.48, 9.90 [00:49:26] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.40, 3.20, 3.31 [00:49:36] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.64, 3.88, 3.86 [00:50:22] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.51, 3.68, 3.93 [00:51:40] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:53:37] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.722 second response time [00:54:59] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.74, 10.54, 11.02 [00:58:59] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.82, 11.83, 11.46 [01:00:26] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:00:36] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:01:24] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.43, 3.60, 3.60 [01:02:24] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.610 second response time [01:02:35] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.534 second response time [01:02:59] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.42, 12.35, 11.70 [01:03:20] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.62, 11.15, 10.06 [01:03:20] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.94, 3.38, 3.53 [01:03:26] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.18, 3.15, 3.06 [01:04:26] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.42, 11.35, 10.47 [01:04:34] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.47, 11.29, 10.22 [01:05:19] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.16, 9.55, 9.61 [01:05:26] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.76, 3.10, 3.06 [01:06:59] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.52, 10.98, 11.39 [01:07:23] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.99, 4.20, 3.80 [01:08:22] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.89, 2.71, 3.23 [01:08:23] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 5.71, 8.68, 9.43 [01:08:26] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.14, 9.71, 10.00 [01:09:22] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.27, 3.89, 3.75 [01:13:07] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.84, 7.22, 6.68 [01:13:23] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.79, 4.30, 3.89 [01:15:02] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.13, 6.75, 6.56 [01:17:00] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.08, 8.67, 9.91 [01:17:09] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [01:17:41] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds [01:17:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.31, 3.86, 3.83 [05:49:42] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp35.miraheze.org - CNAME OK [05:49:58] PROBLEM - cloud18 IPMI Sensors on cloud18 is CRITICAL: IPMI Status: Critical [6 system event log (SEL) entries present] [05:50:13] PROBLEM - test151 HTTPS on test151 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.217 second response time [05:50:40] PROBLEM - mwtask181 HTTPS on mwtask181 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.044 second response time [05:50:41] PROBLEM - db151 NTP time on db151 is CRITICAL: NTP CRITICAL: Offset 0.68553859 secs [05:50:43] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:50:43] PROBLEM - mw182 HTTPS on mw182 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.046 second response time [05:51:08] PROBLEM - mw151 HTTPS on mw151 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.073 second response time [05:51:33] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 10.13, 12.88, 12.41 [05:51:33] PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:33] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:34] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:34] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:35] PROBLEM - db142 Puppet on db142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 14.80, 13.72, 11.59 [05:51:35] PROBLEM - ldap141 Puppet on ldap141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:36] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:37] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:37] PROBLEM - mem141 Puppet on mem141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:37] PROBLEM - bast141 Puppet on bast141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:37] PROBLEM - mw133 Puppet on mw133 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:38] PROBLEM - mw134 Puppet on mw134 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 11.98, 12.06, 10.65 [05:51:40] PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:51] PROBLEM - reports121 Puppet on reports121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:51] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:53] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:54] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:54] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:51:59] PROBLEM - os131 Puppet on os131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:01] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:04] PROBLEM - swiftproxy131 Puppet on swiftproxy131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:05] PROBLEM - cp35 Puppet on cp35 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:05] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:06] PROBLEM - mem131 Puppet on mem131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:06] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.76, 11.30, 11.06 [05:52:08] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:09] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:10] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:10] PROBLEM - jobchron121 Puppet on jobchron121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:10] PROBLEM - cp24 Puppet on cp24 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:10] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:13] PROBLEM - swiftproxy181 Puppet on swiftproxy181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:14] PROBLEM - swiftobject101 Puppet on swiftobject101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:15] PROBLEM - mw143 Puppet on mw143 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:17] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 11.49, 9.87, 8.22 [05:52:25] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:25] PROBLEM - cp25 Puppet on cp25 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:27] PROBLEM - graylog131 Puppet on graylog131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:28] PROBLEM - swiftac111 Puppet on swiftac111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:33] PROBLEM - db131 Puppet on db131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:35] PROBLEM - swiftobject113 Puppet on swiftobject113 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:52:43] RECOVERY - db151 NTP time on db151 is OK: NTP OK: Offset 0.003643721342 secs [05:52:49] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.377 second response time [05:53:00] PROBLEM - swiftobject111 Puppet on swiftobject111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:53:11] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre RESOLVED: MediaWiki Exception Rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [05:54:26] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:55:44] RECOVERY - cp34 Puppet on cp34 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [05:55:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:56:01] RECOVERY - mem131 Puppet on mem131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:57:27] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [05:57:31] RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [05:57:47] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [05:58:17] RECOVERY - cp25 Puppet on cp25 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [05:59:31] RECOVERY - bast141 Puppet on bast141 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [05:59:32] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.76, 10.77, 11.62 [06:01:26] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:01:32] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.62, 10.98, 11.56 [06:01:42] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:01:55] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:02:07] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:02:25] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:02:34] RECOVERY - swiftobject101 Puppet on swiftobject101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:03:32] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.19, 10.62, 11.35 [06:03:35] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:04:16] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.37, 7.23, 7.85 [06:04:18] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:05:17] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:05:27] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:05:32] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.19, 11.38, 11.54 [06:05:50] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:07:24] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:07:32] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:07:53] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:07:54] RECOVERY - swiftproxy131 Puppet on swiftproxy131 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:08:59] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:09:03] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:09:38] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 16.21, 12.68, 11.22 [06:10:04] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:10:12] RECOVERY - cp24 Puppet on cp24 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:10:15] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.87, 7.81, 7.83 [06:10:25] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:10:42] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:11:02] RECOVERY - db142 Puppet on db142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:11:09] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:11:48] RECOVERY - cp35 Puppet on cp35 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:12:53] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:13:20] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:13:27] RECOVERY - mem141 Puppet on mem141 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:13:58] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:13:58] RECOVERY - swiftproxy181 Puppet on swiftproxy181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:14:06] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:14:14] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.28, 7.38, 7.77 [06:16:04] RECOVERY - os131 Puppet on os131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:16:14] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.07, 7.98, 7.93 [06:16:27] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:17:55] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:18:05] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:18:06] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:18:13] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.37, 7.33, 7.69 [06:20:13] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.01, 7.75, 7.81 [06:24:12] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.03, 6.92, 7.51 [06:25:32] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.29, 10.37, 11.58 [06:25:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:28:40] !log [universalomega@reports121] sudo php /srv/TSPortal/artisan config:clear [06:28:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:30:10] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.40, 5.59, 6.76 [06:31:29] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [06:31:32] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.75, 10.26, 11.00 [06:32:06] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [06:32:06] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [06:32:16] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [06:32:22] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [06:34:00] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw133 [06:34:04] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.584 second response time [06:34:05] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 0.635 second response time [06:34:12] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.601 second response time [06:34:20] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.655 second response time [06:35:29] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.549 second response time [06:35:32] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.74, 11.60, 11.47 [06:35:57] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 19 backends are healthy [06:43:38] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.29, 10.18, 11.78 [06:43:41] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.30, 9.78, 11.75 [06:46:26] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [06:49:38] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.38, 10.74, 11.94 [06:51:38] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.18, 10.08, 10.92 [06:51:39] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.48, 11.69, 12.13 [06:52:50] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=enablethedigbelowwiki --no-updates --username-prefix=wikia:gameknight999 /home/reception/ImportDump/enablethedigbelowwiki-20240122201030.xml (START) [06:52:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:53:38] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.37, 9.83, 10.74 [06:53:41] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.89, 11.18, 11.48 [06:55:32] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 15.29, 12.32, 11.11 [06:55:38] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 15.67, 11.67, 11.27 [06:57:32] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.49, 10.95, 10.77 [06:57:38] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.95, 10.63, 10.95 [06:57:41] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.61, 10.76, 11.32 [06:59:02] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.60, 6.83, 6.69 [06:59:56] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [07:01:01] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.38, 5.94, 6.38 [07:01:41] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.29, 10.33, 10.91 [07:03:41] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.28, 9.83, 10.68 [07:04:02] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 0.921 second response time [07:05:32] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.22, 9.05, 9.96 [07:05:38] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 9.75, 9.33, 10.17 [07:05:38] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.32, 10.90, 11.83 [07:05:41] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.28, 8.75, 10.17 [07:10:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.71, 6.78, 7.85 [07:13:38] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.76, 7.66, 9.92 [07:14:40] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.61, 8.48, 8.26 [07:15:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.42, 7.29, 8.00 [07:20:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.90, 8.11, 8.02 [07:30:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.29, 7.00, 7.75 [07:38:48] [02WikiDiscover] 07labster opened pull request 03#107: Improve Special:RandomWiki - 13https://github.com/miraheze/WikiDiscover/pull/107 [07:38:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.96, 7.20, 7.85 [07:40:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.50, 7.55, 7.56 [07:42:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.92, 7.50, 7.52 [07:44:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.75, 7.83, 7.85 [07:45:29] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [07:47:25] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.527 second response time [07:47:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.82, 2.33, 1.59 [07:48:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.18, 7.09, 7.26 [07:49:46] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.08, 1.80, 1.48 [07:50:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.64, 6.66, 7.09 [07:52:35] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 3.88, 5.92, 6.78 [07:52:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.14, 7.69, 7.98 [07:54:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.21, 8.72, 8.29 [07:56:38] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.68, 10.55, 9.11 [07:58:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.45, 6.83, 6.85 [07:58:38] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.34, 9.99, 9.08 [07:59:38] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.32, 10.27, 9.28 [08:00:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.29, 7.49, 7.87 [08:01:38] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 9.85, 10.03, 9.31 [08:04:35] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.01, 6.50, 6.69 [08:08:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.09, 6.94, 6.83 [08:10:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.03, 7.78, 7.61 [08:12:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 14.61, 10.25, 8.07 [08:13:38] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.74, 10.20, 9.59 [08:15:38] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.88, 9.37, 9.37 [08:18:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 4.78, 7.47, 7.62 [08:22:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.78, 7.37, 7.87 [08:26:55] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:27:29] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:27:38] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.67, 11.25, 9.70 [08:27:46] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.222 second response time [08:27:50] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.467 second response time [08:28:27] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:28:35] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.82, 5.71, 6.68 [08:28:36] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 17.00, 13.11, 10.50 [08:28:49] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.14, 9.91, 7.32 [08:28:53] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.597 second response time [08:29:08] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw131 mw134 [08:29:09] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 20.31, 14.74, 10.87 [08:29:26] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.188 second response time [08:29:32] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.29, 11.39, 9.20 [08:29:38] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.51, 11.57, 10.03 [08:29:39] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.92, 10.30, 7.76 [08:29:46] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.608 second response time [08:29:50] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.807 second response time [08:30:24] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.275 second response time [08:30:49] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 8.59, 9.82, 7.64 [08:31:08] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 19 backends are healthy [08:31:32] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.40, 9.76, 8.87 [08:31:39] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.01, 8.59, 7.44 [08:32:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.38, 7.71, 7.48 [08:32:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.36, 7.27, 7.10 [08:32:36] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 6.93, 10.69, 10.22 [08:33:01] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.17, 11.81, 10.54 [08:33:38] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.83, 9.49, 9.56 [08:34:36] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.72, 9.35, 9.79 [08:36:54] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.02, 9.94, 10.08 [08:38:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.60, 7.71, 7.60 [08:40:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [08:40:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.05, 7.80, 7.43 [08:42:39] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [08:44:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 15.67, 9.63, 8.31 [08:44:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 19.80, 10.25, 8.25 [08:48:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.14, 7.96, 7.97 [08:48:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.89, 7.91, 7.73 [08:50:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.75, 8.60, 8.19 [08:50:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.46, 8.15, 7.83 [08:52:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.89, 7.63, 7.89 [08:52:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.64, 7.85, 7.76 [08:58:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.29, 7.84, 7.72 [09:00:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.22, 7.43, 7.60 [09:08:13] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.88, 5.68, 6.59 [09:10:35] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.64, 6.04, 6.78 [09:10:39] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:12:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.20, 7.01, 6.90 [09:13:32] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 19.39, 13.36, 9.64 [09:13:47] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 20.69, 13.85, 10.80 [09:14:14] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 18.15, 13.36, 9.87 [09:14:33] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 20.31, 13.88, 9.91 [09:14:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.08, 7.29, 7.14 [09:18:33] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.26, 11.97, 10.04 [09:18:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.32, 7.44, 7.23 [09:20:33] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 17.34, 13.98, 11.01 [09:20:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.13, 7.29, 7.20 [09:22:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.24, 7.79, 7.35 [09:23:39] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.49, 9.38, 7.55 [09:25:39] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.01, 9.97, 7.97 [09:26:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.44, 7.09, 7.17 [09:27:39] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 8.81, 9.26, 7.94 [09:28:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.42, 7.78, 7.94 [09:28:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.05, 7.50, 7.29 [09:30:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.98, 6.98, 7.11 [09:31:43] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [09:33:41] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.557 second response time [09:34:36] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.25, 6.03, 6.71 [09:36:33] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.60, 10.65, 11.55 [09:39:32] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.06, 10.07, 11.90 [09:39:38] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 7.11, 9.63, 11.77 [09:39:59] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.66, 5.95, 6.79 [09:41:38] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 6.07, 8.18, 11.52 [09:42:33] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.47, 7.95, 10.06 [09:45:32] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 5.67, 7.44, 10.18 [09:45:38] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.41, 7.65, 10.20 [09:45:38] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 4.69, 6.72, 10.19 [10:01:56] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:03:32] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.47, 11.78, 10.02 [10:03:38] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.83, 10.49, 9.44 [10:03:54] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.596 second response time [10:04:25] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.52, 10.71, 9.84 [10:04:32] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.66, 10.40, 9.07 [10:05:32] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 5.78, 9.41, 9.36 [10:05:38] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.57, 9.78, 9.31 [10:06:21] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.51, 9.66, 9.54 [10:06:31] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.43, 9.58, 8.92 [10:11:32] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 16.44, 12.41, 10.56 [10:12:31] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.67, 11.62, 9.95 [10:13:32] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 7.70, 11.02, 10.31 [10:14:31] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.29, 9.52, 9.39 [10:15:32] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 5.32, 8.98, 9.65 [10:23:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.46, 6.81, 6.59 [10:27:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.41, 7.77, 7.03 [10:29:21] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.87, 7.04, 6.30 [10:29:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.47, 7.24, 6.93 [10:31:19] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.66, 6.79, 6.29 [10:33:41] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.98, 6.37, 6.67 [10:54:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.99, 6.91, 6.15 [10:55:32] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.32, 8.67, 7.62 [10:55:38] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.87, 9.96, 8.25 [10:56:48] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.15, 9.91, 9.09 [10:56:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.25, 6.61, 6.13 [10:57:32] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.93, 8.79, 7.80 [10:57:38] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 9.84, 9.97, 8.48 [10:58:44] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 9.43, 9.68, 9.10 [11:04:25] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.14, 10.15, 8.72 [11:04:31] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.54, 10.63, 9.23 [11:04:35] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.71, 10.47, 9.61 [11:06:21] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.28, 10.97, 9.20 [11:06:31] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.69, 10.00, 9.55 [11:08:16] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.16, 9.70, 8.96 [11:08:23] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.41, 9.17, 8.99 [11:08:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.42, 6.87, 6.67 [11:10:25] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.43, 6.41, 6.53 [11:14:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.19, 7.04, 6.84 [11:16:22] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.71, 6.75, 6.74 [11:20:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.78, 7.24, 6.89 [11:22:18] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.14, 6.71, 6.73 [11:41:26] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [11:45:26] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [11:49:00] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.59, 11.19, 9.56 [11:49:28] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.55, 9.70, 8.59 [11:50:09] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.10, 11.62, 9.61 [11:50:56]