[00:02:28] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 15.81, 13.80, 6.54 [00:02:32] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.08, 4.49, 3.60 [00:04:32] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.51, 3.55, 3.37 [00:06:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.27, 4.13, 3.81 [00:06:31] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.47, 2.96, 3.18 [00:10:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.81, 3.38, 3.62 [00:14:01] PROBLEM - cp22 PowerDNS Recursor on cp22 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:14:06] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 19.70, 9.45, 4.78 [00:15:12] PROBLEM - cp22 Puppet on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:16:19] PROBLEM - cp22 SSH on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:17:07] RECOVERY - cp22 Puppet on cp22 is OK: OK: Puppet is currently enabled, last run 31 minutes ago with 0 failures [00:18:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.37, 4.21, 3.79 [00:18:17] RECOVERY - cp22 SSH on cp22 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [00:18:18] RECOVERY - cp22 PowerDNS Recursor on cp22 is OK: DNS OK: 9.207 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [00:22:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.87, 2.97, 3.45 [00:24:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.80, 2.55, 3.23 [00:25:55] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.19, 3.59, 3.21 [00:28:01] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.15, 3.20, 3.88 [00:29:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.13, 3.87, 3.47 [00:33:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.62, 2.75, 3.11 [00:34:01] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.90, 2.21, 3.25 [00:36:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.83, 3.45, 3.18 [00:38:54] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.36, 4.04, 3.42 [00:44:42] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.62, 3.56, 3.45 [00:46:38] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.00, 3.34, 3.38 [00:50:12] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.39, 1.87, 3.82 [00:54:12] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.17, 1.82, 3.32 [00:55:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.38, 4.00, 3.14 [00:57:41] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.15, 3.73, 3.16 [00:59:36] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.41, 4.04, 3.32 [01:01:32] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.94, 3.23, 3.11 [01:06:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.24, 4.48, 3.49 [01:16:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.73, 3.34, 3.68 [01:18:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.21, 4.10, 3.92 [01:24:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.68, 3.72, 3.97 [01:26:35] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.57, 4.46, 3.25 [01:30:01] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.93, 3.66, 2.36 [01:30:31] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.51, 3.30, 3.09 [01:32:01] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.60, 2.93, 2.27 [01:32:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9005b74b4ffb...c93afb5d9e48 [01:32:07] [02miraheze/mw-config] 07Universal-Omega 03c93afb5 - T9884: disable sidebar cache for nonciclopediawiki [01:33:20] miraheze/mw-config - Universal-Omega the build passed. [01:39:32] !log [@mwtask141] starting deploy of {'config': True} to all [01:39:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:39:39] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [01:39:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:40:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.11, 2.90, 3.31 [01:44:13] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.74, 3.56, 2.32 [01:48:12] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.10, 3.89, 2.78 [01:50:12] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.68, 2.74, 2.50 [01:54:25] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:56:40] !log [@test131] starting deploy of {'config': True} to all [01:56:41] !log [@test131] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [01:56:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:56:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:56:55] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [01:58:05] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/6be00e4862f9...2abafaa08c98 [01:58:08] [02miraheze/MirahezeMagic] 07Universal-Omega 032abafaa - Add miraheze-sitenotice-learnmore message [01:59:34] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [02:00:01] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.46, 3.77, 2.46 [02:04:01] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.41, 3.86, 2.94 [02:06:01] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.34, 2.66, 2.61 [02:07:33] miraheze/MirahezeMagic - Universal-Omega the build passed. [02:08:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.79, 4.50, 3.49 [02:15:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.81, 12.50, 8.60 [02:15:56] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:16:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.46, 3.88, 3.70 [02:17:04] PROBLEM - mw141 nutcracker process on mw141 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 115 (nutcracker), command name 'nutcracker' [02:17:23] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 5.72, 3.86, 2.64 [02:17:42] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.41, 9.92, 8.11 [02:20:01] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.15, 3.41, 2.68 [02:20:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.43, 2.94, 3.34 [02:20:15] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:22:01] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.00, 2.42, 2.40 [02:22:01] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:22:13] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:32] PROBLEM - mw131 nutcracker process on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 115 (nutcracker), command name 'nutcracker' [02:24:14] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.615 second response time [02:25:08] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.19, 5.44, 3.70 [02:25:15] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw131 [02:25:58] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw131 [02:26:12] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [02:27:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.44, 4.00, 3.38 [02:27:16] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [02:27:53] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [02:27:56] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [02:27:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.91, 3.25, 1.99 [02:28:59] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.05, 3.02, 3.10 [02:29:11] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:31:11] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.02, 4.25, 2.70 [02:31:58] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [02:32:51] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.55, 4.59, 3.20 [02:33:11] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.71, 3.73, 2.71 [02:33:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.50, 3.25, 2.52 [02:34:46] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.14, 3.24, 2.87 [02:35:11] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 0.86, 2.72, 2.46 [02:36:07] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:36:27] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.245 second response time [02:36:32] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:37:04] RECOVERY - mw141 nutcracker process on mw141 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [02:37:21] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw141 [02:37:42] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 2 backends are down. mw131 mw141 [02:37:43] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:37:46] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw141 [02:38:09] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.36, 4.26, 3.48 [02:38:13] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 2 backends are down. mw132 mw141 [02:39:44] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [02:40:07] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [02:40:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.72, 4.00, 3.48 [02:40:08] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.023 second response time [02:40:13] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.340 second response time [02:40:15] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [02:41:15] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [02:41:31] RECOVERY - mw131 nutcracker process on mw131 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [02:42:40] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.954 second response time [02:43:07] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:43:39] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [02:45:36] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [02:46:06] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 3.85, 4.10, 3.17 [02:46:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.07, 2.92, 3.21 [02:46:12] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.09, 2.66, 1.80 [02:46:48] PROBLEM - swiftobject115 PowerDNS Recursor on swiftobject115 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:46:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.07, 3.61, 3.02 [02:47:08] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:04] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.94, 3.31, 2.99 [02:48:12] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.81, 1.93, 1.64 [02:48:18] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:33] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:48] RECOVERY - swiftobject115 PowerDNS Recursor on swiftobject115 is OK: DNS OK: 0.875 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [02:48:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.92, 3.16, 2.93 [02:49:01] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:02] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:51:14] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:52:39] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.290 second response time [02:53:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.64, 3.82, 3.47 [02:53:04] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.29, 1.81, 3.70 [02:53:07] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.542 second response time [02:53:10] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.254 second response time [02:53:31] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.677 second response time [02:53:54] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [02:54:22] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.531 second response time [02:54:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.02, 3.74, 3.49 [02:55:03] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.35, 1.58, 3.38 [02:56:53] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.25, 3.25, 3.34 [02:57:24] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.85, 3.63, 3.31 [02:57:54] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.95, 3.47, 3.08 [02:58:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.90, 3.43, 3.15 [02:59:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.01, 4.03, 3.50 [02:59:53] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.71, 3.11, 2.98 [03:00:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.65, 2.82, 2.95 [03:04:39] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.74, 4.49, 3.75 [03:10:06] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:10:28] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.08, 3.93, 3.74 [03:12:32] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [03:14:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.12, 2.39, 3.14 [03:15:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.22, 3.76, 3.11 [03:16:00] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 5.73, 4.62, 3.23 [03:16:14] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.04, 2.98, 1.90 [03:17:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.38, 3.39, 3.08 [03:18:12] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.41, 3.08, 2.09 [03:18:35] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.34, 2.66, 3.88 [03:20:00] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 2.30, 3.72, 3.20 [03:22:00] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.30, 2.88, 2.95 [03:28:32] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.71, 2.37, 3.22 [03:30:11] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:32:32] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [03:34:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.52, 4.95, 3.54 [03:37:50] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:43:36] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [03:44:38] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.79, 3.75, 3.89 [03:48:29] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.18, 3.36, 3.65 [03:50:25] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.21, 2.76, 3.38 [03:53:20] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:24] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:55:03] PROBLEM - swiftobject115 PowerDNS Recursor on swiftobject115 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:55:14] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.333 second response time [03:59:11] RECOVERY - swiftobject115 PowerDNS Recursor on swiftobject115 is OK: DNS OK: 0.141 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [04:06:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.96, 3.95, 3.32 [04:08:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.79, 3.94, 3.40 [04:10:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.92, 3.40, 3.29 [04:11:55] PROBLEM - swiftobject115 PowerDNS Recursor on swiftobject115 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:18:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 11.54, 5.81, 4.09 [04:22:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.44, 3.50, 3.04 [04:25:41] PROBLEM - swiftobject115 SSH on swiftobject115 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:30:28] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.05, 3.34, 3.87 [04:30:46] PROBLEM - swiftobject115 Puppet on swiftobject115 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:33:58] RECOVERY - swiftobject115 SSH on swiftobject115 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [04:34:24] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.47, 4.82, 4.28 [04:35:43] RECOVERY - swiftobject115 Puppet on swiftobject115 is OK: OK: Puppet is currently enabled, last run 44 minutes ago with 0 failures [04:39:37] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [04:42:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.62, 3.59, 3.99 [04:47:55] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:48:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.46, 3.13, 3.57 [04:50:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.63, 3.20, 3.54 [04:50:48] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [04:51:38] RECOVERY - swiftobject115 PowerDNS Recursor on swiftobject115 is OK: DNS OK: 9.673 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [04:51:56] PROBLEM - swiftobject115 APT on swiftobject115 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [04:52:01] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.29, 2.41, 3.20 [04:54:38] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.42, 2.96, 2.56 [04:54:53] PROBLEM - swiftobject115 Puppet on swiftobject115 is WARNING: WARNING: Puppet last ran 1 hour ago [04:54:56] PROBLEM - swiftobject115 APT on swiftobject115 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:56:10] PROBLEM - swiftobject115 PowerDNS Recursor on swiftobject115 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:58:06] RECOVERY - swiftobject115 PowerDNS Recursor on swiftobject115 is OK: DNS OK: 1.691 second response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [04:58:38] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.25, 2.37, 2.43 [04:59:54] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.39, 3.74, 2.49 [05:03:46] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.73, 3.34, 3.15 [05:04:23] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:05:41] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.40, 3.42, 3.10 [05:05:43] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.05, 3.95, 3.40 [05:06:57] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [05:07:36] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.27, 2.43, 2.77 [05:09:38] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.20, 3.82, 3.47 [05:13:33] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.34, 2.77, 3.15 [05:22:10] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:22:24] PROBLEM - swiftobject114 PowerDNS Recursor on swiftobject114 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:24:25] RECOVERY - swiftobject114 PowerDNS Recursor on swiftobject114 is OK: DNS OK: 1.903 second response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [05:24:38] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.37, 4.08, 2.63 [05:24:48] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [05:26:38] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.58, 3.71, 2.67 [05:28:38] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.82, 4.23, 2.99 [05:29:08] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.14, 3.44, 2.94 [05:30:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.08, 3.33, 2.81 [05:31:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.44, 2.75, 2.75 [05:33:02] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:35:29] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [05:45:10] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.48, 3.80, 2.31 [05:49:07] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.92, 3.45, 2.56 [05:51:05] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.64, 2.53, 2.33 [06:01:08] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.12, 4.97, 3.06 [06:06:42] RECOVERY - swiftobject115 Puppet on swiftobject115 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:08:38] PROBLEM - swiftobject115 PowerDNS Recursor on swiftobject115 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:09:08] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.18, 3.85, 3.48 [06:11:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.31, 2.63, 3.08 [06:14:54] RECOVERY - swiftobject115 PowerDNS Recursor on swiftobject115 is OK: DNS OK: 9.153 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [06:16:56] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 4.18, 3.35, 2.45 [06:18:56] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 2.54, 3.03, 2.44 [06:19:42] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.27, 3.67, 2.80 [06:21:40] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.71, 2.57, 2.50 [06:23:32] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:26:01] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [06:28:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 2.27, 4.15, 3.29 [06:29:11] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 9.00, 11.60, 6.50 [06:29:26] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.63, 3.63, 3.00 [06:29:33] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.13, 3.70, 2.91 [06:30:35] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.58, 4.50, 2.84 [06:30:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.38, 3.32, 3.10 [06:31:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.16, 2.84, 2.78 [06:31:31] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.20, 2.72, 2.65 [06:31:40] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:32:33] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.52, 3.87, 2.78 [06:34:30] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.47, 2.96, 2.58 [06:36:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.09, 3.53, 3.15 [06:39:08] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.47, 2.19, 3.83 [06:40:22] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [06:40:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.44, 2.84, 2.98 [06:43:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.92, 1.69, 3.21 [06:43:21] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:47:08] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.36, 3.40, 3.57 [06:48:59] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [06:49:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.77, 2.74, 3.30 [06:51:57] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:52:13] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1027930379 secs [06:56:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.81, 3.43, 2.52 [06:58:38] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.49, 3.29, 2.60 [07:00:35] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [07:00:37] PROBLEM - swiftobject114 Puppet on swiftobject114 is WARNING: WARNING: Puppet last ran 1 hour ago [07:01:10] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 9.87, 5.59, 3.71 [07:03:33] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:05:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.00, 3.14, 3.14 [07:06:18] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [07:09:17] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:14:13] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08792665601 secs [07:15:07] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [07:30:55] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:33:48] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [07:34:13] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1043170989 secs [07:35:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.97, 4.09, 2.82 [07:36:13] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09991565347 secs [07:36:46] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:37:47] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.77, 3.92, 2.93 [07:38:03] PROBLEM - swiftobject115 PowerDNS Recursor on swiftobject115 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:39:42] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [07:41:42] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.31, 2.76, 2.69 [07:44:16] RECOVERY - swiftobject114 Puppet on swiftobject114 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:44:59] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:45:39] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.96, 3.57, 2.39 [07:47:37] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.09, 2.64, 2.20 [08:02:56] PROBLEM - swiftobject115 Puppet on swiftobject115 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:05:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.36, 3.44, 2.70 [08:09:05] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.93, 3.76, 3.07 [08:09:25] PROBLEM - swiftobject115 ferm_active on swiftobject115 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:11:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.83, 3.00, 2.86 [08:11:44] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [08:12:35] PROBLEM - swiftobject115 SSH on swiftobject115 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:14:10] PROBLEM - swiftobject115 conntrack_table_size on swiftobject115 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:14:42] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:15:16] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.68, 4.16, 2.56 [08:17:15] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.59, 3.59, 2.55 [08:19:14] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.57, 2.52, 2.29 [08:19:17] PROBLEM - swiftobject115 NTP time on swiftobject115 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:20:09] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) [08:20:32] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [08:20:48] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.13, 3.68, 3.07 [08:21:06] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 951 bytes in 0.333 second response time [08:22:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.92, 3.26, 3.02 [08:23:10] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.57, 3.23, 2.34 [08:25:10] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.36, 2.87, 2.32 [08:28:38] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:28:45] PROBLEM - swiftobject115 Disk Space on swiftobject115 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:29:10] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.02, 3.49, 2.69 [08:31:04] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [08:31:10] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.05, 3.33, 2.73 [08:34:02] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:36:20] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [08:42:08] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:54:38] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.92, 3.09, 1.97 [08:56:38] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.93, 2.99, 2.07 [08:56:43] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [08:59:41] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:01:07] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.479 second response time [09:02:09] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 158 Threads: 12 Questions: 46007 Slow queries: 5159 Opens: 2279 Open tables: 2273 Queries per second avg: 291.183 [09:02:38] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.00, 3.65, 2.78 [09:04:38] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.69, 2.60, 2.49 [09:04:53] PROBLEM - swiftobject114 PowerDNS Recursor on swiftobject114 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:07:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-2981-revert-2967-revert-2965-revert-2961-paladox-patch-5 [+0/-0/±2] 13https://github.com/miraheze/puppet/commit/5107dedef14d [09:07:32] [02miraheze/puppet] 07paladox 035107ded - Revert "Revert "swiftobject114/5: reduce workers" (#2981)" [09:07:34] [02puppet] 07paladox created branch 03revert-2981-revert-2967-revert-2965-revert-2961-paladox-patch-5 - 13https://github.com/miraheze/puppet [09:07:41] [02puppet] 07paladox opened pull request 03#2982: swiftobject114/5: reduce workers - 13https://github.com/miraheze/puppet/pull/2982 [09:07:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/691dd53dc2c9...7ac5b9625e29 [09:07:51] [02miraheze/puppet] 07paladox 037ac5b96 - swiftobject114/5: reduce workers (#2982) [09:07:52] [02miraheze/puppet] 07paladox deleted branch 03revert-2981-revert-2967-revert-2965-revert-2961-paladox-patch-5 [09:07:55] [02puppet] 07paladox closed pull request 03#2982: swiftobject114/5: reduce workers - 13https://github.com/miraheze/puppet/pull/2982 [09:07:56] [02puppet] 07paladox deleted branch 03revert-2981-revert-2967-revert-2965-revert-2961-paladox-patch-5 - 13https://github.com/miraheze/puppet [09:11:03] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:11:29] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:13:02] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.251 second response time [09:13:29] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.295 second response time [09:15:31] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.82, 2.96, 2.18 [09:15:42] RECOVERY - swiftobject115 conntrack_table_size on swiftobject115 is OK: OK: nf_conntrack is 0 % full [09:15:55] RECOVERY - swiftobject115 Disk Space on swiftobject115 is OK: DISK OK - free space: / 653848 MB (73% inode=95%); [09:16:07] RECOVERY - swiftobject115 ferm_active on swiftobject115 is OK: OK ferm input default policy is set [09:16:47] RECOVERY - swiftobject115 SSH on swiftobject115 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:16:57] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:17:09] RECOVERY - swiftobject115 NTP time on swiftobject115 is OK: NTP OK: Offset -0.0002038776875 secs [09:17:29] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.55, 2.58, 2.15 [09:18:21] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:20:21] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.206 second response time [09:21:05] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.283 second response time [09:26:21] PROBLEM - swiftobject115 Puppet on swiftobject115 is WARNING: WARNING: Puppet last ran 1 hour ago [09:26:36] RECOVERY - swiftobject114 PowerDNS Recursor on swiftobject114 is OK: DNS OK: 6.048 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [09:31:08] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.10, 3.49, 2.44 [09:33:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.52, 2.83, 2.33 [09:42:49] RECOVERY - swiftobject115 PowerDNS Recursor on swiftobject115 is OK: DNS OK: 1.328 second response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [09:43:47] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [09:46:45] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:48:24] PROBLEM - swiftobject115 APT on swiftobject115 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [09:49:08] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [09:50:27] PROBLEM - swiftobject115 APT on swiftobject115 is CRITICAL: APT CRITICAL: 16 packages available for upgrade (7 critical updates). [09:52:06] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:52:47] PROBLEM - swiftobject114 PowerDNS Recursor on swiftobject114 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:53:10] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:24] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:28] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:28] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:55:10] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.280 second response time [09:55:23] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.378 second response time [09:55:26] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.30, 3.89, 2.76 [09:55:27] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.202 second response time [09:55:28] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.216 second response time [09:57:22] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.09, 4.38, 3.07 [10:00:38] RECOVERY - swiftobject115 Puppet on swiftobject115 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [10:01:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.25, 3.43, 3.03 [10:03:08] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.29, 2.38, 2.70 [10:06:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.52, 3.72, 2.58 [10:08:11] !log removed entry from renameuser_status (TallAutism2006) [10:08:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:08:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.66, 3.17, 2.52 [10:09:51] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [10:12:49] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:16:42] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.90, 3.68, 2.62 [10:18:37] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [10:18:41] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.80, 2.60, 2.35 [10:31:26] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.20, 4.69, 3.15 [10:32:38] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:33:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.17, 3.35, 2.84 [10:44:04] PROBLEM - swiftobject114 Puppet on swiftobject114 is WARNING: WARNING: Puppet last ran 1 hour ago [10:45:15] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:27] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:43] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:19] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.867 second response time [10:47:26] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.192 second response time [10:47:42] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.192 second response time [10:49:35] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw121 [10:51:34] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [10:51:37] PROBLEM - swiftobject114 Puppet on swiftobject114 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:56:23] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [10:57:23] PROBLEM - swiftobject114 Puppet on swiftobject114 is WARNING: WARNING: Puppet last ran 1 hour ago [10:59:21] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:01:57] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [11:03:26] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [11:07:26]