[00:03:12] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.13, 11.01, 9.90 [00:06:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.35, 1.76, 1.95 [00:07:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.89, 11.36, 10.27 [00:15:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.30, 10.01, 9.43 [00:16:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.23, 1.61 [00:18:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [00:24:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.13, 9.53, 9.62 [00:36:47] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.03, 1.85, 1.62 [00:38:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.25, 1.67, 1.58 [00:43:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [00:46:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.22, 9.40, 10.02 [00:48:50] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.79, 1.71, 1.27 [00:50:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.85, 10.39, 10.26 [00:50:42] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.93, 1.79, 1.10 [00:50:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.29, 1.74, 1.63 [00:50:48] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.58, 1.62, 1.28 [00:52:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.62, 1.34, 1.02 [00:52:47] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.46, 2.23, 1.81 [01:00:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.30, 1.91, 1.89 [01:02:12] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.12, 9.66, 10.11 [01:02:47] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.92, 2.76, 2.20 [01:06:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.06, 1.89, 1.97 [01:12:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.27, 1.66 [01:21:44] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.61, 1.89, 1.73 [01:25:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.00, 10.48, 10.25 [01:27:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.55, 1.67 [01:29:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.46, 9.74, 10.07 [01:35:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.58, 11.23, 10.51 [01:37:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.47, 11.18, 10.58 [01:39:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.14, 11.65, 10.84 [01:41:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.14, 11.25, 10.78 [01:47:40] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.53, 1.91, 1.59 [01:51:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.79, 9.20, 8.29 [01:51:40] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.67, 1.86, 1.65 [01:51:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.36, 11.11, 10.80 [01:53:40] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.53, 1.70, 1.61 [01:54:55] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.52, 9.07, 8.43 [01:55:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.69, 11.79, 11.19 [01:56:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.90, 1.82, 1.59 [01:57:40] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.69, 1.75, 1.65 [01:58:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.68, 1.68, 1.56 [01:59:40] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.58, 2.06, 1.77 [01:59:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.38, 11.47, 11.13 [02:01:40] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.97, 1.67, 1.67 [02:11:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.26, 11.57, 11.56 [02:20:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [02:21:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.46, 11.16, 11.13 [02:22:07] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.73, 1.71, 1.51 [02:25:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.68, 11.61, 11.35 [02:26:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.25, 1.97, 1.67 [02:30:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.21, 1.68, 1.62 [02:35:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [02:35:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.39, 11.46, 11.33 [02:37:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.85, 11.27, 11.28 [02:41:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.29, 11.59, 11.35 [02:43:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.34, 11.01, 11.17 [02:57:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.73, 9.58, 10.16 [03:03:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.66, 10.32, 10.26 [03:05:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [03:05:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 15.00, 11.94, 10.86 [03:07:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.10, 1.68, 1.41 [03:07:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.66, 11.99, 11.03 [03:11:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.12, 1.50, 1.40 [03:15:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [03:21:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.61, 9.24, 10.16 [03:27:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.44, 10.54, 9.78 [03:33:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 10.07, 10.18, 9.86 [03:37:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.99, 10.21, 10.09 [03:39:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.97, 10.10, 10.08 [03:40:40] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.95, 2.35, 1.79 [03:42:38] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.71, 1.77, 1.64 [03:44:35] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.18, 1.59, 1.59 [03:46:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.61, 3.50, 3.93 [03:48:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.94, 3.28, 3.96 [03:53:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.13, 3.21, 3.82 [03:54:13] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.98, 3.51, 3.78 [03:54:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.27, 3.61 [03:54:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.45, 10.99, 10.32 [03:56:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.94, 10.87, 10.35 [03:57:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.94, 1.60 [03:57:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 3.82, 3.93 [03:58:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.71, 11.46, 10.62 [03:59:07] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.18, 1.69, 1.54 [03:59:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.68, 3.89 [04:00:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.64, 10.74, 10.49 [04:02:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.63, 3.63 [04:02:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.84, 10.06, 9.19 [04:04:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.75, 3.66 [04:04:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.68, 9.71, 9.18 [04:06:12] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.27, 3.69, 3.87 [04:06:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 3.64, 3.63 [04:10:30] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.68, 9.24, 9.92 [04:11:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.04, 2.66, 3.36 [04:14:13] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.52, 2.66, 3.30 [04:14:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.70, 3.03, 3.37 [04:15:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.40, 3.52 [04:20:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.44, 10.26, 10.02 [04:22:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.96, 10.07, 10.00 [04:23:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.48, 2.63, 3.20 [04:36:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.29, 3.19 [04:38:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.62, 3.22 [04:38:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.83, 3.42 [04:40:13] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.93, 4.31, 3.60 [04:40:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.89, 3.50 [04:42:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.85, 3.46 [04:44:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 3.96, 3.60 [04:46:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.94, 3.70 [04:48:13] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.35, 4.53, 3.97 [04:48:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.24, 3.71, 3.63 [04:50:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.89, 3.54, 3.67 [04:50:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.22, 3.00, 3.30 [04:50:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.96, 3.72 [04:51:40] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 2.00, 1.62, 1.35 [04:52:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.27, 3.64, 3.63 [04:54:13] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.07, 4.05, 3.82 [04:54:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.08, 4.14, 3.81 [04:55:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.40, 1.67, 1.45 [04:56:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.10, 4.63, 3.91 [04:56:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.09, 3.85, 3.76 [04:57:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.17, 11.25, 10.09 [04:58:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.90, 4.48, 3.99 [04:59:34] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.45, 3.67, 2.37 [04:59:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.51, 10.43, 9.95 [05:05:31] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.84, 1.84, 1.96 [05:05:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.91, 10.17, 9.00 [05:07:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.77, 2.17, 2.06 [05:07:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.99, 9.56, 8.91 [05:11:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.69, 1.55, 1.84 [05:13:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.45, 9.56, 8.67 [05:15:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.85, 9.23, 8.66 [05:16:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.20, 3.36, 3.94 [05:17:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.68, 1.17, 1.59 [05:17:44] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [05:17:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.16, 11.10, 10.70 [05:18:05] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:26] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:34] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:50] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 12.51, 6.59, 3.28 [05:18:50] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:19:23] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:20:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.92, 3.16, 3.86 [05:22:14] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:22:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.00, 2.37, 3.31 [05:23:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.90, 1.78, 1.73 [05:23:53] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:24:09] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:24:29] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:24:49] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 32 minutes ago with 0 failures [05:24:57] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:25:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.03, 1.51, 1.64 [05:25:35] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.418 second response time [05:25:48] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [05:26:10] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [05:26:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.81, 3.57, 3.57 [05:26:24] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 0.345 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [05:26:34] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.333 second response time [05:26:35] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 28504 bytes in 0.614 second response time [05:26:46] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.328 second response time [05:27:15] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.040 second response time [05:28:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.57, 3.97 [05:28:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.63, 3.59 [05:31:56] PROBLEM - dnd.bellinrattin.it - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'dnd.bellinrattin.it' expires in 7 day(s) (Sun 31 Jul 2022 05:03:24 GMT +0000). [05:32:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.24, 9.43, 8.71 [05:33:51] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.72, 2.93, 3.39 [05:34:20] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.77, 9.15, 8.69 [05:36:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.68, 3.57 [05:38:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.30, 3.42, 3.48 [05:41:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.99, 11.33, 11.85 [05:42:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.94, 3.10, 3.33 [05:48:09] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.52, 1.73, 3.78 [05:51:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.34, 3.29 [05:53:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.45, 3.32 [05:54:07] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.70, 2.29, 3.34 [05:55:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.45, 3.35 [05:56:13] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.63, 3.15, 3.37 [05:57:41] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.67, 3.12, 3.23 [06:07:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.13, 9.62, 10.16 [06:10:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.64, 3.90, 3.52 [06:11:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.28, 10.85, 10.54 [06:12:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.88, 3.65, 3.32 [06:12:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.89, 3.58 [06:13:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.59, 9.96, 10.23 [06:14:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.39, 3.51, 3.13 [06:15:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.64, 9.24, 9.94 [06:16:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.62, 4.14, 3.58 [06:18:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.99, 3.59 [06:20:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.85, 4.72, 3.90 [06:20:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 3.96, 3.71 [06:23:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.38, 10.25, 10.02 [06:24:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.68, 3.60, 3.66 [06:24:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.78, 3.62, 3.53 [06:24:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.93, 3.79 [06:28:13] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.61, 3.94, 3.77 [06:29:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.85, 10.13, 10.03 [06:30:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.01, 3.67, 3.68 [06:32:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.63, 3.44, 3.61 [06:32:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.04, 4.32, 3.84 [06:34:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.08, 3.93, 3.89 [06:34:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.93, 3.75 [06:36:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.93, 10.41, 10.19 [06:38:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.18, 2.74, 3.27 [06:38:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.79, 2.87, 3.36 [06:40:13] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.75, 2.41, 3.22 [06:40:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.32, 10.01, 10.11 [06:40:46] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.54, 3.86, 3.23 [06:44:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.92, 3.37, 3.42 [06:44:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 3.64, 3.46 [06:46:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.09, 3.37, 3.39 [06:48:13] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.12, 2.78, 3.18 [06:48:46] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 2.01, 3.86, 3.74 [06:52:44] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.36, 2.91, 3.36 [06:55:24] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.65, 10.71, 10.16 [06:57:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.48, 10.82, 10.26 [06:58:42] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 5.24, 4.38, 2.21 [06:59:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 2.00, 1.92, 1.44 [06:59:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.89, 3.52 [07:01:18] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.48, 9.75, 10.01 [07:01:24] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.93, 2.39, 1.67 [07:01:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.66, 3.50, 3.42 [07:03:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.70, 1.73, 1.51 [07:03:43] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.21, 3.28, 3.34 [07:04:42] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.39, 1.70, 1.68 [07:05:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.37, 1.67, 1.52 [07:07:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.98, 10.46, 10.10 [07:08:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.80, 1.77, 1.72 [07:12:45] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.26, 1.58, 1.66 [07:12:50] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/dfc026ac8695 [07:12:52] [02miraheze/mw-config] 07Reception123 03dfc026a - add moderation email to managewiki settings [07:12:53] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [07:12:54] [url] Page not found · GitHub · GitHub | github.com [07:12:55] [02mw-config] 07Reception123 opened pull request 03#4827: add moderation email to managewiki settings - 13https://github.com/miraheze/mw-config/pull/4827 [07:12:55] [url] Page not found · GitHub · GitHub | github.com [07:13:28] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/dfc026ac8695...96866306ac02 [07:13:29] [url] Comparing dfc026ac8695...96866306ac02 · miraheze/mw-config · GitHub | github.com [07:13:30] [02miraheze/mw-config] 07Reception123 039686630 - Update LocalSettings.php [07:13:31] [02mw-config] 07Reception123 synchronize pull request 03#4827: add moderation email to managewiki settings - 13https://github.com/miraheze/mw-config/pull/4827 [07:13:31] [url] Page not found · GitHub · GitHub | github.com [07:14:09] miraheze/mw-config - Reception123 the build passed. [07:14:42] miraheze/mw-config - Reception123 the build passed. [07:16:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.43, 4.68, 3.56 [07:17:04] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.20, 4.65, 3.74 [07:17:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.67, 4.74, 3.59 [07:18:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.27, 9.79, 10.11 [07:19:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.94, 3.43 [07:23:21] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/1c6fc9639aee...adc2c973996b [07:23:22] [url] Comparing 1c6fc9639aee...adc2c973996b · miraheze/dns · GitHub | github.com [07:23:23] [02miraheze/dns] 07Reception123 03adc2c97 - add slymods.info zone [07:24:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.78, 3.79, 3.64 [07:24:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.16, 3.63, 3.71 [07:25:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.01, 3.36, 3.34 [07:30:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.66, 3.62, 3.55 [07:32:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.88, 3.14, 3.36 [07:34:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.42, 2.61, 3.25 [07:34:57] [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+3/-0/±3] 13https://github.com/miraheze/ssl/compare/41b07db27f53...74983e3c16ca [07:34:57] [url] Comparing 41b07db27f53...74983e3c16ca · miraheze/ssl · GitHub | github.com [07:34:58] [02miraheze/ssl] 07Reception123 035154c2e - add slymods.info cert [07:35:00] [02miraheze/ssl] 07Reception123 03dea6283 - add wiki.milkywayidle.com cert [07:35:01] [02miraheze/ssl] 07Reception123 0374983e3 - add wiki.auraxis.co cert [07:36:31] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/74983e3c16ca...2b9a473a2833 [07:36:31] [url] Comparing 74983e3c16ca...2b9a473a2833 · miraheze/ssl · GitHub | github.com [07:36:32] [02miraheze/ssl] 07Reception123 032b9a473 - add hastursnotebook.org cert [07:44:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.23, 3.57, 3.37 [07:46:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.51, 3.64, 3.43 [07:46:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.86, 3.46 [07:47:41] [02mw-config] 07Reception123 closed pull request 03#4821: T9555: set up CC BY-SA 3.0 for tlhmupaqwiki - 13https://github.com/miraheze/mw-config/pull/4821 [07:47:42] [url] Page not found · GitHub · GitHub | github.com [07:47:43] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/523aefbd1880...22e40d8877f3 [07:47:43] [url] Comparing 523aefbd1880...22e40d8877f3 · miraheze/mw-config · GitHub | github.com [07:47:44] [02miraheze/mw-config] 07ugochimobi 0322e40d8 - T9555: set up CC BY-SA 3.0 for tlhmupaqwiki (#4821) [07:48:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.19, 3.20, 3.30 [07:48:52] miraheze/mw-config - Reception123 the build passed. [07:50:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 3.23, 3.29 [07:53:07] !log [@mwtask111] starting deploy of {'config': True} to all [07:53:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:53:24] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 17s [07:53:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:54:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.73, 3.42, 3.36 [07:56:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 3.19, 3.29 [08:03:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.30, 3.11 [08:04:20] !log [@test101] starting deploy of {'config': True} to all [08:04:22] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [08:04:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:04:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:05:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.43, 2.93, 2.99 [08:12:24] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [08:14:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:18:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.70, 3.47, 3.31 [08:20:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.79, 3.14, 3.19 [08:25:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.77, 10.12, 9.04 [08:27:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.67, 8.93, 8.74 [08:28:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.47, 4.25, 3.42 [08:31:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 3.65, 3.19 [08:32:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.50, 3.56, 3.36 [08:34:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.31, 3.13, 3.22 [08:35:48] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [08:35:55] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:36:28] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 14.51, 7.09, 3.55 [08:36:47] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:36:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.21, 3.68, 3.34 [08:36:53] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:36:53] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:37:16] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:37:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.88, 2.87, 3.02 [08:37:35] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:38:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.80, 3.21, 3.20 [08:40:04] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:41:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.99, 11.44, 9.97 [08:42:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.85, 4.42, 3.69 [08:43:17] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:43:25] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:42] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 0.396 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [08:44:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 3.83, 3.56 [08:45:07] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 21 minutes ago with 0 failures [08:45:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.84, 3.42 [08:45:14] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [08:45:24] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:46:15] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [08:46:24] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.020 second response time [08:46:59] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 28504 bytes in 0.447 second response time [08:47:07] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.316 second response time [08:47:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.32, 3.34, 3.29 [08:47:29] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.026 second response time [08:47:43] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.346 second response time [08:48:20] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 7.54, 5.89, 3.36 [08:50:19] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 5.69, 5.88, 3.68 [08:50:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.43, 3.69, 3.55 [08:52:41] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.55, 3.51 [08:54:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [08:54:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.79, 3.92, 3.64 [08:55:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.40, 10.78, 11.09 [08:55:51] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [08:56:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:57:48] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [08:58:08] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.77, 1.58, 1.26 [08:58:39] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.92, 3.54, 3.57 [08:58:57] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.07, 8.89, 10.16 [09:04:05] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.08, 1.53, 1.39 [09:04:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.08, 3.02, 3.37 [09:08:01] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.15, 0.90, 3.71 [09:08:45] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw101 [09:10:01] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.16, 0.67, 3.28 [09:10:48] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [09:12:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.80, 3.37, 3.12 [09:12:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.49, 3.31 [09:14:36] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.12, 2.85, 2.96 [09:16:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.82, 3.79, 3.43 [09:18:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.21, 3.52, 3.24 [09:18:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.81, 3.30, 3.29 [09:20:27] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.38, 3.19, 3.13 [09:20:45] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw101 [09:20:48] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw101 [09:22:45] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [09:26:36] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw122 [09:26:47] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [09:28:36] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [09:34:46] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw121 [09:36:46] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [09:52:27] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw101 mw111 mw121 [09:52:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [09:53:45] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:55:38] PROBLEM - commons.ballmedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:56:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:58:56] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:01:41] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:03:03] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.121 second response time [10:03:13] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [10:03:41] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.620 second response time [10:07:04] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw102 [10:07:09] PROBLEM - www.opendatascot.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:07:24] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [10:14:52] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [10:17:18] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw101 [10:22:37] PROBLEM - cp21 Stunnel HTTP for mon111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:24:34] RECOVERY - cp21 Stunnel HTTP for mon111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 37919 bytes in 0.036 second response time [10:24:57] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw102 mw111 [10:25:07] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [10:25:27] RECOVERY - commons.ballmedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.ballmedia.org' will expire on Mon 10 Oct 2022 06:16:04 GMT +0000. [10:30:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.69, 3.06, 2.74 [10:30:49] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [10:32:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.97, 3.01, 2.76 [10:33:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [10:34:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.31, 9.52, 8.60 [10:35:56] RECOVERY - www.opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Thu 01 Sep 2022 00:46:11 GMT +0000. [10:36:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.54, 9.08, 8.55 [10:38:25] PROBLEM - cp21 Stunnel HTTP for mon111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:38:34] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw101 [10:38:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.93, 3.47, 2.92 [10:39:04] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw102 mw111 [10:40:23] RECOVERY - cp21 Stunnel HTTP for mon111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 37905 bytes in 2.052 second response time [10:40:46] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [10:42:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.65, 2.88, 2.84 [10:44:54] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw102 mw111 mw122 [10:44:57] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [10:45:19] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:48:56] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw101 [10:50:45] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [10:52:07] PROBLEM - wiki.scpsharp.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:54:38] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 4 backends are down. mw101 mw112 mw121 mw122 [10:54:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.80, 9.73, 7.92 [10:55:52] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:56:52]