[00:04:33] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.99, 1.87, 1.79 [00:06:28] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.99, 1.58, 1.69 [00:14:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.30, 1.89, 1.84 [00:29:29] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.46, 1.75, 1.61 [00:30:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.86, 9.89, 9.12 [00:31:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.53, 1.64, 1.58 [00:32:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.33, 10.10, 8.75 [00:32:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.93, 9.02, 8.89 [00:34:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.09, 9.73, 8.78 [00:35:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.77, 1.75, 1.64 [00:36:36] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.62, 3.01, 2.51 [00:37:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.22, 1.51, 1.56 [00:40:32] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.90, 3.06, 2.66 [01:12:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.49, 1.71, 2.00 [01:13:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.42, 10.20, 9.49 [01:18:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.06, 2.10, 2.03 [01:23:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.98, 10.00, 9.83 [01:24:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.37, 1.88, 1.98 [01:34:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.29, 1.39, 1.66 [01:38:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.61, 1.68 [01:44:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.57, 1.61, 1.66 [01:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.56, 1.99, 1.80 [01:51:56] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.40, 1.67, 1.43 [01:53:53] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.34, 1.64, 1.46 [01:54:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.61, 1.99, 1.91 [02:02:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.13, 2.00, 1.92 [02:03:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.56, 9.36, 9.08 [02:06:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 7.67, 3.03, 1.98 [02:07:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.20, 11.19, 9.88 [02:10:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.08, 10.09, 8.44 [02:11:01] PROBLEM - wiki.cyberfurs.org - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.4029810428619385 seconds [02:11:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.21, 10.00, 8.56 [02:12:10] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.16, 1.66, 1.68 [02:12:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.67, 9.94, 8.60 [02:13:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.47, 9.38, 8.51 [02:13:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.17, 1.69, 1.48 [02:14:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.36, 11.22, 10.50 [02:15:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.76, 1.53 [02:17:20] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.16, 2.03, 1.66 [02:17:55] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.96, 2.53, 2.03 [02:19:13] RECOVERY - fortis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fortis.wiki' will expire on Mon 22 Aug 2022 15:42:16 GMT +0000. [02:19:18] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.99, 1.69 [02:21:16] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.93, 2.09, 1.77 [02:22:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.15, 10.98, 10.50 [02:23:40] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.22, 1.78, 1.87 [02:27:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.02, 1.72, 1.75 [02:28:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.63, 11.44, 10.90 [02:29:06] RECOVERY - www.fortis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fortis.wiki' will expire on Mon 22 Aug 2022 15:42:16 GMT +0000. [02:30:11] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.89, 11.85, 11.11 [02:31:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.46, 1.64 [02:31:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.24, 1.45, 1.66 [02:32:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.89, 10.59, 10.02 [02:32:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.92, 11.43, 11.05 [02:37:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.53, 10.04, 9.97 [02:38:16] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.60, 2.75, 2.17 [02:40:40] RECOVERY - wiki.cyberfurs.org - reverse DNS on sslhost is OK: SSL OK - wiki.cyberfurs.org reverse DNS resolves to cp20.miraheze.org - CNAME OK [02:42:06] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.24, 1.93, 1.96 [02:42:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.22, 9.50, 8.89 [02:44:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.46, 9.29, 8.87 [02:47:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.05, 10.64, 10.19 [02:47:50] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.47, 1.93, 1.90 [02:49:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.86, 9.98, 10.01 [02:49:45] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.48, 1.75, 1.84 [02:51:11] PROBLEM - fortis.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address fortis.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:52:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.50, 11.13, 10.82 [02:55:30] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.33, 1.79, 1.80 [02:57:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.41, 1.55, 1.71 [02:58:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.08, 11.60, 11.24 [03:00:14] PROBLEM - www.fortis.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.fortis.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:01:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.33, 1.51, 1.66 [03:01:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.61, 10.29, 9.57 [03:03:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.36, 9.77, 9.45 [03:04:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.09, 11.64, 11.37 [03:04:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.74, 1.99 [03:05:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.54, 10.16, 9.89 [03:07:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.96, 9.77, 9.79 [03:10:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.00, 11.89, 11.68 [03:16:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.15, 1.58, 1.70 [03:19:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.17, 2.08, 1.61 [03:21:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.28, 1.76, 1.55 [03:24:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.58, 11.10, 11.09 [03:25:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.40, 1.60, 1.53 [03:26:43] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.13, 10.01, 9.03 [03:28:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.53, 1.92, 1.96 [03:30:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.03, 11.44, 11.36 [03:30:32] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.46, 1.79, 1.02 [03:30:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.15, 10.24, 9.36 [03:32:32] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.35, 1.26, 0.92 [03:32:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.08, 9.29, 9.11 [03:42:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.20, 8.53, 10.02 [03:42:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.26, 1.91, 1.89 [03:42:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.37, 9.83, 9.23 [03:44:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.16, 9.40, 9.13 [03:45:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [03:50:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [03:53:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.68, 3.13, 1.98 [03:55:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.50, 3.43, 3.88 [03:57:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.53, 3.77, 3.95 [03:59:32] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.39, 1.99, 1.83 [04:03:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.70, 3.66, 3.89 [04:05:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 4.05, 4.01 [04:05:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.10, 1.56, 1.69 [04:28:32] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.31, 1.71, 0.98 [04:30:32] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.59, 1.29, 0.91 [04:36:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.53, 10.30, 9.69 [04:37:40] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.30, 1.73, 1.54 [04:38:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.97, 9.67, 9.55 [04:39:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.54, 1.49 [04:44:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.32, 1.65, 1.96 [04:50:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.63, 2.22, 2.10 [04:54:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.00, 11.34, 10.33 [04:56:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.10, 10.47, 10.14 [04:58:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.50, 1.80, 1.97 [05:00:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.55, 9.59, 9.91 [05:02:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.18, 2.29, 2.10 [05:06:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.78, 11.34, 10.47 [05:08:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.28, 10.74, 10.35 [05:10:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.31, 10.13, 10.19 [05:21:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.61, 1.74, 1.49 [05:22:17] PROBLEM - storytime.jdstroy.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'storytime.jdstroy.cf' expires in 15 day(s) (Sun 31 Jul 2022 04:57:19 GMT +0000). [05:22:53] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/884a3e667dbd...1a208148635c [05:22:54] [url] Comparing 884a3e667dbd...1a208148635c · miraheze/ssl · GitHub | github.com [05:22:55] [02miraheze/ssl] 07MirahezeSSLBot 031a20814 - Bot: Update SSL cert for storytime.jdstroy.cf [05:23:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.38, 1.59, 1.46 [05:28:12] PROBLEM - wiki.sketch.town - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.sketch.town' expires in 15 day(s) (Sun 31 Jul 2022 05:25:53 GMT +0000). [05:29:06] PROBLEM - dnd.bellinrattin.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dnd.bellinrattin.it' expires in 15 day(s) (Sun 31 Jul 2022 05:03:24 GMT +0000). [05:32:54] PROBLEM - arrieretoile.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'arrieretoile.fr' expires in 15 day(s) (Sun 31 Jul 2022 05:21:21 GMT +0000). [05:34:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1a208148635c...d3b9231c77a1 [05:34:23] [url] Comparing 1a208148635c...d3b9231c77a1 · miraheze/ssl · GitHub | github.com [05:34:24] [02miraheze/ssl] 07MirahezeSSLBot 03d3b9231 - Bot: Update SSL cert for wiki.sketch.town [05:37:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d3b9231c77a1...95feb205799a [05:37:36] [url] Comparing d3b9231c77a1...95feb205799a · miraheze/ssl · GitHub | github.com [05:37:37] [02miraheze/ssl] 07MirahezeSSLBot 0395feb20 - Bot: Update SSL cert for arrieretoile.fr [05:42:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.66, 1.39 [05:44:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.66, 1.73, 1.45 [05:46:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.94, 1.41, 1.36 [05:57:25] RECOVERY - wiki.sketch.town - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sketch.town' will expire on Thu 13 Oct 2022 04:34:16 GMT +0000. [06:02:17] RECOVERY - arrieretoile.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'arrieretoile.fr' will expire on Thu 13 Oct 2022 04:37:30 GMT +0000. [06:06:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.27, 9.87, 9.31 [06:07:03] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.08, 1.69, 1.48 [06:09:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.02, 1.42, 1.40 [06:14:11] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.35, 10.09, 9.74 [06:18:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.33, 10.33, 9.94 [06:20:51] RECOVERY - storytime.jdstroy.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'storytime.jdstroy.cf' will expire on Thu 13 Oct 2022 04:22:47 GMT +0000. [06:22:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.95, 10.98, 10.24 [06:24:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.71, 11.28, 10.45 [06:25:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.51, 3.97 [06:27:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.79, 4.02, 4.10 [06:32:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.11, 11.13, 10.73 [06:33:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.46, 3.40, 3.84 [06:34:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.32, 11.00, 10.72 [06:35:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.96, 3.76, 3.90 [06:36:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.98, 11.57, 10.96 [06:38:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.64, 11.02, 10.83 [06:39:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.50, 3.77 [06:41:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 3.89, 3.87 [06:43:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.65, 3.78 [06:45:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.87, 3.85 [06:46:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.07, 9.38, 10.17 [07:03:29] [02puppet] 07RhinosF1 closed pull request 03#2436: db: don't allow access to testglobal from non test servers - 13https://github.com/miraheze/puppet/pull/2436 [07:03:29] [url] Page not found · GitHub · GitHub | github.com [07:05:29] [discord] o [07:49:11] RECOVERY - fortis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fortis.wiki' will expire on Mon 22 Aug 2022 15:42:16 GMT +0000. [07:57:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.86, 3.79, 3.99 [07:59:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.44, 4.03, 4.05 [08:16:55] PROBLEM - wiki.kourouklides.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.kourouklides.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:21:08] PROBLEM - fortis.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address fortis.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:18:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.79, 3.99 [09:20:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.82, 3.88, 4.01 [09:22:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.63, 3.90 [09:26:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.65, 4.27, 4.06 [10:14:54] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.72, 1.61, 1.24 [10:16:50] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.29, 1.50, 1.25 [11:25:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.74, 10.35, 9.27 [11:27:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.05, 9.80, 9.19 [11:36:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.63, 10.21, 9.58 [11:38:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.65, 10.05, 9.60 [11:41:45] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.98, 1.52, 1.14 [11:43:41] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.85, 1.26, 1.09 [11:56:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.25, 1.66, 1.94 [12:16:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.76, 1.75 [12:30:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.15, 10.09, 9.54 [12:32:18] Reception123: RhinosF1: I created https://github.com/miraheze/puppet/pull/2699 as a solution to the SSL issue. If merged when you revoke a key it will now remove it from git as well. It wouldn’t cover non-le, but since they are so rare and there currently is only one non-le cert that isn’t the mh.org one, I’d be content to just poke you or Infra about it in the rare case it needs doing. [12:32:18] [url] ssl: extend revoke option to also remove key from git by MacFan4000 · Pull Request #2699 · miraheze/puppet · GitHub | github.com [12:40:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.12, 11.28, 10.37 [12:42:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.54, 11.23, 10.48 [12:44:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.12, 10.06, 10.14 [12:53:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.69, 10.83, 10.49 [12:55:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.22, 11.32, 10.71 [12:59:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.41, 11.30, 10.85 [13:15:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.91, 8.99, 9.87 [13:20:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.03, 10.32, 10.17 [13:22:48] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.65, 10.89, 10.38 [13:26:32] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.97, 1.98, 1.54 [13:26:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.60, 10.78, 10.51 [13:30:32] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.09, 1.52, 1.45 [13:32:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.34, 1.64, 1.97 [13:44:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.34, 9.07, 9.90 [13:44:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.24, 1.44, 1.68 [13:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.58, 2.07, 1.88 [13:56:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.99, 1.96 [14:05:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.25, 9.46, 9.07 [14:06:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.04, 1.83, 1.87 [14:07:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.48, 9.00, 8.94 [14:08:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.97, 1.81, 1.86 [14:11:42] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [14:16:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.21, 2.11, 1.98 [14:18:39] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.78, 3.05, 2.85 [14:20:35] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.44, 3.43, 3.00 [14:22:31] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.46, 3.20, 2.96 [14:26:23] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.97, 3.21, 3.03 [14:36:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.96, 10.68, 10.14 [14:38:51] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.56, 9.94, 9.94 [14:39:41] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:55:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.38, 10.64, 10.21 [15:01:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.00, 9.87, 10.05 [15:01:28] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.80, 1.67, 1.33 [15:03:24] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.29, 1.62, 1.36 [15:05:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.28, 11.55, 10.70 [15:05:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.79, 10.80, 9.83 [15:06:10] [discord] SRE, mh is loading abnormally slowly [15:06:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.97, 11.51, 10.78 [15:07:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.25, 10.16, 9.72 [15:07:32] i'm breaking wmf prod today [15:10:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.29, 1.66, 1.95 [15:14:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.21, 9.65, 10.20 [15:18:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.45, 2.01, 1.97 [15:20:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.06, 10.29, 10.30 [15:26:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.26, 1.72, 1.90 [15:30:55] [02ssl] 07Pix1234 opened pull request 03#568: Remove pj-masks-info.cf as it no longer points - 13https://github.com/miraheze/ssl/pull/568 [15:30:56] [url] Page not found · GitHub · GitHub | github.com [15:32:02] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.58, 3.72, 3.08 [15:32:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.29, 1.84, 1.87 [15:34:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.71, 9.68, 10.10 [15:35:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.49, 10.04, 9.61 [15:36:02] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.88, 3.48, 3.14 [15:36:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.75, 1.81 [15:37:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.37, 9.51, 9.46 [15:38:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [15:40:02] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.88, 3.39, 3.19 [15:41:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.51, 10.02, 9.40 [15:43:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.41, 9.97, 9.47 [15:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.91, 2.27, 1.98 [15:51:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.09, 10.54, 10.27 [15:53:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [15:55:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.76, 10.13, 10.16 [15:56:53] MacFan4000: thanks. I'd have John review that as I don't want to mess things/his work up [15:57:29] [discord] I also opened the above PR as the user on that wiki also needs their wiki switch back to subdomain [15:57:34] [discord] If someone could do thr [15:57:37] [discord] That* [16:15:43] RECOVERY - www.fortis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fortis.wiki' will expire on Mon 22 Aug 2022 15:42:16 GMT +0000. [16:16:23] Reception123: it doesn’t even touch that dir [16:16:39] It just uses the clone in /home to remove the key [16:17:51] I see. Either way I've got no access for now so will check later [16:19:13] RECOVERY - fortis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fortis.wiki' will expire on Mon 22 Aug 2022 15:42:16 GMT +0000. [16:23:40] PROBLEM - wiki.zergmafia.top - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zergmafia.top All nameservers failed to answer the query. [16:30:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.97, 1.70, 1.46 [16:32:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.06, 1.41, 1.37 [16:34:05] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.57, 2.04, 1.33 [16:36:00] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.67, 1.50, 1.21 [16:36:34] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.42, 2.13, 1.29 [16:38:28] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.34, 1.73, 1.24 [16:39:15] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:39:23] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:39:57] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:22] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.57, 1.31, 1.14 [16:40:43] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:44] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:41:26] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:41:56] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:41:58] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:02] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:16] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [16:43:29] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28480 bytes in 5.661 second response time [16:43:52] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:43:54] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:43:56] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.386 second response time [16:44:01] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:08] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:10] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.44, 1.71, 1.97 [16:44:16] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.397 second response time [16:44:49] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.317 second response time [16:44:54] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.172 second response time [16:45:48] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.086 second response time [16:45:56] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.461 second response time [16:46:00] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.021 second response time [16:46:05] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.204 second response time [16:46:06] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.672 second response time [16:46:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.00, 2.13, 2.09 [16:50:15] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.01, 2.71, 1.86 [16:51:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.98, 10.21, 9.52 [16:52:46] RECOVERY - wiki.zergmafia.top - reverse DNS on sslhost is OK: SSL OK - wiki.zergmafia.top reverse DNS resolves to cp20.miraheze.org - CNAME OK [16:53:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.29, 9.93, 9.51 [16:55:39] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 8.086 second response time [16:56:02] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.72, 1.90, 1.74 [16:56:03] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.319 second response time [16:57:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.83, 10.77, 9.96 [16:57:58] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 9.88, 4.91, 2.84 [16:58:52] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.122 second response time [16:58:55] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.749 second response time [16:59:07] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.014 second response time [17:00:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.34, 1.56, 1.93 [17:02:29] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:33] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.969 second response time [17:05:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [17:08:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.31, 1.34, 1.66 [17:13:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.91, 10.52, 9.33 [17:14:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.62, 10.25, 9.53 [17:15:20] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.11, 1.44, 1.87 [17:16:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.52, 11.07, 9.91 [17:16:10] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.74, 3.26, 2.75 [17:17:15] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.40, 1.07, 1.69 [17:17:43] PROBLEM - www.fortis.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.fortis.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [17:18:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.02, 10.91, 9.99 [17:18:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.90, 2.10, 1.82 [17:18:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.65, 9.57, 10.06 [17:19:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.76, 9.53, 9.30 [17:19:56] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.75, 9.69, 9.65 [17:20:02] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.15, 3.32, 2.90 [17:21:11] PROBLEM - fortis.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address fortis.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [17:23:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.19, 1.93, 1.29 [17:24:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.32, 11.39, 10.67 [17:24:33] PROBLEM - www.fraterniterebellion.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.fraterniterebellion.com could not be found [17:25:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.82, 1.49, 1.21 [17:26:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.81, 11.52, 10.80 [17:26:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.98, 1.92 [17:26:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.77, 1.69, 1.63 [17:28:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.08, 10.53, 9.98 [17:28:51] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.21, 1.43, 1.54 [17:31:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.30, 9.86, 9.48 [17:32:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.78, 9.86, 9.85 [17:33:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.71, 9.02, 9.22 [17:36:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.18, 11.36, 10.77 [17:38:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.78, 10.67, 10.59 [17:40:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.89, 1.88 [17:50:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.77, 11.31, 10.81 [17:52:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.40, 10.96, 10.75 [17:59:43] PROBLEM - wiki.fraterniterebellion.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fraterniterebellion.com could not be found [18:01:58] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.47, 2.55, 1.62 [18:05:58] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 1.52, 2.30, 1.76 [18:11:31] PROBLEM - www.replication-ops.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.replication-ops.com could not be found [18:14:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.87, 10.42, 9.81 [18:16:05] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.67, 9.75, 9.64 [18:20:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.80, 11.71, 11.14 [18:23:38] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [18:23:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.60, 9.74, 9.55 [18:24:28] PROBLEM - www.fortis.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.fortis.wiki All nameservers failed to answer the query. [18:25:36] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.43, 9.35, 9.44 [18:26:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.04, 11.76, 11.39 [18:28:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.57, 12.07, 11.54 [18:29:57] PROBLEM - wiki.fraterniterebellion.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.fraterniterebellion.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:32:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.54, 11.74, 11.54 [18:34:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.43, 11.95, 11.64 [18:42:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.61, 11.90, 11.86 [18:44:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.04, 11.82, 11.83 [18:45:56] PROBLEM - www.fraterniterebellion.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.fraterniterebellion.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:46:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.77, 11.71, 11.78 [18:49:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.14, 10.02, 9.67 [18:51:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.99, 9.70, 9.58 [18:51:37] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:52:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.17, 11.45, 11.59 [18:54:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.56, 11.63, 11.65 [18:54:27] PROBLEM - www.fortis.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.fortis.wiki could not be found [18:56:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.04, 12.25, 11.86 [19:01:43] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.41, 10.52, 9.28 [19:02:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.87, 11.41, 11.69 [19:02:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.38, 1.98, 1.56 [19:03:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.98, 10.43, 9.42 [19:04:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.92, 1.65, 1.49 [19:05:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.90, 10.80, 9.91 [19:05:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.87, 9.19, 9.07 [19:08:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.49, 11.57, 11.63 [19:09:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.97, 10.01, 9.79 [19:10:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.51, 10.73, 11.32 [19:15:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [19:19:54] RECOVERY - fortis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fortis.wiki' will expire on Mon 22 Aug 2022 15:42:16 GMT +0000. [19:32:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.90, 9.18, 10.02 [19:36:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.16, 10.82, 10.49 [19:38:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.33, 10.48, 10.38 [19:50:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.81, 9.74, 10.11 [19:51:32] PROBLEM - fortis.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address fortis.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:54:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.94, 10.28, 10.25 [19:55:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.43, 9.92, 9.28 [19:56:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.06, 10.97, 10.51 [19:58:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.84, 10.95, 10.55 [19:59:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.89, 9.13, 9.12 [20:12:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.66, 9.63, 10.07 [20:18:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.86, 10.62, 10.32 [20:28:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.44, 9.67, 10.11 [20:32:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.91, 10.24, 10.25 [20:33:56] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.78, 1.58, 1.41 [20:34:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.03, 11.31, 10.64 [20:35:53] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.21, 1.50, 1.41 [20:35:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.20, 10.23, 9.67 [20:37:49]