[00:07:01] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 6799 MB (17% inode=96%); [00:11:06] RECOVERY - cp30 Disk Space on cp30 is OK: DISK OK - free space: / 6954 MB (18% inode=96%); [00:37:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.98, 2.92, 3.80 [00:37:27] PROBLEM - wiki.ameristraliagov.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ameristraliagov.com could not be found [00:43:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.43, 3.42, 3.76 [00:57:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.73, 10.18, 11.89 [01:01:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.92, 3.03, 3.93 [01:07:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.44, 2.22, 3.29 [01:07:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.62, 2.94, 3.99 [01:07:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.57, 8.11, 9.94 [01:09:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.87, 3.84, 4.15 [01:12:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.15, 9.93, 11.86 [01:16:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.60, 1.74, 1.99 [01:16:58] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 10.19, 6.72, 3.91 [01:17:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.29, 3.10, 3.74 [01:18:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.66, 1.92, 2.05 [01:19:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.55, 3.83 [01:20:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.09, 10.77, 11.45 [01:21:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.71, 3.22, 3.67 [01:22:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.28, 10.45, 11.24 [01:26:58] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 4.66, 7.29, 6.07 [01:28:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.36, 12.02, 11.67 [01:28:58] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 3.05, 5.90, 5.71 [01:29:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.93, 2.77, 3.31 [01:29:36] PROBLEM - sbs.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address sbs.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:30:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.19, 1.55, 1.91 [01:32:37] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.06, 2.09, 2.04 [01:32:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.29, 11.84, 10.42 [01:33:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.92, 3.96, 3.72 [01:35:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.98, 3.75 [01:36:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.69, 11.73, 10.70 [01:37:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.76, 3.67, 3.67 [01:38:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.06, 3.91, 3.40 [01:40:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.67, 3.26, 3.22 [01:42:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.71, 13.03, 11.47 [01:43:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.84, 4.43, 3.96 [01:43:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.68, 4.36, 3.69 [01:45:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.86, 3.59 [01:47:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 4.18, 3.74 [01:52:58] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 7.71, 5.99, 4.64 [01:54:58] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 8.08, 6.75, 5.09 [01:56:58] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 5.80, 6.39, 5.17 [01:58:31] PROBLEM - gluster102 NTP time on gluster102 is WARNING: NTP WARNING: Offset 0.1070148349 secs [02:03:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.18, 3.95, 4.00 [02:04:37] RECOVERY - gluster102 NTP time on gluster102 is OK: NTP OK: Offset 0.09434843063 secs [02:09:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.53, 10.21, 11.57 [02:11:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.91, 3.62, 3.70 [02:13:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 3.53, 3.64 [02:19:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.88, 3.75 [02:21:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.97, 3.15, 3.49 [02:25:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.69, 2.66, 3.23 [02:25:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.00, 11.10, 10.92 [02:27:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.02, 3.06, 3.89 [02:29:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.73, 4.01, 3.68 [02:31:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.82, 3.66 [02:31:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.39, 11.57, 11.26 [02:37:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.93, 2.78, 3.25 [02:37:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.74, 2.41, 3.25 [02:39:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.51, 9.36, 10.17 [02:43:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.60, 10.72, 10.55 [02:53:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.10, 9.77, 10.20 [03:00:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.43, 1.68, 1.95 [03:01:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.59, 11.66, 10.67 [03:02:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.45, 6.01, 4.11 [03:02:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.13, 1.83, 1.97 [03:03:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.94, 3.22 [03:03:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.75, 10.69, 10.43 [03:05:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.29, 3.33, 3.08 [03:05:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.43, 11.00, 10.55 [03:08:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.26, 3.52, 3.68 [03:10:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.37, 4.30, 3.96 [03:11:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.64, 4.07, 3.45 [03:11:41] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [03:13:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.69, 3.38 [03:15:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.91, 3.35, 3.29 [03:15:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.21, 3.33, 3.74 [03:19:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.31, 2.18, 3.17 [03:23:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.47, 5.21, 4.25 [03:25:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.44, 5.03, 3.89 [03:28:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.26, 1.67, 1.98 [03:30:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.04, 1.78, 1.98 [03:31:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.52, 3.60, 3.62 [03:32:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.65, 1.76, 1.95 [03:34:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.20, 1.87, 1.96 [03:35:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.21, 2.88, 3.32 [03:36:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.97, 2.00, 2.00 [03:38:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.76, 2.23, 2.08 [03:41:42] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.405889511108398 seconds [03:42:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.55, 1.82, 1.94 [03:44:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.45, 3.53, 3.46 [03:44:18] PROBLEM - gluster102 NTP time on gluster102 is WARNING: NTP WARNING: Offset -0.1062653661 secs [03:46:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.74, 2.93, 3.25 [03:46:21] RECOVERY - gluster102 NTP time on gluster102 is OK: NTP OK: Offset -0.0924102664 secs [03:47:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.62, 9.44, 11.79 [03:50:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.10, 1.89, 1.89 [03:52:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.56, 1.78, 1.85 [03:58:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.42, 1.77, 1.80 [04:00:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.66, 1.64, 1.74 [04:02:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.76, 1.35, 1.62 [04:02:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.77, 9.34, 11.74 [04:04:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.20, 10.33, 11.79 [04:05:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.15, 3.10, 3.83 [04:05:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.32, 8.33, 9.78 [04:06:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.51, 2.16, 1.89 [04:06:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.07, 10.13, 11.51 [04:09:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.08, 2.21, 3.31 [04:10:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.03, 1.74, 1.80 [04:11:37] RECOVERY - wikipedia.yogfront.ooo - reverse DNS on sslhost is OK: SSL OK - wikipedia.yogfront.ooo reverse DNS resolves to cp31.miraheze.org - CNAME OK [04:12:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.67, 2.01, 1.89 [04:16:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.69, 11.19, 11.24 [04:20:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.31, 11.14, 11.23 [04:22:56] PROBLEM - wiki.otir.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.otir.nl' expires in 15 day(s) (Fri 26 Aug 2022 04:20:05 GMT +0000). [04:26:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fcbd18226d2b...ac752ff6bcf5 [04:26:33] [url] Comparing fcbd18226d2b...ac752ff6bcf5 · miraheze/ssl · GitHub | github.com [04:26:34] [02miraheze/ssl] 07MirahezeSSLBot 03ac752ff - Bot: Update SSL cert for wiki.otir.nl [04:26:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.55, 12.33, 11.63 [04:28:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.06, 10.26, 9.69 [04:30:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.44, 2.96 [04:30:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.90, 10.06, 9.68 [04:31:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.88, 3.94, 3.48 [04:31:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.38, 3.04, 2.87 [04:32:35] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [04:34:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.22, 1.56, 1.93 [04:35:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.48, 3.35 [04:37:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 3.19, 3.27 [04:37:24] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [04:38:08] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [04:38:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.00, 1.92, 1.93 [04:39:22] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [04:40:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.74, 10.77, 11.62 [04:41:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.31, 5.07, 4.04 [04:42:17] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [04:42:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.64, 12.05, 11.99 [04:43:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.10, 10.40, 9.58 [04:44:15] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [04:44:26] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ac752ff6bcf5...9fbc02524c2f [04:44:26] [url] Comparing ac752ff6bcf5...9fbc02524c2f · miraheze/ssl · GitHub | github.com [04:44:27] [02miraheze/ssl] 07paladox 039fbc025 - Update wiki.otir.nl.crt [04:44:33] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [04:44:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.94, 3.64, 3.17 [04:44:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.83, 1.82, 1.96 [04:46:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.74, 2.28, 2.12 [04:47:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.76, 10.12, 9.68 [04:49:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.20, 3.54, 3.80 [04:50:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.82, 4.35, 3.59 [04:52:52] RECOVERY - wiki.otir.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.otir.nl' will expire on Tue 08 Nov 2022 03:41:52 GMT +0000. [04:53:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.97, 4.05, 3.96 [04:53:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.34, 11.07, 10.20 [04:55:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.58, 12.54, 10.85 [04:57:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.88, 11.40, 10.64 [05:00:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.66, 3.60, 3.69 [05:00:33] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [05:01:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.61, 12.08, 10.99 [05:02:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.38, 4.16, 3.90 [05:04:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.81, 3.72, 3.78 [05:05:22] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:05:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.86, 11.67, 11.19 [05:06:16] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [05:07:21] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:07:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.23, 12.36, 11.49 [05:08:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.92, 1.46, 1.96 [05:09:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.66, 10.93, 11.07 [05:10:14] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [05:11:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.09, 12.20, 11.54 [05:11:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.62, 2.87, 3.37 [05:12:27] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:12:43] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [05:13:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.84, 3.44, 3.95 [05:13:50] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query yogfront.ooo. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:13:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.29, 10.05, 10.84 [05:14:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.86, 1.16, 1.68 [05:17:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.90, 4.91, 4.31 [05:18:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.69, 1.57, 1.75 [05:19:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.10, 3.64, 3.47 [05:21:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.78, 3.18, 3.31 [05:23:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.15, 9.19, 10.00 [05:24:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.33, 1.56, 1.70 [05:25:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.63, 3.92 [05:29:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.24, 3.65 [05:30:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.87, 11.00, 11.94 [05:31:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.05, 2.87, 3.47 [05:33:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.54, 2.64, 3.30 [05:34:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.50, 11.85, 12.00 [05:37:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.01, 3.42, 3.48 [05:41:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.11, 2.89, 3.27 [05:41:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.37, 9.72, 9.27 [05:43:31] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [05:44:44] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.71, 1.68, 1.63 [05:45:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.12, 9.07, 9.12 [05:46:39] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.18, 1.50, 1.57 [05:46:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.76, 11.12, 11.85 [05:47:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.22, 3.57, 3.56 [05:50:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.34, 2.99, 2.16 [05:51:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.37, 2.72, 3.24 [05:56:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.51, 1.90, 1.92 [06:02:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.37, 1.41, 1.67 [06:02:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.63, 11.83, 11.13 [06:06:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.79, 11.83, 11.36 [06:08:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.57, 1.75, 1.98 [06:08:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.00, 12.12, 11.51 [06:12:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.73, 2.28, 2.13 [06:12:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.39, 11.29, 11.43 [06:13:13] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.405799150466919 seconds [06:13:38] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.24, 1.68, 1.08 [06:14:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.76, 11.83, 11.57 [06:15:38] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.75, 1.71, 1.16 [06:17:01] Join #miraheze-tech [06:19:13] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.50, 2.04, 1.78 [06:19:38] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.22, 1.98, 1.39 [06:20:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.41, 10.81, 11.37 [06:21:38] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.78, 1.90, 1.44 [06:24:56] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.12, 1.79, 1.82 [06:26:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.98, 11.14, 11.26 [06:27:38] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.13, 1.89, 1.56 [06:28:44] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.03, 1.91, 1.86 [06:29:38] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.55, 1.76, 1.55 [06:30:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.00, 10.87, 11.19 [06:31:38] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.18, 1.57, 1.51 [06:34:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.25, 1.64, 1.76 [06:36:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a1fb7272e7f5 [06:36:15] [02miraheze/puppet] 07paladox 03a1fb727 - db: add ability to set replication id [06:36:16] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [06:36:16] [url] Page not found · GitHub · GitHub | github.com [06:36:18] [02puppet] 07paladox opened pull request 03#2771: db: add ability to set replication id - 13https://github.com/miraheze/puppet/pull/2771 [06:36:18] [url] Page not found · GitHub · GitHub | github.com [06:37:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a1fb7272e7f5...934c9989b58e [06:37:30] [url] Comparing a1fb7272e7f5...934c9989b58e · miraheze/puppet · GitHub | github.com [06:37:31] [02miraheze/puppet] 07paladox 03934c998 - Update config.pp [06:37:32] [02puppet] 07paladox synchronize pull request 03#2771: db: add ability to set replication id - 13https://github.com/miraheze/puppet/pull/2771 [06:37:33] [url] Page not found · GitHub · GitHub | github.com [06:38:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/934c9989b58e...5b10104c748c [06:38:31] [url] Comparing 934c9989b58e...5b10104c748c · miraheze/puppet · GitHub | github.com [06:38:32] [02miraheze/puppet] 07paladox 035b10104 - Update db141.yaml [06:38:34] [02puppet] 07paladox synchronize pull request 03#2771: db: add ability to set replication id - 13https://github.com/miraheze/puppet/pull/2771 [06:38:34] [url] Page not found · GitHub · GitHub | github.com [06:38:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.91, 11.18, 11.05 [06:40:20] [02puppet] 07paladox closed pull request 03#2771: db: add ability to set replication id - 13https://github.com/miraheze/puppet/pull/2771 [06:40:21] [url] Page not found · GitHub · GitHub | github.com [06:40:22] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [06:40:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/c58c2306d426...34a76df9da99 [06:40:24] [url] Comparing c58c2306d426...34a76df9da99 · miraheze/puppet · GitHub | github.com [06:40:25] [02miraheze/puppet] 07paladox 0334a76df - db: add ability to set replication id (#2771) [06:40:26] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [06:40:26] ... [06:42:20] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/9c421869f364 [06:42:21] [02miraheze/mw-config] 07paladox 039c42186 - Replace db111 with db141 [06:42:23] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/mw-config [06:42:23] [url] Page not found · GitHub · GitHub | github.com [06:42:24] [02mw-config] 07paladox opened pull request 03#4868: Replace db111 with db141 - 13https://github.com/miraheze/mw-config/pull/4868 [06:42:25] [url] Page not found · GitHub · GitHub | github.com [06:42:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.77, 10.89, 10.97 [06:43:31] miraheze/mw-config - paladox the build passed. [06:44:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.01, 1.45, 1.64 [06:46:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.07, 8.64, 10.01 [06:47:19] paladox: are any wikis using c6? [06:47:35] no [06:48:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.39, 3.58, 2.96 [06:48:31] paladox: ok, that PR is fine then. I was thinking of other ways if they were. [06:50:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.13, 1.92, 1.76 [06:50:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.59, 4.06, 2.96 [06:50:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.59, 12.58, 11.36 [06:51:15] ok [06:52:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.64, 3.44, 2.86 [06:53:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.97, 3.96, 3.45 [06:54:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.36, 1.91, 1.81 [06:54:54] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:54:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.48, 11.46, 11.20 [06:55:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.70, 3.14, 3.21 [06:56:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.68, 11.99, 11.43 [06:58:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 3.92, 3.27 [06:58:37] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.53, 2.12, 1.92 [06:59:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.19, 3.78, 3.50 [07:00:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.33, 3.40, 3.16 [07:00:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.33, 1.53, 1.90 [07:00:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.84, 1.99, 1.89 [07:01:30] paladox: why are no wikis using c6? If it was pooled into CW then I'd think some new wikis would, no? [07:02:11] Also, that PR does not update LS to remove c6 from CW. [07:02:46] Because I moved them to db131 to be able to move db111 to db141 [07:02:56] PROBLEM - db131 Puppet on db131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:03:32] paladox: oh, so you moved all c6 wikis to c5? [07:04:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/34a76df9da99...fd383eac09d9 [07:04:03] [url] Comparing 34a76df9da99...fd383eac09d9 · miraheze/puppet · GitHub | github.com [07:04:04] [02miraheze/puppet] 07paladox 03fd383ea - Fix [07:04:15] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:04:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.40, 1.52, 1.70 [07:05:35] And wikis are still being (as of now created on db141 also), if wikis are created there is a chance anyway, so you may need to still move a couple I guess? [07:06:18] If it's replicating, that's definitely not what we want [07:06:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fd383eac09d9...c2f013a3142a [07:06:54] [url] Comparing fd383eac09d9...c2f013a3142a · miraheze/puppet · GitHub | github.com [07:06:55] [02miraheze/puppet] 07paladox 03c2f013a - Fix [07:06:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.43, 11.49, 11.70 [07:07:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.86, 2.98, 3.29 [07:10:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.98, 1.36, 1.68 [07:10:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.67, 11.76, 11.65 [07:12:34] RECOVERY - wikipedia.yogfront.ooo - reverse DNS on sslhost is OK: SSL OK - wikipedia.yogfront.ooo reverse DNS resolves to cp31.miraheze.org - CNAME OK [07:12:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 3.85, 3.47 [07:12:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.12, 10.80, 11.32 [07:14:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.97, 1.90, 1.74 [07:14:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.33, 11.16, 11.37 [07:16:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.28, 2.08, 1.85 [07:16:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.10, 2.10, 1.82 [07:16:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.21, 10.61, 11.13 [07:17:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.59, 3.32 [07:19:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.16, 3.00, 3.14 [07:20:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.25, 3.56, 3.67 [07:20:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.68, 1.95, 1.83 [07:22:54] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:24:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.04, 9.07, 10.16 [07:26:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.53, 2.71, 3.29 [07:26:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.87, 1.50, 1.69 [07:30:56] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:32:15] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:41:13] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.12, 2.16, 1.87 [07:43:07] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.57, 1.95, 1.83 [07:44:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.86, 1.56, 1.96 [07:48:50] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.38, 1.93, 1.82 [07:50:44] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.90, 1.92, 1.83 [07:52:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.64, 1.85, 1.85 [07:54:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.36, 1.58, 1.75 [07:58:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.51, 1.55, 1.70 [07:58:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.82, 1.35, 1.60 [08:23:48] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.73, 1.83, 1.71 [08:25:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.29, 1.61, 1.64 [08:44:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [08:46:36] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:51:30] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.05, 2.03, 1.63 [08:51:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.20, 1.74, 1.56 [08:53:24] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.37, 1.82, 1.61 [08:55:18] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 7.00, 3.18, 2.08 [09:01:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.06, 1.90, 1.86 [09:04:50] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.60, 1.98, 1.90 [09:06:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.64, 3.52, 2.93 [09:06:43] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.14, 1.45, 1.67 [09:06:44] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.21, 2.10, 1.95 [09:08:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.05, 2.97, 2.80 [09:18:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.65, 2.13, 1.82 [09:26:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.97, 2.00, 1.89 [09:28:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.03, 1.99, 1.89 [09:30:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.57, 1.82, 1.84 [09:34:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.64, 1.95, 1.87 [09:36:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.27, 1.63, 1.96 [09:36:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.68, 1.85, 1.85 [09:39:55] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [09:44:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.46, 1.49, 1.67 [09:50:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.54, 1.88, 1.75 [09:52:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.22, 1.89, 1.87 [09:54:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.91, 1.76, 1.82 [09:56:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.83, 2.92 [09:57:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.72, 10.35, 9.29 [09:58:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 3.39, 2.88 [09:59:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.91, 9.74, 9.19 [10:00:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.84, 1.92, 1.83 [10:01:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.64, 4.25, 3.36 [10:02:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.30, 1.67, 1.75 [10:03:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.31, 3.67, 3.26 [10:04:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.69, 1.37, 1.63 [10:04:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.40, 1.81, 1.94 [10:05:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.41, 2.94, 3.04 [10:08:50] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:10:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.31, 1.91, 1.93 [10:12:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.15, 2.32, 1.86 [10:12:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.52, 1.79, 1.89 [10:16:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.51, 1.98, 1.84 [10:18:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.07, 1.80, 1.84 [10:22:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.99, 1.83, 1.84 [10:26:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.47, 2.06, 1.86 [10:30:31] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.34, 1.70, 1.76 [10:34:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.37, 1.74, 1.74 [10:36:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.22, 1.49, 1.65 [10:36:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.28, 1.58, 1.68 [10:37:55] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp31.miraheze.org - NS RECORDS OK [10:42:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.90, 1.91, 1.78 [10:44:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.16, 1.95, 1.81 [10:51:07] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.42, 2.16, 1.87 [10:53:15] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.62, 1.94, 1.82 [10:57:05] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.89, 1.49, 1.67 [11:03:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.06, 1.83, 1.75 [11:13:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.85, 1.92, 1.91 [11:19:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.15, 1.75, 1.82 [11:23:32] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [11:25:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.90, 1.62, 1.80 [11:25:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [11:29:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.85, 1.35, 1.66 [11:33:00] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:35:50] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.12, 11.14, 9.26 [11:37:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.63, 10.50, 9.25 [11:37:48] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.13, 2.99, 2.34 [11:41:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.38, 10.02, 9.34 [11:47:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.30, 1.70, 1.99 [11:51:32] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:59:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.08, 1.32, 1.66 [12:06:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.79, 1.80, 2.00 [12:07:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.99, 1.76, 1.71 [12:08:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.34, 1.99, 2.04 [12:10:47] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [12:10:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.48, 1.83, 1.98 [12:11:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.04, 1.55, 1.64 [12:12:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:12:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.18, 2.05, 2.04 [12:14:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.27, 1.77, 1.94 [12:17:53] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.02, 1.86, 1.75 [12:18:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.73, 2.17, 2.06 [12:22:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 2.64, 2.19 [12:24:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.89, 3.37, 2.51 [12:26:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.52, 5.17, 3.28 [12:28:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.87, 3.88, 2.97 [12:29:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 24.84, 16.28, 11.47 [12:29:24] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.29, 1.93, 1.98 [12:30:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.35, 4.03, 3.12 [12:31:19] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.31, 2.08, 2.02 [12:35:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.54, 1.88, 1.96 [12:37:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.11, 11.61, 11.44 [12:37:04] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.25, 1.95, 1.97 [12:41:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.56, 1.98, 2.00 [12:42:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.25, 3.70, 3.79 [12:43:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.49, 4.15, 3.90 [12:45:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.97, 3.86 [12:47:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.42, 12.97, 11.62 [12:47:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.30, 1.91, 1.93 [12:47:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.16, 3.70, 3.85 [12:53:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.78, 3.81 [12:54:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.30, 11.86, 10.19 [12:55:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 4.03, 3.84 [12:55:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.36, 3.26, 3.62 [12:57:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.93, 3.68, 3.71 [13:04:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.56, 11.63, 11.08 [13:05:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.30, 3.67, 3.94 [13:09:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.80, 3.61, 3.81 [13:11:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.14, 3.61 [13:11:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.13, 3.55, 3.94 [13:12:04] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 19.34, 13.94, 11.91 [13:15:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.18, 2.92, 3.39 [13:17:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.61, 2.63, 3.40 [13:25:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.64, 4.89, 4.15 [13:26:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.83, 3.62, 3.48 [13:27:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.61, 3.67, 3.79 [13:28:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.57, 2.99, 3.27 [13:31:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.92, 10.93, 11.74 [13:31:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 4.40, 4.05 [13:35:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.53, 3.54, 3.78 [13:39:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.36, 1.74, 1.96 [13:39:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.47, 12.83, 12.05 [13:39:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.98, 4.18, 3.88 [13:41:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.19, 5.03, 3.98 [13:41:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.73, 3.75 [13:43:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.07, 3.90, 3.79 [13:45:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.03, 3.46, 3.56 [13:45:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.87, 2.35, 2.13 [13:45:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.54, 11.59, 11.93 [13:45:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.75, 3.47, 3.64 [13:46:57] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.54, 4.18, 3.80 [13:48:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.39, 3.14, 3.47 [13:49:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.47, 1.89, 1.99 [13:51:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.36, 2.05, 2.04 [13:52:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.87, 2.86, 3.31 [13:53:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.42, 8.08, 10.13 [13:55:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.83, 1.95, 2.00 [13:55:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.17, 2.73, 3.26 [13:59:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.38, 9.74, 11.97 [14:03:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.75, 1.33, 1.68 [14:13:48] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.63, 1.79, 1.75 [14:15:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.96, 8.35, 9.90 [14:17:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.74, 2.19, 1.92 [14:21:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.93, 11.85, 10.84 [14:21:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.42, 10.58, 9.32 [14:21:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.48, 5.11, 3.82 [14:23:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.35, 10.48, 9.42 [14:25:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.84, 11.49, 9.88 [14:26:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.33, 4.80, 3.59 [15:37:00] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.65, 3.86, 2.97 [15:38:54] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.50, 3.98, 3.14 [15:40:54] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.09, 3.40, 3.04 [15:55:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.97, 9.94, 11.81 [15:57:04] [02puppet] 07Universal-Omega opened pull request 03#2772: Raise varnish::interval-check to 60 seconds again - 13https://github.com/miraheze/puppet/pull/2772 [15:57:05] [url] Page not found · GitHub · GitHub | github.com [16:11:10] [02puppet] 07Universal-Omega opened pull request 03#2773: test131: use correct IP for redis (test131 not test101) - 13https://github.com/miraheze/puppet/pull/2773 [16:11:11] [url] Page not found · GitHub · GitHub | github.com [16:12:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d25883645b88...f394b43d6e8f [16:12:12] [url] Comparing d25883645b88...f394b43d6e8f · miraheze/mw-config · GitHub | github.com [16:12:13] [02miraheze/mw-config] 07Universal-Omega 03f394b43 - use correct IP for redis on beta (test131 not test101) [16:13:23] miraheze/mw-config - Universal-Omega the build passed. [16:15:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 2.77, 3.92 [16:17:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.97, 4.27, 4.30 [16:17:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.46, 11.81, 11.05 [16:18:54] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.61, 2.73, 2.37 [16:20:54] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.79, 3.45, 2.68 [16:22:54] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 3.28, 3.40, 2.76 [16:26:38] !log [@test131] starting deploy of {'config': True} to all [16:26:39] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [16:26:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:27:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:29:18] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:54] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.41, 4.08, 3.32 [16:32:54] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.42, 3.90, 3.36 [16:33:20] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 3.161 second response time [16:38:54] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.52, 3.89, 3.49 [16:40:00] !log [@mwtask141] starting deploy of {'config': True} to all [16:40:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:40:18] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 18s [16:40:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:40:54] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.47, 3.40, 3.37 [16:48:04] [02puppet] 07JohnFLewis commented on pull request 03#2772: Raise varnish::interval-check to 60 seconds again - 13https://github.com/miraheze/puppet/pull/2772#issuecomment-1210981866 [16:48:04] [url] Raise varnish::interval-check to 60 seconds again by Universal-Omega · Pull Request #2772 · miraheze/puppet · GitHub | github.com [16:50:28] [02puppet] 07Universal-Omega commented on pull request 03#2772: Raise varnish::interval-check to 60 seconds again - 13https://github.com/miraheze/puppet/pull/2772#issuecomment-1210984657 [16:50:28] [url] Raise varnish::interval-check to 60 seconds again by Universal-Omega · Pull Request #2772 · miraheze/puppet · GitHub | github.com [16:52:30] [02puppet] 07JohnFLewis commented on pull request 03#2772: Raise varnish::interval-check to 60 seconds again - 13https://github.com/miraheze/puppet/pull/2772#issuecomment-1210987176 [16:52:30] [url] Raise varnish::interval-check to 60 seconds again by Universal-Omega · Pull Request #2772 · miraheze/puppet · GitHub | github.com [16:52:31] [02puppet] 07JohnFLewis closed pull request 03#2772: Raise varnish::interval-check to 60 seconds again - 13https://github.com/miraheze/puppet/pull/2772 [16:52:32] [url] Page not found · GitHub · GitHub | github.com [16:53:14] [02puppet] 07Universal-Omega commented on pull request 03#2772: Raise varnish::interval-check to 60 seconds again - 13https://github.com/miraheze/puppet/pull/2772#issuecomment-1210988037 [16:53:15] [url] Raise varnish::interval-check to 60 seconds again by Universal-Omega · Pull Request #2772 · miraheze/puppet · GitHub | github.com [16:57:15] [02puppet] 07JohnFLewis reopened pull request 03#2772: Raise varnish::interval-check to 60 seconds again - 13https://github.com/miraheze/puppet/pull/2772 [16:57:16] [url] Page not found · GitHub · GitHub | github.com [16:57:22] [02puppet] 07JohnFLewis closed pull request 03#2772: Raise varnish::interval-check to 60 seconds again - 13https://github.com/miraheze/puppet/pull/2772 [16:57:23] [url] Page not found · GitHub · GitHub | github.com [16:57:24] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c2f013a3142a...c2b2c6564016 [16:57:24] [url] Comparing c2f013a3142a...c2b2c6564016 · miraheze/puppet · GitHub | github.com [16:57:25] [02miraheze/puppet] 07Universal-Omega 03c2b2c65 - Raise varnish::interval-check to 60 seconds again (#2772) [16:58:30] [02puppet] 07JohnFLewis commented on pull request 03#2772: Raise varnish::interval-check to 60 seconds again - 13https://github.com/miraheze/puppet/pull/2772#issuecomment-1210994545 [16:58:30] [url] Raise varnish::interval-check to 60 seconds again by Universal-Omega · Pull Request #2772 · miraheze/puppet · GitHub | github.com [17:09:30] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f394b43d6e8f...044c7d43e523 [17:09:31] [url] Comparing f394b43d6e8f...044c7d43e523 · miraheze/mw-config · GitHub | github.com [17:09:32] [02miraheze/mw-config] 07paladox 03044c7d4 - Revert "Don't create any new wikis on c4 (db141) temporarily (#4865)" [17:10:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/044c7d43e523...cd77de13e95e [17:10:36] [url] Comparing 044c7d43e523...cd77de13e95e · miraheze/mw-config · GitHub | github.com [17:10:37] [02miraheze/mw-config] 07paladox 03cd77de1 - Add c6 (db141) to wgCreateWikiDatabaseClustersInactive [17:10:51] miraheze/mw-config - paladox the build passed. [17:11:00] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [17:11:16] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 15s [17:11:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:11:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:12:08] miraheze/mw-config - paladox the build passed. [17:12:10] [02puppet] 07paladox closed pull request 03#2773: test131: use correct IP for redis (test131 not test101) - 13https://github.com/miraheze/puppet/pull/2773 [17:12:11] [url] Page not found · GitHub · GitHub | github.com [17:12:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c2b2c6564016...fdf05b23b713 [17:12:13] [url] Comparing c2b2c6564016...fdf05b23b713 · miraheze/puppet · GitHub | github.com [17:12:13] [02miraheze/puppet] 07Universal-Omega 03fdf05b2 - test131: use correct IP for redis (test131 not test101) (#2773) [17:12:48] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to test131 [17:12:49] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to test131 - SUCCESS in 1s [17:12:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:13:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:35:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.48, 9.92, 11.94 [17:37:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.17, 10.58, 11.91 [17:42:44] [02puppet] 07Universal-Omega synchronize pull request 03#2770: Cleanup hieradata - 13https://github.com/miraheze/puppet/pull/2770 [17:42:44] [url] Page not found · GitHub · GitHub | github.com [17:43:01] [02puppet] 07Universal-Omega synchronize pull request 03#2770: Cleanup hieradata - 13https://github.com/miraheze/puppet/pull/2770 [17:43:01] [url] Page not found · GitHub · GitHub | github.com [17:48:25] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:48:27] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:48:29] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:48:33] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:48:36] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:48:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb [17:48:40] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [17:48:40] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:48:40] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:48:42] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:48:42] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:48:43] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:48:45] PROBLEM - cp20 Current Load on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:48:47] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:48:49] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:48:50] PROBLEM - cp20 NTP time on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:48:51] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:48:52] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:49:01] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:49:06] PROBLEM - cp20 Puppet on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:49:08] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:49:08] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:49:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb [17:49:09] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:49:12] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:49:14] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [17:51:53] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw132 [17:53:51] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [17:56:49] [02puppet] 07Universal-Omega opened pull request 03#2774: Remove unused jobchron from hieradata - 13https://github.com/miraheze/puppet/pull/2774 [17:56:50] [url] Page not found · GitHub · GitHub | github.com [17:57:24] [02puppet] 07Universal-Omega synchronize pull request 03#2774: Remove unused jobchron from hieradata - 13https://github.com/miraheze/puppet/pull/2774 [17:57:25] [url] Page not found · GitHub · GitHub | github.com [17:58:17] [02puppet] 07Universal-Omega edited pull request 03#2774: Remove unused jobchron from hieradata - 13https://github.com/miraheze/puppet/pull/2774 [17:58:17] ... [18:05:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.44, 11.26, 11.94 [18:09:51] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.001175165176 secs [18:10:33] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.267 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [18:10:35] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.46, 0.18, 0.07 [18:10:39] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:10:42] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.043 second response time [18:10:43] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.053 second response time [18:10:46] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 12096 MB (31% inode=96%); [18:10:49] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.169 second response time [18:10:50] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3384 bytes in 3.072 second response time [18:10:52] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [18:10:52] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.487 second response time [18:10:58] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [18:10:58] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.028 second response time [18:10:59] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 46 minutes ago with 0 failures [18:11:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.05, 9.78, 11.65 [18:11:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:11:10] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 18% [18:11:10] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [18:11:13] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.020 second response time [18:11:28] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [18:11:41] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.023 second response time [18:11:41] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw122 [18:12:00] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.933 second response time [18:12:16] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.021 second response time [18:12:23] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.018 second response time [18:12:27] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.051 second response time [18:12:28] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [18:13:00] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.023 second response time [18:13:39] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [18:19:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.41, 10.31, 10.60 [18:21:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.55, 11.77, 11.41 [18:21:32] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 4215 MB (10% inode=96%); [18:21:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.42, 10.39, 10.65 [18:21:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [18:24:11] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.89, 1.69, 1.27 [18:25:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.88, 11.66, 11.50 [18:26:11] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.38, 1.88, 1.39 [18:27:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:27:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.52, 9.11, 9.98 [18:29:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.73, 1.45, 1.88 [18:30:11] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.72, 1.91, 1.53 [18:30:30] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw131 [18:31:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.43, 11.47, 11.41 [18:33:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.42, 11.50, 11.42 [18:33:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.53, 1.87, 1.94 [18:34:11] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.55, 1.70, 1.53 [18:34:26] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [18:35:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.24, 12.97, 11.99 [18:35:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.85, 1.49, 1.80 [18:41:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.06, 11.78, 11.83 [18:41:06] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw131 [18:42:24] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw141 [18:44:25] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [18:45:00] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [18:45:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.02, 11.08, 11.41 [18:47:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.51, 10.29, 11.08 [18:47:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.11, 1.44, 1.67 [18:50:41] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 4216 MB (10% inode=96%); [18:51:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.95, 2.01, 1.87 [18:51:04] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw142 [18:52:09] [02puppet] 07MacFan4000 opened pull request 03#2775: check_reverse_dns: allow cname flattening for porkbun dns - 13https://github.com/miraheze/puppet/pull/2775 [18:52:09] [url] Page not found · GitHub · GitHub | github.com [18:53:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.29, 1.74, 1.79 [18:53:02] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [18:57:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 22.55, 14.43, 12.11 [18:57:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.17, 2.63, 2.13 [18:58:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.20, 13.45, 10.78 [19:22:39] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:24:34] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [20:03:39] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw142 [20:05:39] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [20:10:57] [02puppet] 07paladox reviewed pull request 03#2775 commit - 13https://github.com/miraheze/puppet/pull/2775#discussion_r942863983 [20:10:58] [url] check_reverse_dns: allow cname flattening for porkbun dns by MacFan4000 · Pull Request #2775 · miraheze/puppet · GitHub | github.com [20:11:02] [02puppet] 07paladox closed pull request 03#2775: check_reverse_dns: allow cname flattening for porkbun dns - 13https://github.com/miraheze/puppet/pull/2775 [20:11:02] ... [20:11:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fdf05b23b713...46c9f52fc157 [20:11:04] [url] Comparing fdf05b23b713...46c9f52fc157 · miraheze/puppet · GitHub | github.com [20:11:05] [02miraheze/puppet] 07MacFan4000 0346c9f52 - check_reverse_dns: allow cname flattening for porkbun dns (#2775) [20:22:55] paladox: send me a Dm on discord with a link to ^ and I'll make it cleaner in the morning [20:23:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.61, 10.53, 11.84 [20:25:43] Thanks! [20:27:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.48, 3.96 [20:27:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.50, 10.77, 11.58 [20:29:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.28, 1.77, 2.00 [20:29:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.66, 10.26, 11.29 [20:31:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.24, 2.00, 2.05 [20:33:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.44, 1.75, 1.95 [20:35:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.08, 11.80, 11.62 [20:37:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.52, 4.89, 4.23 [20:39:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.02, 2.02, 2.00 [20:41:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.10, 3.65, 3.89 [20:43:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.51, 1.83, 1.93 [20:43:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.27, 11.08, 11.67 [20:49:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.06, 2.35, 2.07 [20:51:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.67, 8.07, 9.97 [20:51:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 2.95, 3.95 [20:53:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.78, 9.50, 11.77 [20:53:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.18, 1.81, 1.91 [20:53:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.92, 2.50, 3.23 [20:55:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.54, 10.36, 11.78 [20:57:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.34, 10.12, 11.53 [20:57:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.29, 1.99, 1.95 [20:57:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.59, 2.31, 3.37 [20:59:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.59, 11.94, 11.93 [20:59:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.57, 1.84, 1.90 [21:00:27] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.59, 5.90, 3.74 [21:01:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.42, 11.04, 11.64 [21:01:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.42, 3.62 [21:04:27] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.52, 6.13, 4.36 [21:05:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.15, 12.93, 12.14 [21:05:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.99, 4.58, 4.04 [21:06:52] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw131 [21:07:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.09, 11.07, 11.57 [21:08:52] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [21:09:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.31, 2.01, 1.89 [21:09:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.91, 3.46, 3.71 [21:11:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.35, 1.69, 1.78 [21:13:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.27, 1.81, 1.81 [21:13:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.55, 2.73, 3.33 [21:15:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.52, 8.00, 10.03 [21:15:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.45, 1.70, 1.77 [21:17:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.18, 1.86, 1.83 [21:17:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cd77de13e95e...ae3bebca4cce [21:17:26] [url] Comparing cd77de13e95e...ae3bebca4cce · miraheze/mw-config · GitHub | github.com [21:17:27] [02miraheze/mw-config] 07Universal-Omega 03ae3bebc - T9650: Update $smwgNamespacesWithSemanticLinks for constantnoblewiki [21:18:56] miraheze/mw-config - Universal-Omega the build passed. [21:21:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.82, 1.97, 1.89 [21:23:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.09, 2.49, 2.09 [21:27:20] !log [@test131] starting deploy of {'config': True} to all [21:27:21] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [21:27:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:27:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:29:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.40, 11.55, 10.64 [21:33:32] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:34:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.92, 10.86, 9.60 [21:34:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.89, 3.94, 3.44 [21:38:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.92, 12.61, 10.59 [21:39:29] !log [@mwtask141] starting deploy of {'config': True} to all [21:39:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:39:39] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 10s [21:39:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:40:41] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.30, 3.27, 3.30 [21:42:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.53, 11.50, 10.71 [21:43:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.99, 11.43, 11.76 [21:47:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.68, 11.62, 11.75 [21:49:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.57, 3.40 [21:50:11] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.24, 12.35, 11.08 [21:51:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.57, 3.43 [21:52:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.73, 3.91, 3.30 [21:52:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.57, 11.75, 11.02 [21:53:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.41, 3.20, 3.31 [21:54:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.35, 12.09, 11.21 [21:57:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.91, 3.72, 3.45 [21:59:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.67, 3.13, 3.28 [21:59:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.99, 11.25, 11.25 [22:03:05] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp30.miraheze.org - NS RECORDS OK [22:05:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.93, 12.15, 11.44 [22:09:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.99, 4.50, 3.76 [22:11:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.24, 4.24, 3.61 [22:17:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.93, 3.72 [22:23:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.35, 3.07, 3.40 [22:23:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.09, 9.73, 11.66 [22:33:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.84, 3.43, 3.97 [22:37:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.53, 8.57, 9.96 [22:39:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.18, 2.56, 3.36 [22:40:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.18, 1.66, 1.99 [22:41:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.46, 9.03, 11.61 [22:42:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.47, 1.94, 2.05 [22:49:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.97, 9.86, 10.71 [22:49:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.79, 4.17, 3.64 [22:51:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.81, 10.88, 11.03 [22:51:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.58, 3.48 [22:53:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.41, 11.15, 9.84 [22:55:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.18, 13.96, 12.15 [22:57:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.81, 1.36, 1.93 [22:57:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.56, 4.54, 3.86 [22:59:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.25, 1.68, 1.97 [23:01:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.87, 1.71, 1.95 [23:01:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.03, 10.98, 10.64 [23:03:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.12, 9.64, 10.18 [23:05:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.73, 3.91, 3.94 [23:06:27] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 20.79, 16.43, 10.02 [23:06:43] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:06:50] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:07:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.70, 10.16, 10.35 [23:07:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 4.07, 4.00 [23:08:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.88, 4.07, 3.30 [23:09:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.54, 11.62, 10.82 [23:10:25] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 5.40, 10.35, 8.96 [23:10:45] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.309 second response time [23:10:53] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [23:11:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.52, 1.86, 1.85 [23:11:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.09, 11.40, 10.85 [23:12:25] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.18, 8.97, 8.62 [23:20:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.82, 3.62, 3.63 [23:25:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.76, 2.90, 3.37 [23:25:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.15, 11.33, 10.88 [23:25:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.02, 2.80, 3.87 [23:29:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 3.45, 3.86 [23:35:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.63, 1.37, 1.86 [23:37:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.43, 3.99, 2.85 [23:39:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.12, 3.30, 3.85 [23:41:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.52, 3.38 [23:43:17] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 9.05, 6.48, 3.94 [23:43:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.97, 3.34, 3.33 [23:45:13] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.00, 6.64, 4.30 [23:45:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.26, 11.34, 11.80 [23:47:10] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.94, 6.79, 4.65 [23:47:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.91, 11.82, 11.89 [23:47:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.24, 4.33, 3.95 [23:51:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.98, 3.94 [23:53:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.71, 11.14, 11.74 [23:57:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.89, 3.78, 3.51