[00:00:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.99, 9.30, 8.59 [00:00:37] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.77, 10.64, 10.14 [00:01:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 15.16, 12.27, 10.29 [00:02:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.72, 8.90, 8.55 [00:03:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.81, 4.92, 4.12 [00:04:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.69, 10.61, 10.25 [00:04:52] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.49, 1.90, 1.76 [00:05:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.64, 11.39, 9.41 [00:06:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.07, 1.67, 1.70 [00:07:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.69, 3.95, 3.93 [00:09:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.33, 4.44, 4.10 [00:10:13] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.98, 11.77, 10.86 [00:11:16] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [00:13:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.70, 9.04, 9.16 [00:15:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.07, 11.12, 11.62 [00:15:59] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.05, 11.56, 11.12 [00:20:37] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=laalternativowiki --no-updates --username-prefix=w /mnt/mediawiki-static/metawiki/ImportDump/laalternativowiki-20220725161859.xml (START) [00:20:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.81, 11.55, 11.53 [00:21:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.72, 11.45, 11.07 [00:23:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.94, 10.85, 11.27 [00:23:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.03, 10.69, 10.85 [00:25:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.95, 2.28, 1.78 [00:27:44] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.28, 1.85, 1.68 [00:27:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.29, 1.74, 2.00 [00:31:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 17.22, 12.84, 11.83 [00:33:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.94, 11.37, 11.43 [00:33:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.52, 1.61 [00:37:05] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.93, 9.51, 10.06 [00:38:03] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=laalternativowiki --no-updates --username-prefix=w /mnt/mediawiki-static/metawiki/ImportDump/laalternativowiki-20220725161859.xml (END - exit=0) [00:38:04] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=laalternativowiki (START) [00:38:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:38:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:15] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [00:43:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.31, 10.36, 10.19 [00:43:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.41, 1.58, 1.70 [00:44:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.31, 3.75, 3.99 [00:46:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.73, 4.02, 4.06 [00:47:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.39, 9.77, 10.02 [00:48:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.58, 11.26, 11.95 [00:49:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.25, 1.89, 1.78 [00:50:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.46, 12.46, 12.34 [00:50:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.68, 3.94 [00:52:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.80, 10.20, 9.40 [00:52:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.53, 3.77, 3.92 [00:53:59] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.06, 9.45, 9.22 [00:54:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.23, 11.51, 11.96 [00:55:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.68, 1.84, 1.83 [00:56:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.30, 12.68, 12.37 [00:56:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.88, 3.28, 4.00 [00:57:12] PROBLEM - gluster111 APT on gluster111 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:57:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.13, 10.89, 10.20 [00:59:15] PROBLEM - gluster111 APT on gluster111 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (15 critical updates). [00:59:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.89, 10.22, 10.04 [01:00:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.60, 4.36, 4.22 [01:01:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 16.27, 11.69, 11.03 [01:01:52] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.92, 3.35, 1.98 [01:03:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.34, 11.38, 11.02 [01:05:29] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.83, 10.13, 10.19 [01:05:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.15, 2.20, 1.91 [01:07:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.39, 3.63, 4.00 [01:08:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.26, 3.35, 3.87 [01:10:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.34, 3.54, 3.86 [01:10:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.04, 3.37, 3.90 [01:11:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.35, 3.94, 4.01 [01:11:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.31, 10.67, 10.60 [01:11:52] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.03, 3.35, 3.09 [01:12:13] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 17.05, 13.56, 11.46 [01:13:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.75, 3.86, 3.97 [01:13:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.15, 10.18, 10.44 [01:13:52] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 2.00, 1.51, 1.38 [01:14:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.85, 10.81, 9.81 [01:14:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 3.92, 3.96 [01:14:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.73, 4.02, 4.07 [01:15:52] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 5.86, 4.59, 3.67 [01:16:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.93, 11.94, 11.26 [01:16:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.75, 3.96 [01:17:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.04, 9.33, 10.11 [01:17:48] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.29, 1.45, 1.39 [01:19:34] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.27, 1.63, 1.25 [01:20:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.81, 10.19, 9.90 [01:21:33] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.63, 1.64, 1.29 [01:22:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.30, 3.65 [01:24:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.75, 3.73 [01:24:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.46, 3.67 [01:25:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 3.57, 3.70 [01:27:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.33, 3.59 [01:27:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.24, 9.21, 10.16 [01:28:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.49, 10.02, 9.74 [01:28:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.18, 4.01, 3.86 [01:29:42] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=laalternativowiki (END - exit=0) [01:29:43] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=laalternativowiki --active --update (END - exit=0) [01:29:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:29:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:30:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.80, 9.16, 9.46 [01:30:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.69, 3.76 [01:31:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.36, 3.91, 3.80 [01:32:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.72, 3.32, 3.72 [01:33:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.23, 2.07, 1.81 [01:33:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.34, 3.30, 3.59 [01:35:26] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.22, 1.92, 1.80 [01:37:24] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.22, 2.32, 1.94 [01:37:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.53, 2.96, 3.39 [01:38:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.80, 3.42, 3.51 [01:39:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.19, 1.87, 1.82 [01:41:52] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.65, 1.37, 3.60 [01:42:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.15, 3.63, 3.57 [01:42:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.73, 3.66 [01:43:52] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.09, 1.22, 3.27 [01:44:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.66, 3.60 [01:44:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.74, 3.66 [01:45:16] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.34, 1.49, 1.66 [01:50:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 3.75, 3.61 [01:53:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.59, 1.85, 1.75 [01:53:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.57, 3.50 [01:54:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.70, 3.65 [01:55:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.07, 1.62, 1.69 [01:57:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.41, 3.17, 3.35 [01:58:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.53, 4.52, 3.97 [01:59:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.23, 10.57, 9.87 [02:01:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 16.33, 12.67, 10.72 [02:02:53] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.08, 1.89, 1.70 [02:03:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.49, 11.14, 10.44 [02:03:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.35, 1.61, 1.98 [02:04:07] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.53, 4.69, 3.89 [02:05:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.10, 10.06, 10.13 [02:06:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.84, 3.91 [02:06:52] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.98, 1.97, 1.78 [02:10:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.65, 10.66, 11.89 [02:10:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.97, 4.22, 4.08 [02:10:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.69, 1.68, 1.69 [02:13:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.58, 10.61, 9.63 [02:13:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.03, 1.32, 1.68 [02:14:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 15.74, 12.42, 12.23 [02:15:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.86, 3.89 [02:16:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.99, 11.55, 11.95 [02:16:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.04, 3.65, 3.88 [02:16:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.72, 4.00 [02:17:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.59, 9.24, 9.31 [02:17:57] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.94, 3.92 [02:18:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.91, 4.61, 4.21 [02:18:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.54, 4.35, 4.19 [02:19:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.29, 1.93, 1.82 [02:27:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.70, 1.92, 1.91 [02:31:36] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.85, 1.78, 1.62 [02:32:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.01, 8.97, 10.19 [02:33:34] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.23, 1.64, 1.59 [02:35:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.63, 1.48, 1.67 [02:36:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.21, 10.60, 10.61 [02:40:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.24, 8.35, 9.71 [02:43:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.65, 12.60, 10.32 [02:45:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.68, 11.42, 10.16 [02:47:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.73, 1.99, 1.75 [02:49:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.07, 9.77, 9.79 [02:49:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.26, 10.22, 9.63 [02:51:46] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.29, 9.91, 9.59 [02:54:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.87, 3.21, 3.86 [02:55:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.58, 4.00 [02:56:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 3.87, 4.03 [02:57:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 4.01, 4.11 [02:58:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.14, 3.50, 3.87 [03:00:00] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.70, 1.83, 1.61 [03:02:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.18, 3.38, 4.00 [03:03:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.70, 4.00 [03:03:56] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.03, 1.77, 1.66 [03:04:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.49, 3.96, 4.10 [03:04:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 3.60, 3.80 [03:05:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.27, 4.45, 4.21 [03:06:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 16.20, 11.89, 10.31 [03:07:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.25, 1.63, 1.64 [03:08:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.21, 11.47, 10.37 [03:08:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.56, 3.77 [03:10:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.75, 4.15, 3.96 [03:12:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.86, 3.88 [03:14:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.60, 10.09, 10.14 [03:14:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.77, 4.34, 4.03 [03:15:39] PROBLEM - www.wikitipp.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.wikitipp.de' expires in 10 day(s) (Sun 07 Aug 2022 04:23:09 GMT +0000). [03:16:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.77, 3.48, 3.92 [03:17:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4d4fe39238fb...b6c88174fb03 [03:17:25] [url] Comparing 4d4fe39238fb...b6c88174fb03 · miraheze/ssl · GitHub | github.com [03:17:26] [02miraheze/ssl] 07MirahezeSSLBot 03b6c8817 - Bot: Update SSL cert for www.wikitipp.de [03:20:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 4.06, 4.05 [03:21:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.80, 10.71, 10.22 [03:22:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.84, 3.98 [03:25:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.75, 11.86, 10.92 [03:26:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.74, 3.95, 4.01 [03:27:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.07, 11.86, 11.02 [03:29:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.52, 3.83, 3.97 [03:29:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.02, 11.20, 10.89 [03:32:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.74, 3.92 [03:33:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.62, 4.18, 4.00 [03:35:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.69, 3.63, 3.80 [03:35:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.70, 11.87, 11.22 [03:36:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.54, 4.05, 4.01 [03:37:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.20, 11.44, 11.13 [03:38:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.78, 3.92 [03:39:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.48, 4.27, 3.95 [03:42:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.90, 3.79, 3.98 [03:43:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.58, 3.75 [03:43:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.38, 1.60, 1.94 [03:44:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.28, 4.18, 4.05 [03:45:38] PROBLEM - www.wikitipp.de - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikitipp.de and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:47:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.93, 2.08, 2.03 [03:48:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.77, 3.82, 3.97 [03:49:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.36, 3.89, 3.81 [03:50:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.98, 4.03, 4.02 [03:51:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.77, 11.24, 10.96 [03:51:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.89, 3.83 [03:52:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 4.14, 4.07 [03:53:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.19, 10.69, 10.82 [03:54:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.55, 3.68, 3.91 [03:54:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.66, 3.73, 3.94 [03:55:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.38, 1.84, 2.00 [03:59:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.49, 8.89, 9.98 [03:59:52] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.92, 1.78, 1.62 [04:01:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.55, 3.66, 3.61 [04:01:50] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.40, 1.65, 1.59 [04:02:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.01, 3.82, 3.79 [04:03:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.93, 3.65, 3.61 [04:03:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.23, 1.28, 1.65 [04:04:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.54, 4.55, 4.09 [04:05:12] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.08, 10.52, 10.19 [04:05:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.10, 4.57, 3.95 [04:05:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.79, 10.27, 9.62 [04:07:48] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.52, 9.69, 9.48 [04:09:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.75, 10.48, 10.21 [04:11:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.82, 10.10, 10.10 [04:11:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.56, 3.94, 3.92 [04:13:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.15, 4.24, 4.04 [04:15:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.92, 3.95 [04:19:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.66, 1.95, 1.74 [04:22:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.65, 3.94 [04:23:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 4.03, 3.96 [04:23:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.40, 1.80, 1.74 [04:25:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.85, 3.90 [04:25:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.40, 1.64, 1.69 [04:27:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.80, 4.01, 3.96 [04:31:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.19, 3.51, 3.81 [04:32:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.48, 3.50, 3.90 [04:32:51] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.38, 1.94, 1.79 [04:36:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.64, 4.32, 4.12 [04:36:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.23, 4.25, 3.92 [04:37:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.47, 4.19, 3.94 [04:38:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.51, 9.78, 9.12 [04:40:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.32, 9.56, 9.13 [04:40:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.39, 9.52, 8.91 [04:41:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.85, 3.88 [04:42:36] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.69, 9.32, 8.90 [04:45:59] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.35, 1.94, 1.55 [04:46:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 3.53, 3.82 [04:47:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.53, 3.67 [04:48:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.62, 4.06, 3.96 [04:50:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.72, 3.86 [04:55:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.63, 3.72 [04:55:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.58, 1.97, 1.84 [04:57:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.16, 3.68, 3.71 [04:59:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.36, 3.15, 3.51 [04:59:45] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.44, 1.66 [05:00:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.59, 3.62, 3.97 [05:03:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.52, 2.54, 3.20 [05:07:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.33, 10.78, 10.23 [05:09:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.86, 9.53, 9.85 [05:10:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.23, 3.22 [05:10:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.17, 3.85, 3.71 [05:12:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.22, 3.36, 3.27 [05:12:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.62, 3.60, 3.63 [05:16:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.08, 2.73, 3.26 [05:18:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.87, 2.79, 3.23 [05:31:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.10, 1.50, 1.91 [05:39:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.18, 1.38, 1.69 [05:49:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.45, 1.89, 1.73 [05:55:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.42, 1.89, 1.83 [05:57:04] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.06, 3.93, 3.38 [05:58:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.71, 3.36 [06:00:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.23, 3.80, 3.43 [06:01:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.37, 1.51, 1.67 [06:02:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.59, 3.39 [06:04:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.83, 3.25, 3.29 [06:06:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.32, 3.16 [06:07:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.48, 1.95, 1.79 [06:08:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.65, 3.15, 3.12 [06:12:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.39, 3.50, 3.37 [06:13:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.43, 3.10 [06:14:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.81, 3.63, 3.33 [06:14:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.82, 2.87, 3.16 [06:14:52] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.89, 1.95, 1.62 [06:15:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.17, 3.24, 3.06 [06:16:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.35, 3.29, 3.25 [06:18:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.98, 1.51, 1.51 [06:27:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.96, 3.34, 3.11 [06:28:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.69, 3.59, 3.32 [06:28:47] [02puppet] 07Reception123 commented on pull request 03#2713: users: add varnishlog and varnishstat to cache-admins - 13https://github.com/miraheze/puppet/pull/2713#issuecomment-1197719777 [06:28:48] [url] users: add varnishlog and varnishstat to cache-admins by Universal-Omega · Pull Request #2713 · miraheze/puppet · GitHub | github.com [06:30:17] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.35, 3.40, 3.27 [06:31:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.81, 3.38, 3.20 [06:32:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.71, 10.38, 8.85 [06:38:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.85, 11.77, 9.97 [06:40:57] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:06] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:07] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 35.60, 18.24, 10.89 [06:41:26] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 21.09, 18.94, 12.01 [06:41:34] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.50, 9.93, 7.64 [06:42:27] PROBLEM - mw101 nutcracker process on mw101 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 116 (nutcracker), command name 'nutcracker' [06:42:53] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:42:56] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.792 second response time [06:43:02] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [06:43:05] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.314 second response time [06:43:24] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.310 second response time [06:43:25] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw101 mw112 [06:43:33] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw101 mw112 [06:43:34] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [06:44:47] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.291 second response time [06:45:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.17, 10.04, 8.25 [06:45:53] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.65, 11.97, 9.15 [06:46:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.76, 3.79, 3.31 [06:46:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.84, 3.81, 3.35 [06:47:25] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [06:47:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.56, 3.69, 3.24 [06:47:33] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [06:47:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.65, 10.86, 9.09 [06:47:57] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 24002 bytes in 0.526 second response time [06:48:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.85, 3.83, 3.41 [06:49:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.81, 9.57, 8.84 [06:50:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 4.04, 3.54 [06:51:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.66, 11.39, 11.38 [06:51:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.79, 3.40 [06:51:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.61, 10.94, 10.84 [06:52:03] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:56:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.94, 3.80 [06:56:58] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.15, 8.32, 10.01 [06:57:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.89, 9.21, 10.05 [06:58:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.85, 4.16, 3.91 [06:58:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.44, 10.84, 11.85 [06:58:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.37, 3.64, 3.64 [06:59:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.74, 3.12, 3.26 [07:00:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.74, 3.78 [07:00:27] RECOVERY - mw101 nutcracker process on mw101 is OK: PROCS OK: 1 process with UID = 116 (nutcracker), command name 'nutcracker' [07:01:59] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.565 second response time [07:06:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.41, 2.87, 3.25 [07:10:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.40, 8.39, 9.96 [07:11:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.65, 3.29, 3.40 [07:13:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.80, 2.96, 3.26 [07:20:02] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:28:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 2.93, 2.74 [07:34:16] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.16, 2.97, 2.87 [07:42:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.40, 4.03, 3.25 [07:43:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.70, 3.75, 3.11 [07:44:07] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.32, 3.33, 3.08 [07:45:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.32, 3.19, 2.98 [07:57:12] PROBLEM - gluster111 APT on gluster111 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [07:59:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.71, 3.24 [07:59:15] PROBLEM - gluster111 APT on gluster111 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (15 critical updates). [08:00:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.27, 3.24, 3.13 [08:04:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.86, 3.45 [08:08:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.95, 3.04, 3.21 [08:20:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.38, 10.40, 8.81 [08:21:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.74, 3.51, 3.21 [08:21:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.11, 3.71, 3.19 [08:22:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.26, 9.88, 8.81 [08:22:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.52, 3.18 [08:24:44] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.73, 3.16, 1.73 [08:25:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.58, 3.25, 3.19 [08:25:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.86, 3.38 [08:26:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.69, 4.02, 3.45 [08:28:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.27, 10.38, 9.39 [08:28:41] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.62, 1.70, 1.44 [08:28:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.86, 3.46 [08:29:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.66, 4.55, 3.72 [08:29:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.75, 4.15, 3.57 [08:30:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.93, 9.88, 9.32 [08:31:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.89, 3.54 [08:32:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.71, 4.00, 3.66 [08:34:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.84, 2.99, 3.22 [08:35:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.41, 3.08, 3.30 [08:36:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.80, 2.85, 3.27 [08:39:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.54, 3.43 [08:40:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.83, 3.51, 3.36 [08:42:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.97, 2.93, 3.16 [08:47:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.86, 3.10, 3.30 [09:12:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 3.02, 2.62 [09:14:10] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [09:14:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.81, 2.50, 2.48 [09:21:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.26, 2.91 [09:22:07] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query songcontests.eu. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:23:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.32, 3.13, 2.90 [09:26:19] PROBLEM - gluster121 APT on gluster121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [09:27:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.18, 1.54, 1.99 [09:28:22] PROBLEM - gluster121 APT on gluster121 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (15 critical updates). [09:31:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.60, 3.93, 3.23 [09:31:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.34, 3.05 [09:32:19] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.86, 1.58, 1.22 [09:33:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.82, 3.26 [09:33:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.03, 3.65, 3.22 [09:34:19] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.03, 1.37, 1.18 [09:35:14] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.94, 4.24, 3.49 [09:35:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.89, 3.34 [09:35:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.51, 1.36, 1.69 [09:37:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.82, 3.42 [09:37:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.60, 3.30 [09:39:09] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.30, 3.23 [09:39:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.50, 3.28, 3.22 [09:40:10] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [09:40:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.71, 3.70, 3.42 [09:42:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.13, 3.38, 3.33 [09:45:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.23, 1.81, 1.73 [09:46:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.43, 3.32 [09:47:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.55, 3.33 [09:48:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.26, 2.93, 3.14 [09:49:04] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.21, 1.93, 1.51 [09:49:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.30, 3.31, 3.26 [09:50:59] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.34, 1.75, 1.50 [09:51:13] RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: SSL OK - fc.songcontests.eu reverse DNS resolves to cp20.miraheze.org - CNAME OK [09:51:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 3.58, 3.43 [09:52:54] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.52, 1.31, 1.37 [09:55:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.22, 3.93, 3.58 [09:56:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.74, 1.53, 1.44 [09:57:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.50, 3.45 [09:58:39] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.39, 1.49, 1.44 [09:59:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.74, 3.32, 3.40 [10:02:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.44, 3.78, 3.57 [10:04:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.36, 3.22, 3.39 [10:08:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.30, 3.50, 3.19 [10:08:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.58, 3.51 [10:10:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.68, 3.96, 3.41 [10:10:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.20, 4.48, 3.84 [10:12:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.70, 9.60, 8.72 [10:12:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.85, 3.42 [10:12:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.87, 3.45 [10:12:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.89, 3.70 [10:14:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.79, 8.85, 8.55 [10:14:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.57, 3.35 [10:14:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.13, 4.13, 3.58 [10:14:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 3.88, 3.71 [10:16:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.70, 3.26, 3.26 [10:16:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.89, 3.76, 3.50 [10:16:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.00, 3.13, 3.45 [10:18:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.05, 3.88, 3.55 [10:18:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.37, 2.74, 3.26 [10:22:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.71, 3.56 [10:24:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.66, 3.55 [10:26:19] PROBLEM - gluster121 APT on gluster121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [10:26:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.40, 3.29, 3.43 [10:28:21] PROBLEM - gluster121 APT on gluster121 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (15 critical updates). [10:32:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.94, 3.04, 3.25 [10:32:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.67, 3.71, 3.52 [10:34:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.21, 3.04, 3.28 [10:37:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.31, 1.67, 1.98 [10:45:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.13, 1.27, 1.66 [10:49:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.33, 1.86, 1.82 [10:53:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.90, 3.45, 3.27 [10:53:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.56, 1.89, 1.87 [10:55:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.32, 4.08, 3.52 [10:57:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.64, 3.42 [10:58:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.48, 2.96, 3.20 [10:59:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.25, 1.43, 1.65 [11:07:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.82, 1.80, 1.73 [11:09:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.49, 1.66, 1.68 [11:10:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.03, 3.50, 3.27 [11:12:09] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/62f7e590ab14...ab3032b43b45 [11:12:10] [url] Comparing 62f7e590ab14...ab3032b43b45 · miraheze/ImportDump · GitHub | github.com [11:12:11] [02miraheze/ImportDump] 07translatewiki 03ab3032b - Localisation updates from https://translatewiki.net. [11:12:11] [url] Main page - translatewiki.net | translatewiki.net [11:12:12] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/3c82cd84b8e6...461154bc5ad1 [11:12:13] [url] Comparing 3c82cd84b8e6...461154bc5ad1 · miraheze/CreateWiki · GitHub | github.com [11:12:14] [02miraheze/CreateWiki] 07translatewiki 03461154b - Localisation updates from https://translatewiki.net. [11:12:14] [url] Main page - translatewiki.net | translatewiki.net [11:12:15] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/landing/compare/cd717d5bff71...289ccacbc8f8 [11:12:16] [url] Comparing cd717d5bff71...289ccacbc8f8 · miraheze/landing · GitHub | github.com [11:12:17] [02miraheze/landing] 07translatewiki 03289ccac - Localisation updates from https://translatewiki.net. [11:12:17] [url] Main page - translatewiki.net | translatewiki.net [11:12:18] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/fa2bf9b6c2f4...300a952ae29a [11:12:19] [url] Comparing fa2bf9b6c2f4...300a952ae29a · miraheze/MirahezeMagic · GitHub | github.com [11:12:20] [02miraheze/MirahezeMagic] 07translatewiki 03300a952 - Localisation updates from https://translatewiki.net. [11:12:20] [url] Main page - translatewiki.net | translatewiki.net [11:12:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.75, 3.37, 3.25 [11:15:08] miraheze/landing - translatewiki the build passed. [11:15:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 4.02, 3.44 [11:18:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.62, 3.98, 3.34 [11:19:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.68, 4.14, 3.62 [11:19:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.44, 1.94, 1.73 [11:20:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.85, 3.37 [11:21:00] miraheze/ImportDump - translatewiki the build passed. [11:21:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.94, 3.88, 3.60 [11:21:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.92, 1.95, 1.76 [11:22:07] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - fc.songcontests.eu All nameservers failed to answer the query. [11:22:35] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.22, 3.21, 3.19 [11:23:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.21, 2.00, 1.79 [11:25:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.09, 3.79, 3.79 [11:25:44] miraheze/CreateWiki - translatewiki the build passed. [11:25:46] miraheze/MirahezeMagic - translatewiki the build passed. [11:25:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.81, 1.89, 1.78 [11:28:19] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.95, 1.45, 1.12 [11:30:19] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.42, 1.37, 1.12 [11:31:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.52, 2.10, 1.88 [11:32:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.80, 3.05, 3.35 [11:36:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.93, 3.40, 3.46 [11:38:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.23, 3.37, 3.45 [11:40:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.13, 4.35, 3.82 [11:40:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.95, 3.80, 3.62 [11:41:10] PROBLEM - wiki.hsins.eu - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.hsins.eu. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:44:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.55, 3.21 [11:44:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.77, 4.04, 3.75 [11:46:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.82, 3.72 [11:50:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.77, 3.45 [11:50:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.18, 3.55, 3.77 [11:56:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.65, 3.45 [11:58:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.91, 3.02, 3.25 [12:00:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.17, 3.02, 3.35 [12:02:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.43, 3.77, 3.66 [12:04:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.85, 3.66 [12:05:26] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:06:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.54, 3.40 [12:08:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.81, 3.67 [12:09:49] PROBLEM - wiki.hsins.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.hsins.eu All nameservers failed to answer the query. [12:10:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.38, 3.74, 3.49 [12:10:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.52, 9.60, 8.62 [12:12:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.97, 10.13, 8.96 [12:12:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.81, 3.64 [12:13:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.36, 3.49, 3.48 [12:15:57] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.18, 3.85, 3.59 [12:16:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 3.94, 3.72 [12:17:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.56, 3.35, 3.44 [12:19:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.74, 3.22, 3.38 [12:20:20] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query fc.songcontests.eu. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:20:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.79, 3.73, 3.70 [12:22:02] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.48, 1.77, 1.47 [12:22:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.70, 11.77, 10.15 [12:22:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.66, 3.72 [12:23:57] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.09, 1.59, 1.44 [12:27:47] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.04, 1.85, 1.58 [12:28:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 3.40, 3.51 [12:29:42] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.03, 1.46, 1.46 [12:29:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.14, 1.65, 1.99 [12:30:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.03, 11.83, 10.64 [12:30:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 10.21, 5.47, 4.26 [12:31:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.00, 4.85, 3.92 [12:34:22] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp20.miraheze.org - CNAME OK [12:35:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.05, 9.95, 8.23 [12:37:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.03, 9.09, 8.12 [12:39:09] RECOVERY - wiki.hsins.eu - reverse DNS on sslhost is OK: SSL OK - wiki.hsins.eu reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:41:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.41, 1.46, 1.70 [12:44:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.03, 10.04, 11.12 [12:48:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 17.70, 13.07, 11.97 [12:49:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.18, 2.19, 1.87 [12:50:07] RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: SSL OK - fc.songcontests.eu reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:50:46] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.85, 1.86, 1.46 [12:51:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.76, 9.19, 8.39 [12:53:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.88, 9.25, 8.52 [12:54:35] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.34, 1.65, 1.46 [12:55:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.27, 1.89, 1.89 [13:01:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.97, 11.45, 9.81 [13:01:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.22, 1.44, 1.68 [13:03:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.34, 11.36, 9.98 [13:06:50] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:07:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.25, 9.31, 9.47 [13:14:05] PROBLEM - gluster101 APT on gluster101 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [13:15:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.18, 1.93, 1.75 [13:16:06] PROBLEM - gluster101 APT on gluster101 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (15 critical updates). [13:23:01] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.98, 1.75, 1.25 [13:24:23] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.88, 1.98, 1.46 [13:24:59] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.31, 1.56, 1.23 [13:26:19] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.73, 1.79, 1.45 [13:28:19] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.08, 1.79, 1.48 [13:30:19] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.55, 1.83, 1.54 [13:34:19] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.30, 1.51, 1.48 [13:35:40] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [13:36:33] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.36, 9.51, 9.15 [13:38:28] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.13, 8.53, 8.83 [13:42:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.58, 10.18, 11.56 [13:46:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.60, 11.45, 11.71 [13:52:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.68, 2.10, 1.41 [13:53:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.61, 10.18, 9.42 [13:53:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.43, 11.17, 9.20 [13:54:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.57, 1.30 [13:55:02] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.62, 9.83, 9.39 [13:55:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.14, 10.47, 9.20 [13:57:55] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.98, 8.88, 8.77 [14:05:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.32, 10.25, 9.31 [14:07:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.59, 9.17, 9.03 [14:11:08] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 15.91, 11.47, 9.54 [14:11:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.31, 1.61, 1.96 [14:13:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.67, 10.87, 9.54 [14:15:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.89, 9.92, 9.34 [14:16:19] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.11, 2.53, 1.83 [14:17:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.44, 1.83, 1.92 [14:21:33] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.37, 10.73, 9.68 [14:23:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.69, 10.43, 9.70 [14:24:19] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.81, 1.80, 1.87 [14:25:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.25, 1.76, 1.93 [14:27:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.09, 9.46, 9.52 [14:31:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.75, 11.04, 10.14 [14:32:19] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.69, 1.17, 1.58 [14:33:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.96, 9.77, 9.78 [14:34:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.54, 9.81, 11.86 [14:36:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.16, 11.67, 12.33 [14:38:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.77, 10.48, 11.81 [14:39:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.35, 1.51, 1.67 [14:41:24] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.36, 10.21, 9.39 [14:42:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.05, 11.72, 12.03 [14:43:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.04, 9.46, 9.21 [14:44:19] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.77, 1.93, 1.59 [14:46:19] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.24, 1.73, 1.57 [14:46:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.39, 11.29, 11.91 [14:48:19] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.11, 1.50, 1.50 [14:49:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.47, 1.85, 1.69 [14:52:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 15.35, 11.88, 11.79 [14:53:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.93, 1.95, 1.78 [14:54:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.90, 10.65, 11.36 [14:59:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.47, 2.12, 1.88 [15:00:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.56, 10.64, 11.00 [15:01:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.43, 11.64, 10.33 [15:01:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.11, 10.24, 9.46 [15:02:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.50, 10.33, 10.86 [15:04:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 20.80, 13.49, 11.90 [15:06:42] anyone around to run this maintenance script for me? https://phabricator.miraheze.org/T9552 It will error out, and I need to share that error with the maintainer [15:06:42] [url] ⚓ T9552 Cargo data is not storing on specific pages | phabricator.miraheze.org [15:07:21] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.49, 11.24, 10.48 [15:09:16] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.00, 10.84, 10.41 [15:09:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.03, 9.89, 9.61 [15:10:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.12, 11.36, 11.56 [15:10:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.46, 2.02, 1.44 [15:11:11] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.49, 12.30, 11.02 [15:12:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.93, 11.81, 11.67 [15:12:52] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.55, 1.77, 1.42 [15:13:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.60, 11.79, 10.99 [15:14:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.50, 11.64, 11.66 [15:14:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.74, 1.42, 1.33 [15:18:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.26, 11.82, 11.66 [15:20:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.77, 11.40, 11.53 [15:21:39] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=sixflagswiki --no-updates --username-prefix=wikia:sixflags /mnt/mediawiki-static/metawiki/ImportDump/sixflagswiki-20220726065637.xml (START) [15:21:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:22:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.18, 12.01, 11.73 [15:24:17] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.71, 11.61, 10.27 [15:24:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.54, 11.58, 9.75 [15:26:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.79, 10.72, 10.11 [15:26:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.27, 10.86, 9.72 [15:29:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.29, 11.33, 10.43 [15:30:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.65, 11.83, 10.31 [15:31:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.37, 11.81, 11.01 [15:34:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.49, 11.45, 10.48 [15:35:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.61, 10.90, 9.44 [15:38:58] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.48, 9.79, 10.00 [15:39:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.82, 10.85, 11.08 [15:39:08] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.81, 10.04, 9.39 [15:39:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.35, 11.20, 11.32 [15:44:19] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.26, 1.44, 1.17 [15:46:19] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.84, 1.53, 1.24 [15:47:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.09, 10.63, 10.78 [15:47:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 16.16, 12.30, 11.42 [15:48:19] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.30, 1.53, 1.28 [15:48:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.27, 9.53, 9.45 [15:49:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.30, 9.80, 10.44 [15:50:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 16.02, 12.65, 10.66 [15:52:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.73, 11.78, 10.58 [15:54:31] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=sixflagswiki --no-updates --username-prefix=wikia:sixflags /mnt/mediawiki-static/metawiki/ImportDump/sixflagswiki-20220726065637.xml (END - exit=0) [15:54:32] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=sixflagswiki (START) [15:54:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:54:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:54:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.75, 12.31, 10.91 [15:55:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.40, 11.44, 10.94 [15:56:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.25, 11.12, 10.64 [15:57:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.62, 11.11, 10.88 [15:59:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.94, 11.24, 11.68 [16:01:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 15.82, 12.95, 12.23 [16:01:32] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.30, 1.83, 1.44 [16:03:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.59, 10.45, 9.79 [16:03:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.55, 1.40, 1.33 [16:03:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.47, 1.65, 1.97 [16:04:58] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.67, 9.58, 10.14 [16:05:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.84, 11.28, 10.90 [16:06:50] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.07, 8.95, 9.40 [16:07:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.48, 10.56, 10.68 [16:07:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.47, 11.44, 11.85 [16:09:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.47, 10.86, 10.73 [16:13:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.64, 11.51, 11.19 [16:13:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.08, 1.32, 1.66 [16:17:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.74, 11.50, 11.25 [16:19:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.08, 10.18, 10.79 [16:19:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.68, 2.24, 1.95 [16:20:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.27, 9.80, 9.42 [16:22:03] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [16:22:58] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.80, 9.67, 9.44 [16:23:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.94, 2.02, 1.45 [16:25:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.49, 1.73, 1.42 [16:25:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.40, 1.85, 1.91 [16:26:54] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.96, 1.98, 1.35 [16:27:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.10, 9.16, 9.99 [16:27:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.56, 1.61, 1.40 [16:28:49] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.08, 1.58, 1.28 [16:29:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 15.73, 11.99, 11.01 [16:30:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.72, 11.13, 10.25 [16:31:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.70, 10.68, 10.41 [16:32:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.55, 11.05, 10.34 [16:33:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.40, 1.49, 1.70 [16:34:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.74, 9.68, 8.81 [16:34:35] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=sixflagswiki (END - exit=0) [16:34:40] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=sixflagswiki --active --update (END - exit=0) [16:34:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:34:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.42, 11.41, 10.55 [16:35:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.59, 10.76, 10.49 [16:35:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.31, 11.31, 10.61 [16:37:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.02, 12.16, 11.07 [16:37:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.72, 1.74 [16:39:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.71, 10.60, 9.50 [16:40:16] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.85, 9.74, 9.15 [16:41:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.84, 11.93, 11.63 [16:42:58] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.66, 9.57, 10.08 [16:43:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.60, 11.64, 11.52 [16:43:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.05, 10.86, 9.82 [16:43:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.39, 2.02, 1.84 [16:45:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.06, 10.91, 11.28 [16:45:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.43, 10.58, 9.86 [16:49:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.49, 11.30, 11.74 [16:49:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.15, 10.84, 10.08 [16:51:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.38, 13.22, 12.43 [16:51:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.01, 11.55, 11.42 [16:52:02] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:53:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.13, 11.71, 11.98 [16:53:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.31, 10.94, 10.30 [16:53:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.83, 11.16, 11.29 [16:53:33] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.90, 1.71, 1.44 [16:55:21] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.27, 9.84, 9.97 [16:55:31] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.40, 1.53, 1.40 [16:59:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.80, 11.02, 11.10 [17:01:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.85, 11.73, 11.38 [17:05:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.72, 12.23, 11.87 [17:07:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.63, 11.36, 11.27 [17:09:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.49, 10.17, 9.87 [17:11:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.25, 11.47, 10.40 [17:13:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.87, 11.02, 10.36 [17:14:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.04, 3.40, 4.00 [17:15:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.38, 10.80, 11.34 [17:15:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.25, 11.47, 10.58 [17:17:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.77, 10.92, 10.50 [17:17:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.03, 10.12, 9.27 [17:19:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.16, 9.65, 9.20 [17:21:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.57, 9.99, 10.20 [17:21:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.44, 11.85, 11.50 [17:21:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.69, 9.91, 9.32 [17:22:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.69, 4.34, 4.09 [17:23:01] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.58, 1.81, 1.43 [17:23:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.96, 11.93, 11.61 [17:23:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.92, 11.33, 9.91 [17:24:59] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.72, 1.44 [17:25:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.40, 10.19, 10.26 [17:25:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 16.60, 13.41, 12.17 [17:25:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 12.00, 11.46, 10.12 [17:26:57] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.52, 1.60, 1.43 [17:27:06] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.70, 11.23, 10.62 [17:27:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.75, 12.45, 10.65 [17:29:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.24, 11.03, 10.61 [17:29:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.37, 11.88, 10.67 [17:31:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.72, 9.97, 10.13 [17:32:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.62, 3.47, 3.94 [17:32:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 2.09, 7.54, 9.39 [17:33:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.02, 2.83, 3.85 [17:33:33] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1075 bytes in 7.381 second response time [17:34:00] I'm getting a lot of "Connection refused (db101)" errors [17:34:10] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1075 bytes in 0.167 second response time [17:34:14] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 78.74, 71.08, 32.29 [17:34:28] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1075 bytes in 0.235 second response time [17:34:30] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1075 bytes in 0.113 second response time [17:34:35] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1075 bytes in 0.159 second response time [17:34:49] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1075 bytes in 0.599 second response time [17:34:51] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1075 bytes in 0.128 second response time [17:34:58] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1334 bytes in 0.777 second response time [17:34:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.09, 2.68, 3.82 [17:35:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 3.07, 7.97, 11.28 [17:35:10] PROBLEM - db101 MariaDB on db101 is CRITICAL: Can't connect to MySQL server on 'db101.miraheze.org' (115) [17:35:17] JohnLewis, paladox, Reception123: ^ [17:35:25] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw101 mw102 mw112 [17:35:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 4.07, 9.86, 11.71 [17:35:31] Restarting [17:35:33] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw111 mw112 mw122 [17:35:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw111 mw122 [17:35:53] isn't this now the 2/3 OOM since paladox increased InnoDB buffer pool? [17:36:19] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 4 backends are down. mw101 mw102 mw111 mw121 [17:36:26] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.446 second response time [17:36:30] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.385 second response time [17:36:32] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 0.376 second response time [17:36:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 2.47, 6.90, 10.91 [17:36:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.06, 2.53, 3.40 [17:36:48] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.437 second response time [17:36:51] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.650 second response time [17:36:54] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 32403 bytes in 0.715 second response time [17:37:06] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 5.77, 3.98, 2.59 [17:37:08] RECOVERY - db101 MariaDB on db101 is OK: Uptime: 107 Threads: 120 Questions: 64156 Slow queries: 3975 Opens: 3578 Open tables: 3572 Queries per second avg: 599.588 [17:37:25] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [17:37:28] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.408 second response time [17:37:33] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [17:37:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [17:38:19] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [17:38:19] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.410 second response time [17:39:06] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.87, 3.94, 2.75 [17:39:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.27, 2.48, 3.33 [17:40:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.02, 3.42 [17:40:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.96, 3.36, 3.65 [17:41:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.08, 9.59, 10.78 [17:41:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.42, 1.67, 1.97 [17:42:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.06, 3.05, 3.39 [17:42:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.79, 3.05, 3.50 [17:43:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.30, 9.01, 10.43 [17:43:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.27, 3.47 [17:43:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.99, 8.35, 10.04 [17:44:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.42, 8.47, 10.09 [17:44:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 3.56, 3.63 [17:45:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.33, 3.14, 3.39 [17:45:57] !log increase glust101 disk by 200g [17:46:03] typo! [17:46:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:46:30] !log cancelled will do later [17:46:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:47:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.45, 8.93, 10.08 [17:47:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.47, 1.82, 1.91 [17:48:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.18, 10.76, 10.66 [17:48:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.36, 3.52 [17:50:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.08, 9.62, 10.25 [17:51:49] !log sudo -u www-data php /srv/mediawiki/w/extensions/Cargo/maintenance/cargoRecreateData.php --wiki=solarawiki --table releases --replacement [17:51:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:52:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.64, 2.92, 3.29 [17:53:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.04, 3.83, 3.54 [17:55:54] [02dns] 07Universal-Omega opened pull request 03#300: Add mazakwiki.org zone - 13https://github.com/miraheze/dns/pull/300 [17:55:54] [url] Page not found · GitHub · GitHub | github.com [17:55:58] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.52, 1.88, 1.97 [17:56:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.27, 4.03, 3.56 [17:56:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.94, 3.94, 3.62 [17:58:10] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/13bea569fcd0 [17:58:11] [02miraheze/ssl] 07Universal-Omega 0313bea56 - Add wiki.puucraft.net cert [17:58:13] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [17:58:13] [url] Page not found · GitHub · GitHub | github.com [17:58:28] [02ssl] 07Universal-Omega opened pull request 03#573: Add multiple certs - 13https://github.com/miraheze/ssl/pull/573 [17:58:28] [url] Page not found · GitHub · GitHub | github.com [17:58:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.93, 12.76, 11.40 [17:59:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.40, 1.83, 1.91 [18:00:33] [02dns] 07Universal-Omega opened pull request 03#301: Add metroidpedia.com zone - 13https://github.com/miraheze/dns/pull/301 [18:00:33] [url] Page not found · GitHub · GitHub | github.com [18:01:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 15.74, 11.57, 10.05 [18:01:32] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.84, 2.09, 1.68 [18:02:01] [02dns] 07Universal-Omega opened pull request 03#302: Add ncuwiki.com zone - 13https://github.com/miraheze/dns/pull/302 [18:02:01] [url] Page not found · GitHub · GitHub | github.com [18:03:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.65, 10.84, 9.99 [18:03:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.33, 1.83, 1.63 [18:03:33] [02dns] 07Universal-Omega opened pull request 03#303: Add chakra.moe zone - 13https://github.com/miraheze/dns/pull/303 [18:03:34] ... [18:03:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.90, 9.83, 8.47 [18:04:38] [02dns] 07Universal-Omega opened pull request 03#304: Add smafallfest.org zone - 13https://github.com/miraheze/dns/pull/304 [18:04:39] ... [18:05:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.01, 10.40, 9.96 [18:05:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.32, 2.59, 1.92 [18:05:38] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.88, 4.33, 7.61 [18:05:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.38, 9.57, 8.54 [18:07:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.51, 9.87, 9.82 [18:08:39] RECOVERY - cp30 APT on cp30 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:08:43] RECOVERY - db121 APT on db121 is OK: APT OK: 21 packages available for upgrade (0 critical updates). [18:08:46] RECOVERY - cloud12 APT on cloud12 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [18:08:59] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:09:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 15.60, 12.10, 10.70 [18:09:08] RECOVERY - cloud11 APT on cloud11 is OK: APT OK: 49 packages available for upgrade (0 critical updates). [18:09:16] RECOVERY - mw102 APT on mw102 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:09:22] RECOVERY - cp20 APT on cp20 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:09:23] RECOVERY - ns2 APT on ns2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:09:24] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.19, 1.82, 1.75 [18:09:26] RECOVERY - mw101 APT on mw101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:09:32] RECOVERY - cloud10 APT on cloud10 is OK: APT OK: 49 packages available for upgrade (0 critical updates). [18:09:33] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.30, 4.08, 6.75 [18:09:40] RECOVERY - cp21 APT on cp21 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:09:41] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:09:44] RECOVERY - prometheus101 APT on prometheus101 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [18:09:52] RECOVERY - db101 APT on db101 is OK: APT OK: 21 packages available for upgrade (0 critical updates). [18:09:52] RECOVERY - cp31 APT on cp31 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:09:55] [02ssl] 07Universal-Omega edited pull request 03#573: Add wiki.puucraft.net cert - 13https://github.com/miraheze/ssl/pull/573 [18:09:55] [url] Page not found · GitHub · GitHub | github.com [18:10:04] RECOVERY - db112 APT on db112 is OK: APT OK: 21 packages available for upgrade (0 critical updates). [18:10:08] RECOVERY - mw112 APT on mw112 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:10:23] RECOVERY - mem101 APT on mem101 is OK: APT OK: 21 packages available for upgrade (0 critical updates). [18:10:32] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/13bea569fcd0...0c3a3cda5935 [18:10:32] [url] Comparing 13bea569fcd0...0c3a3cda5935 · miraheze/ssl · GitHub | github.com [18:10:33] [02miraheze/ssl] 07Universal-Omega 030c3a3cd - Update certs.yaml [18:10:35] [02ssl] 07Universal-Omega synchronize pull request 03#573: Add wiki.puucraft.net cert - 13https://github.com/miraheze/ssl/pull/573 [18:10:35] ... [18:10:40] RECOVERY - mem121 APT on mem121 is OK: APT OK: 21 packages available for upgrade (0 critical updates). [18:10:42] !log applied critical apt updates [18:10:47] RECOVERY - db111 APT on db111 is OK: APT OK: 21 packages available for upgrade (0 critical updates). [18:10:48] RECOVERY - ldap111 APT on ldap111 is OK: APT OK: 21 packages available for upgrade (0 critical updates). [18:10:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:11:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.76, 11.88, 10.79 [18:11:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.49, 1.66, 1.70 [18:11:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.95, 10.94, 10.36 [18:11:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.08, 10.67, 9.64 [18:12:21] [02ssl] 07Universal-Omega closed pull request 03#573: Add wiki.puucraft.net cert - 13https://github.com/miraheze/ssl/pull/573 [18:12:21] [url] Page not found · GitHub · GitHub | github.com [18:12:22] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [18:12:24] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/b6c88174fb03...31f5fbda972b [18:12:24] [url] Comparing b6c88174fb03...31f5fbda972b · miraheze/ssl · GitHub | github.com [18:12:25] [02miraheze/ssl] 07Universal-Omega 0331f5fbd - Add wiki.puucraft.net cert (#573) [18:12:27] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [18:12:27] [url] Page not found · GitHub · GitHub | github.com [18:13:31] I'm Guest20 [18:13:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.40, 10.20, 9.59 [18:14:27] Can someone merge all of https://github.com/miraheze/dns/pulls if able? [18:14:27] [url] Pull requests · miraheze/dns · GitHub | github.com [18:17:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.24, 8.59, 9.73 [18:17:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.59, 9.41, 9.97 [18:21:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.88, 10.40, 9.79 [18:22:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.00, 2.02, 1.80 [18:22:31] I would also appreciate if https://github.com/miraheze/puppet/pull/2699 could be reviewed [18:22:32] [url] ssl: extend revoke option to also remove key from git by MacFan4000 · Pull Request #2699 · miraheze/puppet · GitHub | github.com [18:25:34] [02mediawiki] 07Universal-Omega unlabeled pull request 03#8313: Bump skins/Cosmos from `48e1b8a` to `c58ac13` - 13https://github.com/miraheze/mediawiki/pull/8313 [18:25:35] [url] Page not found · GitHub · GitHub | github.com [18:25:43] [02mediawiki] 07Universal-Omega commented on pull request 03#8313: Bump skins/Cosmos from `48e1b8a` to `c58ac13` - 13https://github.com/miraheze/mediawiki/pull/8313#issuecomment-1198493007 [18:25:43] [url] Bump skins/Cosmos from `48e1b8a` to `c58ac13` by dependabot[bot] · Pull Request #8313 · miraheze/mediawiki · GitHub | github.com [18:25:46] [02mediawiki] 07dependabot[bot] edited pull request 03#8313: Bump skins/Cosmos from `48e1b8a` to `c58ac13` - 13https://github.com/miraheze/mediawiki/pull/8313 [18:25:46] [url] Page not found · GitHub · GitHub | github.com [18:26:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.75, 1.60, 1.70 [18:26:38] [02mediawiki] 07dependabot[bot] edited pull request 03#8313: Bump skins/Cosmos from `48e1b8a` to `c58ac13` - 13https://github.com/miraheze/mediawiki/pull/8313 [18:26:38] [url] Page not found · GitHub · GitHub | github.com [18:26:46] [02mediawiki] 07dependabot[bot] opened pull request 03#8314: Bump skins/Cosmos from `48e1b8a` to `6b1ec96` - 13https://github.com/miraheze/mediawiki/pull/8314 [18:26:47] [url] Page not found · GitHub · GitHub | github.com [18:26:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Cosmos-6b1ec96 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ca7f3d835f63 [18:26:49] [02miraheze/mediawiki] 07dependabot[bot] 03ca7f3d8 - Bump skins/Cosmos from `48e1b8a` to `6b1ec96` [18:26:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Cosmos-6b1ec96 - 13https://github.com/miraheze/mediawiki [18:26:51] ... [18:26:52] [02mediawiki] 07dependabot[bot] commented on pull request 03#8313: Bump skins/Cosmos from `48e1b8a` to `c58ac13` - 13https://github.com/miraheze/mediawiki/pull/8313#issuecomment-1198493974 [18:26:52] [url] Bump skins/Cosmos from `48e1b8a` to `c58ac13` by dependabot[bot] · Pull Request #8313 · miraheze/mediawiki · GitHub | github.com [18:26:54] [02mediawiki] 07dependabot[bot] closed pull request 03#8313: Bump skins/Cosmos from `48e1b8a` to `c58ac13` - 13https://github.com/miraheze/mediawiki/pull/8313 [18:26:54] [url] Page not found · GitHub · GitHub | github.com [18:26:55] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Cosmos-c58ac13 - 13https://github.com/miraheze/mediawiki [18:26:55] ... [18:26:57] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Cosmos-c58ac13 [18:26:58] [02mediawiki] 07dependabot[bot] assigned pull request 03#8314: Bump skins/Cosmos from `48e1b8a` to `6b1ec96` - 13https://github.com/miraheze/mediawiki/pull/8314 [18:26:58] [url] Page not found · GitHub · GitHub | github.com [18:27:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#8314: Bump skins/Cosmos from `48e1b8a` to `6b1ec96` - 13https://github.com/miraheze/mediawiki/pull/8314 [18:27:19] ... [18:27:20] [02mediawiki] 07github-actions[bot] labeled pull request 03#8314: Bump skins/Cosmos from `48e1b8a` to `6b1ec96` - 13https://github.com/miraheze/mediawiki/pull/8314 [18:27:20] [url] Page not found · GitHub · GitHub | github.com [18:27:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#8314: Bump skins/Cosmos from `48e1b8a` to `6b1ec96` - 13https://github.com/miraheze/mediawiki/pull/8314 [18:27:22] [url] Page not found · GitHub · GitHub | github.com [18:27:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.27, 10.25, 8.87 [18:28:16] [02dns] 07JohnFLewis closed pull request 03#304: Add smafallfest.org zone - 13https://github.com/miraheze/dns/pull/304 [18:28:17] [url] Page not found · GitHub · GitHub | github.com [18:28:18] [02miraheze/dns] 07JohnFLewis pushed 032 commits to 03master [+2/-0/±0] 13https://github.com/miraheze/dns/compare/adc2c973996b...7025f8cae5ff [18:28:18] [url] Comparing adc2c973996b...7025f8cae5ff · miraheze/dns · GitHub | github.com [18:28:19] [02miraheze/dns] 07Universal-Omega 030ff23be - Add smafallfest.org zone [18:28:21] [02miraheze/dns] 07JohnFLewis 037025f8c - Merge pull request #304 from Universal-Omega/patch-34 [18:28:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.23, 11.34, 9.73 [18:28:51] [02dns] 07JohnFLewis closed pull request 03#303: Add chakra.moe zone - 13https://github.com/miraheze/dns/pull/303 [18:28:51] ... [18:28:52] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/7025f8cae5ff...79e0e35654a3 [18:28:53] [url] Comparing 7025f8cae5ff...79e0e35654a3 · miraheze/dns · GitHub | github.com [18:28:54] [02miraheze/dns] 07Universal-Omega 0379e0e35 - Add chakra.moe zone (#303) [18:29:14] [02dns] 07JohnFLewis closed pull request 03#302: Add ncuwiki.com zone - 13https://github.com/miraheze/dns/pull/302 [18:29:14] [url] Page not found · GitHub · GitHub | github.com [18:29:15] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/79e0e35654a3...5ebaf348bc25 [18:29:16] [url] Comparing 79e0e35654a3...5ebaf348bc25 · miraheze/dns · GitHub | github.com [18:29:17] [02miraheze/dns] 07Universal-Omega 035ebaf34 - Add ncuwiki.com zone (#302) [18:29:28] [02dns] 07JohnFLewis closed pull request 03#301: Add metroidpedia.com zone - 13https://github.com/miraheze/dns/pull/301 [18:29:28] [url] Page not found · GitHub · GitHub | github.com [18:29:29] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/5ebaf348bc25...928c9871e1f2 [18:29:30] [url] Comparing 5ebaf348bc25...928c9871e1f2 · miraheze/dns · GitHub | github.com [18:29:31] [02miraheze/dns] 07Universal-Omega 03928c987 - Add metroidpedia.com zone (#301) [18:29:54] [02dns] 07JohnFLewis closed pull request 03#300: Add mazakwiki.org zone - 13https://github.com/miraheze/dns/pull/300 [18:29:54] [url] Page not found · GitHub · GitHub | github.com [18:29:55] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/928c9871e1f2...65ce0e1c0f6a [18:29:56] [url] Comparing 928c9871e1f2...65ce0e1c0f6a · miraheze/dns · GitHub | github.com [18:29:57] [02miraheze/dns] 07Universal-Omega 0365ce0e1 - Add mazakwiki.org zone (#300) [18:31:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.06, 10.46, 10.06 [18:31:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.83, 12.61, 10.94 [18:31:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.39, 10.26, 9.58 [18:31:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.34, 9.81, 9.04 [18:31:57] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.18, 9.35, 9.80 [18:33:19] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.17, 2.40, 1.76 [18:33:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.46, 9.97, 9.55 [18:34:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.03, 10.75, 10.25 [18:35:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.10, 11.20, 10.78 [18:35:13] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.93, 1.97, 1.67 [18:35:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.30, 10.43, 9.47 [18:36:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.66, 11.35, 10.55 [18:37:08] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.63, 1.56, 1.56 [18:38:36] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.68, 11.37, 10.63 [18:39:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.91, 2.24, 1.82 [18:39:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.01, 11.46, 10.07 [18:41:06] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.03, 11.82, 10.33 [18:41:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.11, 10.47, 9.89 [18:41:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.60, 11.19, 10.25 [18:42:53] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.33, 11.94, 11.17 [18:43:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.68, 11.51, 10.38 [18:43:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.91, 1.95, 1.80 [18:43:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.63, 11.01, 10.31 [18:45:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.08, 2.38, 1.97 [18:45:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.55, 10.13, 9.89 [18:47:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.26, 1.94, 1.86 [18:47:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.17, 11.45, 10.54 [18:49:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.10, 1.93, 1.86 [18:49:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.34, 10.24, 10.11 [18:49:10] RECOVERY - miraheze.cf - reverse DNS on sslhost is OK: SSL OK - miraheze.cf reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [18:51:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.71, 9.71, 9.92 [18:51:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.31, 1.70, 1.79 [18:53:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.48, 1.69 [18:54:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.34, 11.29, 11.61 [18:54:49] !log increase gluster101 disk by 200g [18:55:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:55:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.14, 11.93, 11.39 [18:56:33] !log increase gluster111 disk by 200g [18:56:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.51, 11.48, 11.59 [18:56:48] PROBLEM - gluster101 Gluster Disk Space on gluster101 is WARNING: DISK WARNING - free space: /mnt/mediawiki-static 177961 MB (6% inode=79%); [18:56:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:58] RECOVERY - gluster101 Disk Space on gluster101 is OK: DISK OK - free space: / 177957 MB (17% inode=82%); [18:57:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.50, 12.12, 11.52 [18:58:11] RECOVERY - gluster111 Disk Space on gluster111 is OK: DISK OK - free space: / 175766 MB (17% inode=82%); [18:58:18] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 10.04, 9.88, 10.17 [18:58:30] RECOVERY - gluster121 Disk Space on gluster121 is OK: DISK OK - free space: / 176173 MB (17% inode=82%); [18:58:31] !log increase gluster121 disk by 200g [18:58:44] RECOVERY - gluster101 Gluster Disk Space on gluster101 is OK: DISK OK - free space: /mnt/mediawiki-static 529842 MB (17% inode=82%); [18:58:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.01, 10.78, 10.08 [19:00:59] !log root@gluster101:/home/paladox# gluster volume rebalance static fix-layout start [19:01:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:01:20] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.77, 1.89, 1.50 [19:02:18] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.68, 11.21, 10.63 [19:02:19] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.27, 2.01, 1.65 [19:03:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.86, 10.89, 11.35 [19:03:20] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.31, 1.69, 1.47 [19:04:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.43, 10.50, 10.47 [19:04:19] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.65, 1.55, 1.53 [19:04:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.97, 10.66, 11.69 [19:04:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.43, 11.28, 10.43 [19:07:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.89, 10.81, 10.38 [19:07:53] RECOVERY - miraheze.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.cf' will expire on Sat 20 Aug 2022 20:31:27 GMT +0000. [19:08:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.91, 10.53, 10.47 [19:09:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.69, 10.15, 10.17 [19:10:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.78, 11.99, 11.62 [19:10:40] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.75, 12.79, 11.28 [19:17:56] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.59, 11.01, 11.11 [19:18:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.52, 10.46, 10.36 [19:20:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.91, 11.02, 11.34 [19:21:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.68, 11.48, 11.34 [19:23:11] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/65ce0e1c0f6a...691e0b2dea11 [19:23:11] [url] Comparing 65ce0e1c0f6a...691e0b2dea11 · miraheze/dns · GitHub | github.com [19:23:12] [02miraheze/dns] 07paladox 03691e0b2 - Add new acme challenge for orain.org [19:24:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.23, 12.16, 11.72 [19:24:51] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/31f5fbda972b...bedb16844755 [19:24:51] [url] Comparing 31f5fbda972b...bedb16844755 · miraheze/ssl · GitHub | github.com [19:24:52] [02miraheze/ssl] 07paladox 03bedb168 - Update orain.org.crt [19:24:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 17.02, 12.08, 11.12 [19:25:56] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.00, 11.82, 11.42 [19:26:00] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/cdaa4128559a [19:26:02] [02miraheze/ssl] 07Universal-Omega 03cdaa412 - Add mazakwiki.org cert [19:26:03] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [19:26:04] [url] Page not found · GitHub · GitHub | github.com [19:26:05] [02ssl] 07Universal-Omega opened pull request 03#574: Add mazakwiki.org cert - 13https://github.com/miraheze/ssl/pull/574 [19:26:05] [url] Page not found · GitHub · GitHub | github.com [19:26:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.55, 11.51, 10.69 [19:26:42] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cdaa4128559a...f011600eb821 [19:26:42] [url] Comparing cdaa4128559a...f011600eb821 · miraheze/ssl · GitHub | github.com [19:26:43] [02miraheze/ssl] 07Universal-Omega 03f011600 - Update certs.yaml [19:26:45] [02ssl] 07Universal-Omega synchronize pull request 03#574: Add mazakwiki.org cert - 13https://github.com/miraheze/ssl/pull/574 [19:26:45] [url] Page not found · GitHub · GitHub | github.com [19:27:07] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=mundogaturrowiki --no-updates --username-prefix=wikia:mundogaturro /mnt/mediawiki-static/metawiki/ImportDump/mundogaturrowiki-20220728165608.xml (START) [19:27:25] RECOVERY - orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [19:27:31] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [19:27:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:28:07] [02ssl] 07Universal-Omega closed pull request 03#574: Add mazakwiki.org cert - 13https://github.com/miraheze/ssl/pull/574 [19:28:08] [url] Page not found · GitHub · GitHub | github.com [19:28:09] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/bedb16844755...cc93d5ad5356 [19:28:09] [url] Comparing bedb16844755...cc93d5ad5356 · miraheze/ssl · GitHub | github.com [19:28:10] [02miraheze/ssl] 07Universal-Omega 03cc93d5a - Add mazakwiki.org cert (#574) [19:28:12] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:28:13] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [19:28:14] [url] Page not found · GitHub · GitHub | github.com [19:28:17] !log update java on puppet111 [19:28:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:29:21] RECOVERY - isv.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [19:30:19] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/ed307209e1dd [19:30:20] [02miraheze/ssl] 07Universal-Omega 03ed30720 - Add metroidpedia.com cert [19:30:22] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [19:30:22] [url] Page not found · GitHub · GitHub | github.com [19:30:23] [02ssl] 07Universal-Omega opened pull request 03#575: Add metroidpedia.com cert - 13https://github.com/miraheze/ssl/pull/575 [19:30:24] ... [19:30:47] !log reboot mw111 - load at 40+ [19:30:56] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.37, 1.90, 1.46 [19:31:13] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ed307209e1dd...d6a467914937 [19:31:13] [url] Comparing ed307209e1dd...d6a467914937 · miraheze/ssl · GitHub | github.com [19:31:14] [02miraheze/ssl] 07Universal-Omega 03d6a4679 - Update certs.yaml [19:31:16] [02ssl] 07Universal-Omega synchronize pull request 03#575: Add metroidpedia.com cert - 13https://github.com/miraheze/ssl/pull/575 [19:31:16] [url] Page not found · GitHub · GitHub | github.com [19:31:18] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [19:31:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 1.57, 0.34, 0.11 [19:31:29] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:31:47] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.247 second response time [19:31:57] RECOVERY - puppet111 APT on puppet111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:32:06] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:17] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [19:32:19] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:25] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:32:27] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d6a467914937...6d947b9e8c59 [19:32:28] [url] Comparing d6a467914937...6d947b9e8c59 · miraheze/ssl · GitHub | github.com [19:32:29] [02miraheze/ssl] 07Universal-Omega 036d947b9 - Update redirects.yaml [19:32:30] [02ssl] 07Universal-Omega synchronize pull request 03#575: Add metroidpedia.com cert - 13https://github.com/miraheze/ssl/pull/575 [19:32:31] [url] Page not found · GitHub · GitHub | github.com [19:32:31] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:14] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [19:33:21] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw111 [19:33:54] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/ssl/compare/cc93d5ad5356...5b1b9b86293a [19:33:54] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 4 backends are down. mw102 mw111 mw112 mw121 [19:33:55] [url] Comparing cc93d5ad5356...5b1b9b86293a · miraheze/ssl · GitHub | github.com [19:33:56] [02miraheze/ssl] 07Universal-Omega 035b1b9b8 - Add metroidpedia.com cert (#575) [19:33:57] [02ssl] 07Universal-Omega closed pull request 03#575: Add metroidpedia.com cert - 13https://github.com/miraheze/ssl/pull/575 [19:33:58] [url] Page not found · GitHub · GitHub | github.com [19:33:59] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [19:33:59] [url] Page not found · GitHub · GitHub | github.com [19:34:00] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:34:25] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:42] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [19:34:55] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:00] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:04] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:27] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.720 second response time [19:35:30] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:43] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:47] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:36:01] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.053 second response time [19:36:03] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:36:13] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.984 second response time [19:36:13] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 4 backends are down. mw101 mw102 mw112 mw122 [19:36:14] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.128 second response time [19:36:21] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.680 second response time [19:36:23] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.543 second response time [19:36:28] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.532 second response time [19:36:30] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [19:36:42] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [19:36:48] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.020 second response time [19:36:55] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [19:36:56] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.80, 1.90, 1.81 [19:37:00] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.314 second response time [19:37:15] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [19:37:21] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [19:37:24] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.271 second response time [19:37:36] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:37:46] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [19:38:44] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/5d74cd160f1b [19:38:45] [02miraheze/ssl] 07Universal-Omega 035d74cd1 - Add ncuwiki.com cert [19:38:47] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [19:38:47] [url] Page not found · GitHub · GitHub | github.com [19:38:48] [02ssl] 07Universal-Omega opened pull request 03#576: Add ncuwiki.com cert - 13https://github.com/miraheze/ssl/pull/576 [19:38:49] ... [19:38:57] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.46, 1.39, 1.63 [19:39:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.48, 8.00, 3.88 [19:39:09] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [19:39:30] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:39:34] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.022 second response time [19:39:44] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.332 second response time [19:39:46] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5d74cd160f1b...1c0eba91915b [19:39:47] [url] Comparing 5d74cd160f1b...1c0eba91915b · miraheze/ssl · GitHub | github.com [19:39:47] [02miraheze/ssl] 07Universal-Omega 031c0eba9 - Update certs.yaml [19:39:49] [02ssl] 07Universal-Omega synchronize pull request 03#576: Add ncuwiki.com cert - 13https://github.com/miraheze/ssl/pull/576 [19:39:49] [url] Page not found · GitHub · GitHub | github.com [19:39:50] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:40:02] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.312 second response time [19:40:04] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.582 second response time [19:40:04] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.309 second response time [19:40:05] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:40:11] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [19:40:24] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:40:37] [02ssl] 07Universal-Omega closed pull request 03#576: Add ncuwiki.com cert - 13https://github.com/miraheze/ssl/pull/576 [19:40:37] ... [19:40:38] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/5b1b9b86293a...98fcccff0e1e [19:40:39] [url] Comparing 5b1b9b86293a...98fcccff0e1e · miraheze/ssl · GitHub | github.com [19:40:40] [02miraheze/ssl] 07Universal-Omega 0398fcccf - Add ncuwiki.com cert (#576) [19:40:41] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:40:43] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [19:40:43] [url] Page not found · GitHub · GitHub | github.com [19:40:47] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:41:21] PROBLEM - db111 Puppet on db111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:41:51] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [19:42:11] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.464 second response time [19:42:14] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/84c6916ca805 [19:42:15] [02miraheze/ssl] 07Universal-Omega 0384c6916 - Add chakra.moe cert [19:42:17] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [19:42:17] [url] Page not found · GitHub · GitHub | github.com [19:42:18] [02ssl] 07Universal-Omega opened pull request 03#577: Add chakra.moe cert - 13https://github.com/miraheze/ssl/pull/577 [19:42:19] [url] Page not found · GitHub · GitHub | github.com [19:42:41] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/84c6916ca805...39eba4ad0bcf [19:42:42] [url] Comparing 84c6916ca805...39eba4ad0bcf · miraheze/ssl · GitHub | github.com [19:42:43] [02miraheze/ssl] 07Universal-Omega 0339eba4a - Update certs.yaml [19:42:44] [02ssl] 07Universal-Omega synchronize pull request 03#577: Add chakra.moe cert - 13https://github.com/miraheze/ssl/pull/577 [19:42:45] [url] Page not found · GitHub · GitHub | github.com [19:42:50] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 7.222 second response time [19:42:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.32, 10.87, 5.91 [19:43:31] [02ssl] 07Universal-Omega closed pull request 03#577: Add chakra.moe cert - 13https://github.com/miraheze/ssl/pull/577 [19:43:31] ... [19:43:32] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:43:34] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [19:43:34] [url] Page not found · GitHub · GitHub | github.com [19:43:35] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/98fcccff0e1e...52c7e44ef597 [19:43:36] [url] Comparing 98fcccff0e1e...52c7e44ef597 · miraheze/ssl · GitHub | github.com [19:43:37] [02miraheze/ssl] 07Universal-Omega 0352c7e44 - Add chakra.moe cert (#577) [19:44:53] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.89, 9.48, 6.00 [19:44:57] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/e57aa2bddc22 [19:44:59] [02miraheze/ssl] 07Universal-Omega 03e57aa2b - Add smafallfest.org cert [19:45:00] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [19:45:01] ... [19:45:02] [02ssl] 07Universal-Omega opened pull request 03#578: Add smafallfest.org cert - 13https://github.com/miraheze/ssl/pull/578 [19:45:02] [url] Page not found · GitHub · GitHub | github.com [19:45:24] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:27] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e57aa2bddc22...dbf7e73d4e08 [19:45:28] [url] Comparing e57aa2bddc22...dbf7e73d4e08 · miraheze/ssl · GitHub | github.com [19:45:29] [02miraheze/ssl] 07Universal-Omega 03dbf7e73 - Update certs.yaml [19:45:30] [02ssl] 07Universal-Omega synchronize pull request 03#578: Add smafallfest.org cert - 13https://github.com/miraheze/ssl/pull/578 [19:45:31] [url] Page not found · GitHub · GitHub | github.com [19:45:54] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 4.40, 8.27, 11.55 [19:45:58] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:46:06] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.496 second response time [19:46:14] [02ssl] 07Universal-Omega closed pull request 03#578: Add smafallfest.org cert - 13https://github.com/miraheze/ssl/pull/578 [19:46:14] [url] Page not found · GitHub · GitHub | github.com [19:46:15] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/52c7e44ef597...d33d30a1ffa0 [19:46:16] [url] Comparing 52c7e44ef597...d33d30a1ffa0 · miraheze/ssl · GitHub | github.com [19:46:17] [02miraheze/ssl] 07Universal-Omega 03d33d30a - Add smafallfest.org cert (#578) [19:46:18] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:46:20] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [19:46:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.10, 8.25, 11.62 [19:46:20] [url] Page not found · GitHub · GitHub | github.com [19:46:37] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.284 second response time [19:47:19] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.576 second response time [19:47:48] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.634 second response time [19:47:57] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.027 second response time [19:48:02] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.595 second response time [19:50:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.21, 7.92, 11.84 [19:50:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.32, 8.00, 11.92 [19:51:24] RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [19:52:16] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.93, 6.65, 9.81 [19:52:37] [02puppet] 07MacFan4000 commented on pull request 03#2699: ssl: extend revoke option to also remove key from git - 13https://github.com/miraheze/puppet/pull/2699#issuecomment-1198566537 [19:52:37] [url] ssl: extend revoke option to also remove key from git by MacFan4000 · Pull Request #2699 · miraheze/puppet · GitHub | github.com [19:52:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.14, 7.82, 11.53 [19:54:40] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 26.02, 13.77, 13.22 [19:55:56] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.71, 10.98, 11.09 [19:56:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.73, 10.49, 11.58 [19:56:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 22.07, 13.51, 11.73 [19:56:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 21.27, 17.76, 11.12 [19:57:37] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [20:00:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.42, 11.35, 11.24 [20:00:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.62, 11.39, 11.37 [20:00:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.37, 9.35, 10.40 [20:01:04] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:36] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [20:02:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.85, 10.66, 10.72 [20:03:01] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.322 second response time [20:04:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 30.73, 18.01, 13.55 [20:04:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.61, 11.32, 11.40 [20:04:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.67, 9.86, 10.43 [20:05:04] [02mediawiki] 07Universal-Omega commented on pull request 03#8314: Bump skins/Cosmos from `48e1b8a` to `6b1ec96` - 13https://github.com/miraheze/mediawiki/pull/8314#issuecomment-1198575970 [20:05:04] [url] Bump skins/Cosmos from `48e1b8a` to `6b1ec96` by dependabot[bot] · Pull Request #8314 · miraheze/mediawiki · GitHub | github.com [20:05:06] [02mediawiki] 07dependabot[bot] edited pull request 03#8314: Bump skins/Cosmos from `48e1b8a` to `6b1ec96` - 13https://github.com/miraheze/mediawiki/pull/8314 [20:05:07] [url] Page not found · GitHub · GitHub | github.com [20:05:37] [02mediawiki] 07Universal-Omega commented on pull request 03#8132: Bump extensions/SimpleBlogPage from `a2825df` to `c61c3cb` - 13https://github.com/miraheze/mediawiki/pull/8132#issuecomment-1198576402 [20:05:38] [url] Bump extensions/SimpleBlogPage from `a2825df` to `c61c3cb` by dependabot[bot] · Pull Request #8132 · miraheze/mediawiki · GitHub | github.com [20:05:40] [02mediawiki] 07dependabot[bot] edited pull request 03#8132: Bump extensions/SimpleBlogPage from `a2825df` to `c61c3cb` - 13https://github.com/miraheze/mediawiki/pull/8132 [20:05:41] [url] Page not found · GitHub · GitHub | github.com [20:05:55] [02mediawiki] 07dependabot[bot] edited pull request 03#8314: Bump skins/Cosmos from `48e1b8a` to `6b1ec96` - 13https://github.com/miraheze/mediawiki/pull/8314 [20:05:56] [url] Page not found · GitHub · GitHub | github.com [20:06:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Cosmos-62d897e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5a70a5f0c331 [20:06:02] [02miraheze/mediawiki] 07dependabot[bot] 035a70a5f - Bump skins/Cosmos from `48e1b8a` to `62d897e` [20:06:03] [02mediawiki] 07dependabot[bot] opened pull request 03#8315: Bump skins/Cosmos from `48e1b8a` to `62d897e` - 13https://github.com/miraheze/mediawiki/pull/8315 [20:06:04] [url] Page not found · GitHub · GitHub | github.com [20:06:05] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Cosmos-62d897e - 13https://github.com/miraheze/mediawiki [20:06:05] ... [20:06:06] [02mediawiki] 07dependabot[bot] commented on pull request 03#8314: Bump skins/Cosmos from `48e1b8a` to `6b1ec96` - 13https://github.com/miraheze/mediawiki/pull/8314#issuecomment-1198576739 [20:06:06] [url] Bump skins/Cosmos from `48e1b8a` to `6b1ec96` by dependabot[bot] · Pull Request #8314 · miraheze/mediawiki · GitHub | github.com [20:06:08] [02mediawiki] 07dependabot[bot] closed pull request 03#8314: Bump skins/Cosmos from `48e1b8a` to `6b1ec96` - 13https://github.com/miraheze/mediawiki/pull/8314 [20:06:08] [url] Page not found · GitHub · GitHub | github.com [20:06:09] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Cosmos-6b1ec96 [20:06:11] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Cosmos-6b1ec96 - 13https://github.com/miraheze/mediawiki [20:06:11] ... [20:06:12] [02mediawiki] 07dependabot[bot] assigned pull request 03#8315: Bump skins/Cosmos from `48e1b8a` to `62d897e` - 13https://github.com/miraheze/mediawiki/pull/8315 [20:06:12] [url] Page not found · GitHub · GitHub | github.com [20:06:26] [02mediawiki] 07dependabot[bot] edited pull request 03#8132: Bump extensions/SimpleBlogPage from `a2825df` to `c61c3cb` - 13https://github.com/miraheze/mediawiki/pull/8132 [20:06:27] ... [20:06:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#8315: Bump skins/Cosmos from `48e1b8a` to `62d897e` - 13https://github.com/miraheze/mediawiki/pull/8315 [20:06:30] [url] Page not found · GitHub · GitHub | github.com [20:06:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#8315: Bump skins/Cosmos from `48e1b8a` to `62d897e` - 13https://github.com/miraheze/mediawiki/pull/8315 [20:06:31] [url] Page not found · GitHub · GitHub | github.com [20:06:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#8315: Bump skins/Cosmos from `48e1b8a` to `62d897e` - 13https://github.com/miraheze/mediawiki/pull/8315 [20:06:33] [url] Page not found · GitHub · GitHub | github.com [20:06:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/SimpleBlogPage-3173559 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ddc5735c5521 [20:06:36] [02miraheze/mediawiki] 07dependabot[bot] 03ddc5735 - Bump extensions/SimpleBlogPage from `a2825df` to `3173559` [20:06:37] [02mediawiki] 07dependabot[bot] opened pull request 03#8316: Bump extensions/SimpleBlogPage from `a2825df` to `3173559` - 13https://github.com/miraheze/mediawiki/pull/8316 [20:06:38] ... [20:06:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/SimpleBlogPage-3173559 - 13https://github.com/miraheze/mediawiki [20:06:41] [02mediawiki] 07Universal-Omega labeled pull request 03#8315: Bump skins/Cosmos from `48e1b8a` to `62d897e` - 13https://github.com/miraheze/mediawiki/pull/8315 [20:06:57] [02mediawiki] 07Universal-Omega labeled pull request 03#8315: Bump skins/Cosmos from `48e1b8a` to `62d897e` - 13https://github.com/miraheze/mediawiki/pull/8315 [20:06:58] [02mediawiki] 07Universal-Omega unlabeled pull request 03#8315: Bump skins/Cosmos from `48e1b8a` to `62d897e` - 13https://github.com/miraheze/mediawiki/pull/8315 [20:07:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#8316: Bump extensions/SimpleBlogPage from `a2825df` to `3173559` - 13https://github.com/miraheze/mediawiki/pull/8316 [20:07:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#8316: Bump extensions/SimpleBlogPage from `a2825df` to `3173559` - 13https://github.com/miraheze/mediawiki/pull/8316 [20:07:15] [02mediawiki] 07Universal-Omega assigned pull request 03#8316: Bump extensions/SimpleBlogPage from `a2825df` to `3173559` - 13https://github.com/miraheze/mediawiki/pull/8316 [20:07:19] [02mediawiki] 07Universal-Omega labeled pull request 03#8316: Bump extensions/SimpleBlogPage from `a2825df` to `3173559` - 13https://github.com/miraheze/mediawiki/pull/8316 [20:07:21] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:07:24] [02mediawiki] 07Universal-Omega labeled pull request 03#8316: Bump extensions/SimpleBlogPage from `a2825df` to `3173559` - 13https://github.com/miraheze/mediawiki/pull/8316 [20:07:25] [02mediawiki] 07Universal-Omega unlabeled pull request 03#8316: Bump extensions/SimpleBlogPage from `a2825df` to `3173559` - 13https://github.com/miraheze/mediawiki/pull/8316 [20:07:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.31, 9.77, 11.46 [20:07:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 5.54, 9.11, 11.74 [20:08:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.58, 11.60, 11.90 [20:08:07] [02mediawiki] 07Universal-Omega closed pull request 03#8315: Bump skins/Cosmos from `48e1b8a` to `62d897e` - 13https://github.com/miraheze/mediawiki/pull/8315 [20:08:08] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/skins/Cosmos-62d897e [20:08:10] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/skins/Cosmos-62d897e - 13https://github.com/miraheze/mediawiki [20:08:11] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/81d6ca62902a...2710b2ab639f [20:08:12] [url] Comparing 81d6ca62902a...2710b2ab639f · miraheze/mediawiki · GitHub | github.com [20:08:13] [02miraheze/mediawiki] 07dependabot[bot] 032710b2a - Bump skins/Cosmos from `48e1b8a` to `62d897e` (#8315) [20:08:20] [02mediawiki] 07Universal-Omega closed pull request 03#8316: Bump extensions/SimpleBlogPage from `a2825df` to `3173559` - 13https://github.com/miraheze/mediawiki/pull/8316 [20:08:21] [url] Page not found · GitHub · GitHub | github.com [20:08:22] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/2710b2ab639f...a0ef8a8d7322 [20:08:23] [url] Comparing 2710b2ab639f...a0ef8a8d7322 · miraheze/mediawiki · GitHub | github.com [20:08:23] [02miraheze/mediawiki] 07dependabot[bot] 03a0ef8a8 - Bump extensions/SimpleBlogPage from `a2825df` to `3173559` (#8316) [20:08:25] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/extensions/SimpleBlogPage-3173559 [20:08:25] RECOVERY - db111 Puppet on db111 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [20:08:26] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/extensions/SimpleBlogPage-3173559 - 13https://github.com/miraheze/mediawiki [20:08:27] [url] Page not found · GitHub · GitHub | github.com [20:08:38] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [20:08:42] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:08:42] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.00, 9.20, 10.02 [20:08:51] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:09:15] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:09:18] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.373 second response time [20:10:26] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:10:35] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:11:28] !log [universalomega@mwtask111] starting deploy of {'world': True} to all [20:11:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.36, 1.83, 1.60 [20:12:25] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.39, 8.52, 10.04 [20:12:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:13:22] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.25, 7.41, 9.97 [20:13:45] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.54, 1.37, 1.46 [20:13:51] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 4.38, 6.49, 9.73 [20:14:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.59, 9.88, 12.00 [20:16:13] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 40.90, 20.25, 15.43 [20:16:43] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.564 second response time [20:16:52] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.252 second response time [20:16:52] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.974 second response time [20:16:59] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.836 second response time [20:17:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.05, 10.38, 10.57 [20:17:30] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.315 second response time [20:17:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.03, 10.37, 10.68 [20:18:14] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw102 [20:18:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.28, 13.44, 11.71 [20:18:27] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 25.46, 14.94, 11.51 [20:18:41] !log ALTER TABLE searchindex ADD FULLTEXT INDEX si_title (si_title), ADD FULLTEXT INDEX si_text (si_text); on enfswimtfgirlswiki [20:19:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.75, 10.71, 10.66 [20:19:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:19:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.08, 10.12, 10.55 [20:19:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.08, 11.15, 10.99 [20:20:12] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [20:22:35] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.59, 2.03, 1.69 [20:24:33] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.13, 1.84, 1.67 [20:25:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.27, 10.72, 10.59 [20:25:45] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.24, 11.94, 11.32 [20:26:20] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:30] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:36] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 11.87, 8.07, 4.26 [20:26:41] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:48] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:48] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.54, 11.34, 10.92 [20:26:51] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:55] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [20:27:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.98, 10.96, 11.05 [20:28:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.43, 1.65, 1.64 [20:28:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.85, 11.22, 10.91 [20:29:11] !log [universalomega@test101] starting deploy of {'world': True} to all [20:29:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.20, 11.36, 10.90 [20:29:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:29:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 15.05, 12.53, 11.60 [20:30:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.70, 11.09, 11.37 [20:30:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.42, 12.27, 11.32 [20:31:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.10, 12.56, 11.40 [20:31:17] !log [universalomega@mwtask111] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [20:31:22] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:31:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:31:29] !log [universalomega@mwtask111] starting deploy of {'world': True} to all [20:31:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:32:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.82, 11.25, 11.37 [20:32:36] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 9.084 second response time [20:32:36] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 5.188 second response time [20:32:51] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 0.548 second response time [20:32:51] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.310 second response time [20:33:09] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.320 second response time [20:35:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.77, 11.71, 11.38 [20:35:31] !log [universalomega@test101] finished deploy of {'world': True} to all - SUCCESS in 380s [20:35:41] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.78, 11.98, 11.79 [20:35:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:36:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.89, 10.67, 11.10 [20:37:40] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.13, 12.15, 11.87 [20:38:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.16, 10.06, 11.81 [20:38:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.19, 11.76, 11.68 [20:39:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.94, 13.44, 12.04 [20:43:16] !log [universalomega@mwtask111] finished deploy of {'world': True} to all - SUCCESS in 707s [20:43:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:43:58] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.83, 2.38, 4.00 [20:45:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.84, 11.69, 11.99 [20:46:48] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.65, 11.43, 11.02 [20:47:36] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.97, 12.60, 12.28 [20:47:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.67, 1.46, 3.26 [20:48:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.77, 10.36, 10.68 [20:49:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.79, 11.19, 11.78 [20:49:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.70, 11.83, 11.57 [20:50:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.75, 11.58, 11.05 [20:51:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.24, 11.37, 11.74 [20:51:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.57, 11.41, 11.07 [20:52:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.35, 11.67, 11.18 [20:53:11] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.92, 1.66, 1.50 [20:53:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.50, 10.37, 11.33 [20:53:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.66, 11.05, 11.37 [20:53:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.27, 10.17, 10.67 [20:55:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.38, 1.42 [20:59:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 21.75, 11.99, 11.39 [21:00:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 33.28, 19.62, 14.08 [21:01:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.97, 10.99, 11.08 [21:02:54] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.73, 1.50, 1.28 [21:03:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.78, 13.59, 11.74 [21:05:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.87, 10.50, 10.90 [21:06:54] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.14, 1.79, 1.44 [21:07:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.24, 11.77, 11.34 [21:07:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.51, 11.90, 11.53 [21:07:42] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [21:08:02] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:08:54] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.45, 1.66, 1.44 [21:09:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.10, 10.11, 10.79 [21:09:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.42, 12.13, 11.65 [21:09:58] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.820 second response time [21:11:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.59, 11.63, 11.53 [21:13:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.82, 11.80, 11.58 [21:13:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 16.51, 12.03, 11.32 [21:19:52] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.55, 2.81, 1.97 [21:24:13] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.48, 3.25, 2.52 [21:25:51] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.08, 1.91, 1.87 [21:26:09] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.25, 2.90, 2.48 [21:27:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.01, 10.13, 11.89 [21:27:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.17, 10.66, 11.72 [21:29:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.66, 10.79, 11.89 [21:30:02] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=mundogaturrowiki --no-updates --username-prefix=wikia:mundogaturro /mnt/mediawiki-static/metawiki/ImportDump/mundogaturrowiki-20220728165608.xml (END - exit=0) [21:30:03] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=mundogaturrowiki (START) [21:30:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:30:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:31:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.47, 9.91, 11.43 [21:33:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.01, 1.46, 1.67 [21:36:27] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.20, 2.40, 1.87 [21:37:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.49, 10.61, 11.86 [21:38:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.09, 1.94, 1.76 [21:39:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.29, 10.21, 11.84 [21:39:18] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.61, 8.59, 10.09 [21:41:17] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.01, 10.96, 11.87 [21:42:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.96, 1.53, 1.64 [21:45:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.77, 10.34, 11.43 [21:49:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 15.34, 12.20, 11.67 [21:51:12] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.95, 9.10, 10.08 [21:51:29] PROBLEM - en.omniversalis.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.omniversalis.org' expires in 15 day(s) (Sat 13 Aug 2022 21:32:36 GMT +0000). [21:53:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.69, 11.93, 11.68 [21:54:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d33d30a1ffa0...dea14591ed49 [21:54:10] [url] Comparing d33d30a1ffa0...dea14591ed49 · miraheze/ssl · GitHub | github.com [21:54:11] [02miraheze/ssl] 07MirahezeSSLBot 03dea1459 - Bot: Update SSL cert for en.omniversalis.org [21:56:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.46, 10.41, 11.89 [21:57:52] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.29, 6.60, 4.86 [21:58:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.34, 11.03, 11.93 [21:59:17] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.85, 10.57, 10.67 [21:59:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.16, 8.22, 10.19 [21:59:53] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.02, 6.61, 5.08 [22:00:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.32, 11.02, 11.81 [22:01:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.63, 10.02, 10.45 [22:01:52] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.96, 7.56, 5.62 [22:02:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.18, 12.28, 12.20 [22:03:17] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.00, 10.85, 10.69 [22:03:52] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.00, 7.14, 5.72 [22:05:52] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.74, 5.87, 5.42 [22:07:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.76, 11.89, 11.26 [22:09:17] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.51, 12.52, 11.54 [22:12:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.54, 11.38, 10.17 [22:12:32] [02mediawiki] 07Universal-Omega unlabeled pull request 03#8293: Bump extensions/MobileDetect from `ee9a00f` to `7efd1f4` - 13https://github.com/miraheze/mediawiki/pull/8293 [22:12:32] [url] Page not found · GitHub · GitHub | github.com [22:12:33] [02mediawiki] 07Universal-Omega labeled pull request 03#8293: Bump extensions/MobileDetect from `ee9a00f` to `7efd1f4` - 13https://github.com/miraheze/mediawiki/pull/8293 [22:12:33] [url] Page not found · GitHub · GitHub | github.com [22:14:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.10, 11.09, 10.21 [22:19:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.51, 10.41, 9.86 [22:21:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.60, 12.03, 10.67 [22:21:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.62, 11.72, 10.70 [22:23:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.70, 11.31, 11.96 [22:23:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.34, 11.25, 10.53 [22:23:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.82, 10.89, 10.51 [22:23:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.39, 9.87, 9.79 [22:25:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.92, 11.45, 10.64 [22:25:17] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.58, 11.44, 11.87 [22:27:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.60, 11.22, 10.65 [22:27:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.99, 11.75, 10.91 [22:28:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.53, 11.35, 12.00 [22:29:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.26, 11.14, 10.78 [22:29:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.80, 10.44, 10.06 [22:30:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 15.95, 12.75, 12.40 [22:31:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.01, 11.18, 11.63 [22:33:06] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.96, 9.30, 9.97 [22:37:29] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.34, 10.19, 10.19 [22:38:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.40, 10.67, 11.77 [22:41:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.10, 11.02, 10.68 [22:41:17] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 15.31, 11.44, 11.32 [22:41:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.89, 10.19, 10.24 [22:43:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.09, 10.17, 10.43 [22:43:15] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=mundogaturrowiki (END - exit=0) [22:43:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.57, 11.33, 11.35 [22:43:17] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=mundogaturrowiki --active --update (END - exit=0) [22:43:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:43:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:44:53] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.45, 1.98, 1.26 [22:45:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.27, 9.97, 10.16 [22:46:53] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.88, 1.58, 1.20 [22:47:12] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.87, 9.52, 10.14 [22:49:17] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.49, 8.65, 10.20 [22:50:49] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.68, 8.63, 10.03 [22:50:59] RECOVERY - en.omniversalis.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.omniversalis.org' will expire on Wed 26 Oct 2022 20:54:04 GMT +0000. [23:01:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.12, 1.58, 1.98 [23:07:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.97, 11.26, 11.93 [23:09:37] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.16, 1.26, 1.66 [23:11:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.63, 11.73, 11.91 [23:12:15] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 16.09, 11.70, 10.40 [23:12:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.13, 10.04, 9.65 [23:13:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.98, 11.58, 11.85 [23:14:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.18, 11.52, 10.51 [23:14:49] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.90, 9.31, 9.44 [23:17:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.81, 12.46, 12.09 [23:19:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.41, 2.09, 1.85 [23:20:12] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.04, 9.56, 10.02 [23:21:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.91, 9.78, 8.77 [23:23:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.83, 9.80, 8.91 [23:25:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.24, 1.77, 1.82 [23:25:37] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.94, 3.52, 2.01 [23:29:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.59, 11.51, 9.94 [23:31:37] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.16, 1.45, 1.66 [23:31:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.76, 10.16, 9.64 [23:33:37] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.53, 3.91, 2.91 [23:33:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.76, 1.55, 1.45 [23:35:53] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.62, 1.24, 1.35 [23:39:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.82, 1.73 [23:41:37] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.31, 3.06, 2.95 [23:42:02] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.55, 9.67, 9.13 [23:44:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.36, 8.76, 8.86 [23:50:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.14, 9.65, 9.22 [23:50:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.81, 2.25, 1.69 [23:52:14] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.70, 9.52, 9.22 [23:54:34] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.37, 1.90, 1.68 [23:58:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.89, 10.57, 9.73 [23:58:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.28, 1.52, 1.57