[00:01:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.12, 12.03, 11.11 [00:01:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.81, 11.80, 11.33 [00:03:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.53, 11.93, 11.22 [00:03:12] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.58, 3.92, 3.15 [00:03:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.63, 11.76, 11.39 [00:03:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.52, 4.09, 3.34 [00:05:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.36, 12.82, 11.68 [00:05:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.81, 3.13, 2.96 [00:07:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.02, 11.02, 11.10 [00:09:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.86, 3.71, 3.47 [00:09:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.75, 10.70, 10.99 [00:11:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.89, 11.24, 11.49 [00:13:29] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.33, 3.00, 3.24 [00:17:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.49, 11.57, 11.26 [00:17:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.90, 6.57, 6.06 [00:19:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.51, 11.24, 11.16 [00:23:35] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.25, 6.19, 6.09 [00:31:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.27, 10.36, 10.51 [00:33:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.93, 3.72, 3.21 [00:33:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.89, 9.74, 10.27 [00:35:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.71, 3.07, 3.03 [00:35:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.71, 11.36, 10.78 [00:39:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.75, 3.62, 3.27 [00:39:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.16, 7.63, 6.69 [00:41:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.11, 3.09, 3.12 [00:43:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.66, 11.76, 11.35 [00:45:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.14, 3.55, 3.32 [00:45:53] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.81, 3.62, 3.25 [00:47:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.16, 3.18, 3.22 [00:47:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.16, 11.79, 11.40 [00:47:47] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.94, 2.99, 3.06 [00:51:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.28, 3.85, 3.43 [00:53:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.72, 7.50, 7.58 [00:53:31] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.79, 3.61, 3.40 [00:55:21] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.08, 4.24, 3.57 [00:55:25] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.12, 3.30, 3.32 [00:55:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.87, 11.96, 11.73 [00:57:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.36, 3.61, 3.44 [00:57:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.23, 12.22, 11.84 [01:00:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.82, 11.47, 10.97 [01:01:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.35, 3.32, 3.40 [01:01:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.13, 7.10, 7.25 [01:03:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.59, 7.05, 7.23 [01:05:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.53, 3.61, 3.49 [01:05:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.87, 7.30, 7.27 [01:06:53] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.58, 6.97, 5.14 [01:07:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.63, 7.43, 7.32 [01:09:21] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.10, 3.46, 3.44 [01:10:53] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.32, 6.44, 5.47 [01:11:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.13, 3.34, 3.40 [01:11:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.52, 7.96, 7.52 [01:13:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.49, 7.79, 7.51 [01:15:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.36, 8.01, 7.65 [01:17:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.41, 7.32, 7.44 [01:21:17] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.96, 3.44, 3.07 [01:21:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.46, 8.11, 7.67 [01:23:12] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.86, 3.34, 3.08 [01:23:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.27, 7.00, 7.32 [01:24:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.89, 11.14, 11.95 [01:28:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.18, 12.18, 12.17 [01:32:19] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:33:51] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.31, 4.54, 3.54 [01:34:16] RECOVERY - cp23 APT on cp23 is OK: APT OK: 36 packages available for upgrade (0 critical updates). [01:35:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.27, 6.93, 6.99 [01:35:47] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.21, 3.93, 3.43 [01:37:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.93, 6.59, 6.86 [01:37:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.80, 3.31, 3.27 [01:39:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.73, 11.39, 11.86 [01:42:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.78, 10.76, 11.73 [01:43:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.33, 7.50, 7.18 [01:44:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.69, 11.81, 12.03 [01:45:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.95, 7.59, 7.25 [01:46:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.48, 11.44, 11.87 [01:47:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.97, 11.81, 11.80 [01:49:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.43, 8.23, 7.52 [01:51:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.03, 10.78, 11.41 [01:53:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.68, 6.82, 7.16 [01:55:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.94, 11.64, 11.59 [01:57:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.62, 5.94, 6.75 [01:58:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.12, 11.81, 11.51 [02:00:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.78, 11.59, 11.45 [02:04:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.22, 8.89, 7.70 [02:04:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.64, 12.20, 11.72 [02:05:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.78, 3.35, 3.02 [02:06:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.68, 11.75, 11.59 [02:07:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.00, 2.86, 2.88 [02:08:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.13, 12.39, 11.87 [02:10:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.84, 7.16, 7.32 [02:10:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.83, 11.04, 11.43 [02:12:05] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:12:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.47, 11.49, 11.54 [02:14:46] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=nycsubwaywiki --no-updates --username-prefix=w /home/agent/ImportDump/nycsubwaywiki-20230508233109.xml (START) [02:14:47] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=nycsubwaywiki --no-updates --username-prefix=w /home/agent/ImportDump/nycsubwaywiki-20230508233109.xml (END - exit=256) [02:14:47] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=nycsubwaywiki (START) [02:14:49] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=nycsubwaywiki (END - exit=0) [02:14:50] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=nycsubwaywiki --active --update (END - exit=0) [02:14:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:14:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:12] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.18, 3.46, 2.61 [02:15:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:33] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=nycsubwaywiki --no-updates --username-prefix=w /home/agent/ImportDump/nycsubwaywiki-20230508233109.xml (START) [02:15:34] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=nycsubwaywiki --no-updates --username-prefix=w /home/agent/ImportDump/nycsubwaywiki-20230508233109.xml (END - exit=256) [02:15:35] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=nycsubwaywiki (START) [02:15:37] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=nycsubwaywiki (END - exit=0) [02:15:38] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=nycsubwaywiki --active --update (END - exit=0) [02:15:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:16:51] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=nycsubwaywiki --no-updates --username-prefix=w /home/agent/ImportDump/nycsubwaywiki-20230508233109.xml (START) [02:16:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:18:40] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=nycsubwaywiki --no-updates --username-prefix=w /home/agent/ImportDump/nycsubwaywiki-20230508233109.xml (END - exit=0) [02:18:41] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=nycsubwaywiki (START) [02:18:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:18:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:18:53] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=nycsubwaywiki (END - exit=0) [02:18:54] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=nycsubwaywiki --active --update (END - exit=0) [02:18:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.46, 3.21, 2.73 [02:19:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.02, 7.14, 7.10 [02:21:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.36, 11.17, 11.97 [02:21:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.39, 6.99, 7.04 [02:22:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.48, 11.42, 11.85 [02:23:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.28, 11.53, 11.99 [02:29:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.72, 7.26, 7.06 [02:30:12] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.39, 4.72, 3.56 [02:31:31] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.17, 6.34, 6.74 [02:32:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.45, 11.14, 11.53 [02:37:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.86, 7.24, 6.96 [02:38:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.68, 11.34, 11.76 [02:39:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.74, 6.61, 6.76 [02:41:36] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:41:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.04, 3.73, 3.87 [02:45:03] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.zulaclub.net. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:49:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 6.67, 6.63 [02:49:37] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.05, 2.16, 3.11 [02:49:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.69, 9.88, 11.49 [02:51:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.62, 7.48, 6.90 [02:52:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.80, 9.11, 10.17 [02:53:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.90, 6.62, 6.65 [02:59:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.74, 6.96, 6.84 [03:01:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.08, 7.45, 7.04 [03:03:12] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.75, 5.92, 3.69 [03:03:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.11, 7.21, 7.03 [03:03:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.50, 7.79, 9.71 [03:04:14] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.54, 5.39, 4.05 [03:07:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.57, 6.32, 6.69 [03:08:07] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.40, 3.89, 3.75 [03:11:12] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.95, 3.55, 3.51 [03:11:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.88, 11.51, 10.50 [03:12:00] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.90, 2.56, 3.22 [03:13:12] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.97, 2.66, 3.19 [03:13:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.61, 10.93, 10.39 [03:14:39] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [03:19:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.62, 6.74, 6.49 [03:21:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.21, 7.52, 6.81 [03:21:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.99, 9.86, 10.14 [03:23:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.40, 7.23, 6.80 [03:25:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.09, 6.78, 6.68 [03:35:32] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:57:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.40, 9.43, 9.35 [03:59:48] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.52, 10.72, 9.84 [04:00:41] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=crimsoningotwiki --no-updates --username-prefix=crimsoningot /home/agent/ImportDump/crimsoningotwiki-20230507214049.xml (START) [04:00:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:01:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.16, 10.68, 9.94 [04:05:22] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [04:06:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.63, 10.36, 9.59 [04:07:08] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=crimsoningotwiki --no-updates --username-prefix=crimsoningot /home/agent/ImportDump/crimsoningotwiki-20230507214049.xml (END - exit=0) [04:07:09] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=crimsoningotwiki (START) [04:07:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:07:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:07:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.64, 9.67, 9.79 [04:10:34] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=crimsoningotwiki (END - exit=0) [04:10:35] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=crimsoningotwiki --active --update (END - exit=0) [04:10:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:10:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:11:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.14, 11.08, 10.31 [04:12:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.61, 9.93, 9.67 [04:13:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.79, 11.31, 10.49 [04:15:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.83, 9.79, 10.03 [04:17:47] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1258994937 secs [04:19:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.95, 10.74, 10.38 [04:22:53] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.16, 5.05, 2.70 [04:23:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.92, 9.59, 10.02 [04:24:53] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.39, 5.11, 3.02 [04:31:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.21, 10.38, 10.35 [04:33:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.45, 9.09, 9.88 [04:39:47] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.08462026715 secs [04:53:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.50, 7.02, 6.35 [04:55:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.66, 7.11, 6.48 [04:57:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.74, 6.38, 6.29 [05:01:03] !log reception@mon141:~$ sudo service relaybot restart [05:01:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:01:21] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.07, 3.42, 2.60 [05:03:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.91, 2.75, 2.46 [05:04:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.71, 10.52, 9.49 [05:06:54] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.79, 9.40, 9.20 [05:14:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.38, 6.48, 6.24 [05:16:04] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.52, 6.13, 6.15 [05:41:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.55, 6.89, 6.36 [05:45:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.25, 6.75, 6.44 [05:51:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.49, 10.27, 9.14 [05:55:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.81, 9.79, 9.22 [06:19:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.46, 8.68, 8.54 [06:19:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.87, 6.57, 5.97 [06:21:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.90, 8.67, 8.56 [06:21:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.39, 5.90, 5.81 [06:36:50] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1658446193 secs [06:39:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.50, 11.12, 10.08 [06:49:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.47, 9.79, 9.95 [06:57:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.60, 6.84, 6.34 [06:59:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.12, 6.03, 6.10 [07:08:50] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.06152397394 secs [07:20:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.81, 6.98, 6.28 [07:21:18] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.81, 11.77, 10.21 [07:22:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.61, 6.73, 6.27 [07:26:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.14, 6.83, 6.46 [07:26:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.39, 11.39, 10.11 [07:28:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.52, 6.43, 6.36 [07:28:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.22, 12.10, 10.51 [07:34:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.82, 11.37, 10.76 [07:36:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.11, 12.43, 11.24 [07:38:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.20, 11.84, 11.18 [07:38:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.36, 11.73, 11.75 [07:44:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.37, 6.82, 6.48 [07:46:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.39, 12.13, 11.83 [07:46:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.06, 7.26, 6.67 [07:48:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.06, 6.92, 6.60 [07:50:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.51, 11.42, 11.63 [07:50:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.13, 7.72, 6.95 [07:52:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.84, 6.82, 6.72 [07:54:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.66, 6.30, 6.55 [07:54:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.47, 8.83, 9.87 [07:58:09] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.04, 3.41, 2.74 [08:00:05] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.90, 3.05, 2.68 [08:00:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.93, 7.71, 7.03 [08:04:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.61, 9.06, 10.05 [08:04:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.92, 7.09, 6.90 [08:04:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.59, 11.60, 10.38 [08:06:10] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.44, 6.45, 6.69 [08:10:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.81, 11.19, 10.67 [08:11:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.58, 12.74, 11.31 [08:12:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.52, 11.01, 10.81 [08:13:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.21, 11.32, 10.96 [08:19:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.32, 11.16, 10.87 [08:23:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.66, 11.97, 11.36 [08:24:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.92, 6.99, 6.73 [08:26:40] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.02, 6.73, 6.67 [08:27:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.24, 12.10, 11.59 [08:28:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.60, 9.18, 10.03 [08:29:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.86, 11.22, 11.32 [08:29:51] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.40, 3.22, 1.97 [08:30:19] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:31:51] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.66, 2.57, 1.88 [08:32:16] RECOVERY - cp23 APT on cp23 is OK: APT OK: 36 packages available for upgrade (0 critical updates). [08:33:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.21, 12.07, 11.64 [08:35:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.74, 11.57, 11.51 [08:39:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.21, 11.60, 11.48 [08:47:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.58, 11.83, 11.75 [08:47:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.38, 10.39, 10.00 [08:49:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.92, 8.66, 7.48 [08:49:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.89, 9.77, 9.82 [08:51:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.82, 11.73, 11.69 [08:53:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.24, 7.89, 7.44 [08:55:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.55, 7.80, 7.44 [08:56:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.83, 6.97, 7.17 [09:00:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.80, 6.21, 6.79 [09:05:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.80, 11.41, 11.83 [09:06:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.08, 10.51, 10.06 [09:08:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.43, 11.19, 10.35 [09:09:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.81, 13.04, 12.35 [09:12:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.73, 11.86, 10.89 [09:19:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.57, 11.31, 11.88 [09:24:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.81, 9.41, 10.15 [09:25:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.31, 12.08, 11.88 [09:31:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.27, 11.62, 11.80 [09:34:16] [02dns] 07Reception123 closed pull request 03#410: Remove 13 zones - 13https://github.com/miraheze/dns/pull/410 [09:34:19] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-13/±0] 13https://github.com/miraheze/dns/compare/cc6b749b8ebc...7d3fc7befe9d [09:34:21] [02miraheze/dns] 07MacFan4000 037d3fc7b - Remove 13 zones (#410) [09:35:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.42, 11.71, 11.75 [09:36:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.52, 11.64, 10.72 [09:37:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.89, 11.87, 11.82 [09:38:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.43, 11.61, 10.82 [09:39:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.08, 12.01, 11.90 [09:43:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.39, 11.08, 11.59 [09:45:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.45, 12.37, 11.97 [09:46:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.30, 11.35, 10.79 [09:47:10] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cpt-ra.bid. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:48:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.96, 11.67, 10.98 [09:53:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.25, 11.64, 11.91 [09:56:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.57, 11.92, 11.07 [09:58:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.86, 11.08, 10.88 [10:04:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.09, 10.46, 10.54 [10:06:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.90, 10.23, 10.45 [10:07:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.71, 10.42, 10.71 [10:09:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.62, 10.88, 10.86 [10:11:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.49, 11.87, 11.23 [10:12:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.72, 11.93, 11.08 [10:13:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.55, 11.24, 11.10 [10:14:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.22, 10.70, 10.72 [10:22:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.15, 10.88, 10.70 [10:23:35] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [10:25:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.85, 10.82, 10.69 [10:42:16] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.58, 7.23, 6.65 [10:42:30] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [10:45:52] RECOVERY - with.cpt-ra.bid - reverse DNS on sslhost is OK: SSL OK - with.cpt-ra.bid reverse DNS resolves to cp22.miraheze.org - CNAME OK [10:46:09] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.60, 6.76, 6.57 [10:51:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.59, 10.82, 11.63 [10:52:41] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:03:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.20, 10.41, 11.99 [11:05:18] RECOVERY - ldap141 NTP time on ldap141 is OK: NTP OK: Offset -0.0004383921623 secs [11:09:42] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.63, 11.39, 11.17 [11:11:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.37, 12.42, 12.20 [11:11:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.93, 10.56, 10.88 [11:12:01] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [11:13:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.31, 11.61, 11.23 [11:14:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.41, 6.98, 6.60 [11:15:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.38, 11.59, 11.93 [11:15:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.22, 10.84, 10.98 [11:16:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.51, 6.40, 6.45 [11:17:39] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.03, 11.38, 11.14 [11:19:34] RECOVERY - swiftobject122 NTP time on swiftobject122 is OK: NTP OK: Offset -0.0001624822617 secs [11:19:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.37, 10.93, 11.01 [11:29:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.06, 9.76, 10.32 [11:31:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.81, 9.93, 10.33 [11:33:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.92, 10.96, 10.62 [11:35:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.75, 11.02, 10.62 [11:37:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.85, 11.82, 11.10 [11:37:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.86, 11.28, 10.80 [11:41:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.24, 12.67, 11.44 [11:43:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.47, 11.98, 11.32 [11:45:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.69, 12.02, 11.33 [11:45:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.25, 13.84, 12.07 [11:51:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.59, 7.36, 6.57 [11:53:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.37, 11.94, 11.74 [11:55:23] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.86, 11.95, 11.75 [11:55:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.66, 6.97, 6.62 [11:57:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.04, 8.36, 7.14 [11:59:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.91, 7.86, 7.12 [12:05:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.24, 7.51, 7.07 [12:07:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.53, 6.99, 6.92 [12:09:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.20, 7.72, 7.17 [12:11:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.45, 7.00, 6.96 [12:15:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.89, 6.30, 6.68 [12:33:57] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.020 second response time [12:35:05] PROBLEM - db121 PowerDNS Recursor on db121 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:35:16] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 110.31, 53.67, 22.30 [12:35:28] PROBLEM - db121 SSH on db121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:47] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.020 second response time [12:36:05] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:20] PROBLEM - db121 APT on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:36:24] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:34] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 3 backends are down. mw121 mw122 mw132 [12:36:43] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 5 backends are down. mw121 mw122 mw131 mw141 mw142 [12:36:45] PROBLEM - db121 Backups SQL on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:37:10] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:47] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 5 backends are down. mw121 mw122 mw132 mw141 mw142 [12:38:18] PROBLEM - db121 Puppet on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:38:19] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:24] PROBLEM - db121 conntrack_table_size on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:38:33] PROBLEM - db121 NTP time on db121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:34] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 4 backends are down. mw131 mw132 mw141 mw142 [12:39:25] PROBLEM - db121 ferm_active on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:39:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 4.16, 8.18, 11.46 [12:39:59] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.382 second response time [12:40:14] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.539 second response time [12:41:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 1.48, 6.87, 10.91 [12:41:09] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.458 second response time [12:41:51] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.561 second response time [12:42:11] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.378 second response time [12:42:15] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.356 second response time [12:42:20] RECOVERY - db121 ferm_active on db121 is OK: OK ferm input default policy is set [12:43:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 1.80, 5.04, 9.44 [12:45:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.26, 11.07, 11.51 [12:45:17] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:45:42] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.012 second response time [12:45:57] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.029 second response time [12:46:02] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.010 second response time [12:46:17] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.006 second response time [12:47:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 4.55, 8.25, 10.42 [12:48:26] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 1.00, 5.70, 9.22 [12:49:20] Hmm [12:49:27] PROBLEM - db121 ferm_active on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:49:57] Reception123: not sure what is going on with mw* but users are reporting issues [12:50:02] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.627 second response time [12:50:19] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.377 second response time [12:50:37] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 55% [12:51:23] RECOVERY - db121 ferm_active on db121 is OK: OK ferm input default policy is set [12:51:28] RECOVERY - db121 Backups SQL on db121 is OK: FILE_AGE OK: /var/log/sql-backup.log is 360435 seconds old and 1751 bytes [12:51:29] RECOVERY - db121 NTP time on db121 is OK: NTP OK: Offset -0.001795142889 secs [12:51:47] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.813 second response time [12:51:59] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.851 second response time [12:52:02] PROBLEM - db121 MariaDB Connections on db121 is CRITICAL: PHP Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Connection failed: Connection refused [12:52:08] RECOVERY - db121 conntrack_table_size on db121 is OK: OK: nf_conntrack is 0 % full [12:52:13] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 32 minutes ago with 0 failures [12:52:13] RECOVERY - db121 SSH on db121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:52:18] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) [12:52:24] RECOVERY - db121 PowerDNS Recursor on db121 is OK: DNS OK: 0.610 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [12:52:32] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.397 second response time [12:52:35] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 17% [12:52:55] Reception123: paladox: seems like an issue with db121 mariadb [12:53:14] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.254 second response time [12:53:23] RECOVERY - db121 APT on db121 is OK: APT OK: 55 packages available for upgrade (0 critical updates). [12:53:59] RECOVERY - db121 MariaDB Connections on db121 is OK: OK connection usage: 1.8%Current connections: 9 [12:54:14] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 167 Threads: 24 Questions: 27579 Slow queries: 2096 Opens: 2264 Open tables: 2258 Queries per second avg: 165.143 [12:55:07] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 15.05, 7.71, 5.03 [12:55:47] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [12:56:33] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [12:56:34] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [12:56:43] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [12:56:46] Now it restarted and one of you will need to unset the RO [12:57:07] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.34, 6.93, 5.08 [12:59:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.30, 10.00, 7.94 [13:03:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.93, 11.59, 9.17 [13:04:53] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.28, 10.76, 9.25 [13:06:09] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.30, 3.38, 2.41 [13:06:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.44, 10.01, 9.14 [13:08:05] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.08, 2.96, 2.38 [13:09:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.36, 11.92, 10.00 [13:10:50] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.65, 12.59, 10.40 [13:12:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.39, 6.78, 5.80 [13:13:46] I won't have access until at least 2 hours [13:13:51] @paladox [13:14:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.15, 7.74, 6.26 [13:15:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.98, 7.85, 6.49 [13:16:44] I have no access [13:23:44] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.55, 6.66, 6.53 [13:26:53] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.40, 6.36, 3.52 [13:28:53] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.32, 4.87, 3.34 [13:29:52] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [13:31:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.45, 7.10, 6.74 [13:32:29] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 0.82, 0.87, 7.68 [13:33:32] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.82, 6.64, 6.62 [13:36:28] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.45, 0.76, 6.09 [13:37:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.92, 7.59, 7.09 [13:45:05] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.73, 2.93, 2.01 [13:45:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.35, 7.23, 7.08 [13:47:05] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.89, 2.17, 1.85 [13:47:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.61, 6.96, 7.01 [13:59:14] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [14:01:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.03, 7.91, 7.34 [14:03:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.81, 7.52, 7.27 [14:06:46] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.21, 3.45, 2.94 [14:07:16] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1409535706 secs [14:08:46] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 1.99, 3.05, 2.86 [14:17:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.55, 6.05, 6.61 [14:21:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.41, 7.48, 7.11 [14:25:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.08, 7.61, 7.18 [14:27:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.40, 6.98, 6.99 [14:29:39] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a303d73f7416...3f361ad6d0f7 [14:29:42] [02miraheze/mw-config] 07AgentIsai 033f361ad - Add myself to wgImportDumpUsersNotifiedOnAllRequests [14:30:38] miraheze/mw-config - AgentIsai the build passed. [14:30:53] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 12.75, 8.05, 4.21 [14:32:24] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=bapham123wiki --no-updates --username-prefix=wikia:themandelacataloguevn /home/agent/ImportDump/bapham123wiki-20230509141155.xml (START) [14:32:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:33:16] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09654673934 secs [14:34:53] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.72, 7.75, 5.03 [14:36:53] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.48, 7.31, 5.15 [14:37:08] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=bapham123wiki --no-updates --username-prefix=wikia:themandelacataloguevn /home/agent/ImportDump/bapham123wiki-20230509141155.xml (END - exit=0) [14:37:09] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=bapham123wiki (START) [14:37:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:37:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:38:50] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.48, 3.38, 2.56 [14:38:53] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.33, 5.75, 4.84 [14:40:51] !log [@mwtask141] starting deploy of {'config': True} to all [14:40:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:41:00] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [14:41:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:41:09] !log MariaDB [(none)]> set global read_only = 0; [14:41:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:41:46] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=bapham123wiki (END - exit=0) [14:41:47] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=bapham123wiki --active --update (END - exit=0) [14:41:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:42:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:42:48] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.06, 3.16, 2.70 [14:46:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 4.00, 3.50, 2.87 [14:47:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.97, 7.39, 7.17 [14:48:22] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.21, 2.93, 2.73 [14:51:17] [02puppet] 07MacFan4000 opened pull request 03#3212: Disable OS shell account and remove from icinga - 13https://github.com/miraheze/puppet/pull/3212 [14:51:54] [02mw-config] 07brandon-wm opened pull request 03#5227: Remove OrangeStar - 13https://github.com/miraheze/mw-config/pull/5227 [14:53:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.62, 7.20, 7.31 [14:56:46] [02mw-config] 07MacFan4000 closed pull request 03#5227: Remove OrangeStar - 13https://github.com/miraheze/mw-config/pull/5227 [14:56:47] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3f361ad6d0f7...a844b0a79398 [14:56:49] [02miraheze/mw-config] 07brandon-wm 03a844b0a - Remove OrangeStar (#5227) [14:57:06] !log [@test131] starting deploy of {'config': True} to all [14:57:07] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [14:57:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:57:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:57:44] miraheze/mw-config - brandon-wm the build passed. [14:57:49] miraheze/mw-config - MacFan4000 the build passed. [14:59:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.27, 5.89, 6.66 [14:59:27] !log [@mwtask141] starting deploy of {'config': True} to all [14:59:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:59:38] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 11s [14:59:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:06:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.01, 8.37, 7.38 [15:08:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.37, 7.53, 7.20 [15:10:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.01, 8.05, 7.39 [15:12:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.38, 7.52, 7.26 [15:14:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.76, 7.85, 7.40 [15:16:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.32, 7.19, 7.21 [15:20:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:20:07] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [15:23:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.99, 8.48, 7.77 [15:27:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.18, 7.30, 7.47 [15:30:02] [02puppet] 07Reception123 synchronize pull request 03#3212: Disable OS shell account and remove from icinga - 13https://github.com/miraheze/puppet/pull/3212 [15:30:15] [02puppet] 07Reception123 closed pull request 03#3212: Disable OS shell account and remove from icinga - 13https://github.com/miraheze/puppet/pull/3212 [15:30:18] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/ee80d6925fbf...01f2c7523543 [15:30:20] [02miraheze/puppet] 07MacFan4000 0301f2c75 - Disable OS shell account and remove from icinga (#3212) [15:34:53] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.44, 6.03, 3.21 [15:35:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.25, 7.86, 7.37 [15:37:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.25, 3.48, 2.72 [15:39:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.59, 3.39, 2.80 [15:39:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.88, 7.10, 7.20 [15:41:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 13.04, 8.53, 7.66 [15:42:53] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.65, 6.34, 4.89 [15:43:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.60, 7.82, 7.50 [15:49:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.27, 7.92, 7.61 [15:49:56] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:55:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.91, 7.44, 7.64 [15:59:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.31, 7.94, 7.78 [16:01:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.53, 7.35, 7.58 [16:03:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.29, 7.51, 7.58 [16:05:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:06:21] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1173978448 secs [16:07:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.34, 6.63, 7.20 [16:10:35] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.61, 3.22, 2.26 [16:11:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.11, 8.07, 7.66 [16:12:35] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.92, 2.68, 2.18 [16:13:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.74, 7.95, 7.68 [16:14:08] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:16:04] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.04348701239 secs [16:16:11] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.515 second response time [16:24:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.07, 4.05, 2.99 [16:26:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.93, 3.77, 3.03 [16:28:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.37, 4.18, 3.25 [16:28:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.60, 6.21, 6.78 [16:30:46] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.07, 5.28, 3.53 [16:32:16] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.97, 2.87, 1.71 [16:32:46] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.68, 7.04, 4.39 [16:33:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:34:16] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.22, 2.31, 1.65 [16:34:46] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.40, 5.77, 4.27 [16:34:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.18, 7.19, 7.09 [16:36:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.99, 3.62, 3.43 [16:36:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.66, 8.33, 7.53 [16:38:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.96, 3.03, 3.23 [16:38:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.66, 7.19, 7.20 [16:40:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.42, 7.85, 7.42 [16:43:39] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [16:44:09] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.85, 3.12, 1.80 [16:44:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.18, 7.41, 7.40 [16:44:52] PROBLEM - graylog131 NTP time on graylog131 is WARNING: NTP WARNING: Offset -0.1036792099 secs [16:46:07] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.05, 2.34, 1.67 [16:46:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.73, 7.64, 7.46 [16:46:55] RECOVERY - graylog131 NTP time on graylog131 is OK: NTP OK: Offset -0.09055429697 secs [16:48:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.36, 7.22, 7.31 [16:49:21] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [16:52:46] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.64, 6.82, 4.71 [16:54:46] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.87, 5.31, 4.42 [16:54:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.99, 7.29, 7.21 [16:56:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.50, 6.89, 7.08 [17:03:17] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [17:06:16] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.98, 2.67, 1.75 [17:07:10] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [17:10:16] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.45, 2.84, 2.06 [17:10:46] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.11, 6.45, 6.77 [17:13:26] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:14:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.57, 6.57, 6.82 [17:18:46] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.35, 6.10, 6.59 [17:21:15] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1103660762 secs [17:22:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.85, 7.33, 7.03 [17:23:15] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.09055408835 secs [17:23:27] [02miraheze/ImportDump] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/0cbbe7b1e29c...8c17db4a61f4 [17:23:29] [02miraheze/ImportDump] 07AgentIsai 038c17db4 - Add backline to main queue page on request viewer page [17:26:39] [02miraheze/ImportDump] 07AgentIsai pushed 031 commit to 03redirect-if-logged-out [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/86260c7e304c [17:26:42] [02miraheze/ImportDump] 07AgentIsai 0386260c7 - Check if logged in [17:26:45] [02ImportDump] 07AgentIsai created branch 03redirect-if-logged-out - 13https://github.com/miraheze/ImportDump [17:27:23] [02ImportDump] 07AgentIsai opened pull request 03#42: Redirect logged out users directly to the login page - 13https://github.com/miraheze/ImportDump/pull/42 [17:29:00] [02miraheze/ImportDump] 07github-actions[bot] pushed 031 commit to 03redirect-if-logged-out [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/86260c7e304c...658da13a2b7d [17:29:01] [02miraheze/ImportDump] 07github-actions 03658da13 - CI: lint code to MediaWiki standards [17:29:04] [02ImportDump] 07github-actions[bot] synchronize pull request 03#42: Redirect logged out users directly to the login page - 13https://github.com/miraheze/ImportDump/pull/42 [17:29:27] [02miraheze/ImportDump] 07AgentIsai pushed 031 commit to 03redirect-if-logged-out [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/658da13a2b7d...1e79608129a9 [17:29:30] [02miraheze/ImportDump] 07AgentIsai 031e79608 - Add hook for custom Special:UserLogin message [17:29:31] [02ImportDump] 07AgentIsai synchronize pull request 03#42: Redirect logged out users directly to the login page - 13https://github.com/miraheze/ImportDump/pull/42 [17:30:10] [02ImportDump] 07AgentIsai synchronize pull request 03#42: Redirect logged out users directly to the login page - 13https://github.com/miraheze/ImportDump/pull/42 [17:30:12] [02miraheze/ImportDump] 07AgentIsai pushed 031 commit to 03redirect-if-logged-out [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/1e79608129a9...dbb3d4458a92 [17:30:15] [02miraheze/ImportDump] 07AgentIsai 03dbb3d44 - Add LoginFormValidErrorMessagesHook [17:31:17] [02miraheze/ImportDump] 07AgentIsai pushed 031 commit to 03redirect-if-logged-out [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/dbb3d4458a92...3bdd9f76f224 [17:31:20] [02miraheze/ImportDump] 07AgentIsai 033bdd9f7 - Update extension.json [17:31:21] [02ImportDump] 07AgentIsai synchronize pull request 03#42: Redirect logged out users directly to the login page - 13https://github.com/miraheze/ImportDump/pull/42 [17:32:03] miraheze/ImportDump - AgentIsai the build has errored. [17:32:13] [02miraheze/ImportDump] 07AgentIsai pushed 031 commit to 03redirect-if-logged-out [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/3bdd9f76f224...0b23f1374cf0 [17:32:14] [02miraheze/ImportDump] 07AgentIsai 030b23f13 - Update i18n to reflect new place in UserLogin page [17:32:15] [02ImportDump] 07AgentIsai synchronize pull request 03#42: Redirect logged out users directly to the login page - 13https://github.com/miraheze/ImportDump/pull/42 [17:33:25] miraheze/ImportDump - AgentIsai the build passed. [17:33:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:33:43] [02miraheze/ImportDump] 07github-actions[bot] pushed 031 commit to 03redirect-if-logged-out [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/0b23f1374cf0...c81410b85474 [17:33:46] [02miraheze/ImportDump] 07github-actions 03c81410b - CI: lint code to MediaWiki standards [17:33:48] [02ImportDump] 07github-actions[bot] synchronize pull request 03#42: Redirect logged out users directly to the login page - 13https://github.com/miraheze/ImportDump/pull/42 [17:34:46] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.73, 6.55, 6.78 [17:40:22] miraheze/ImportDump - AgentIsai the build passed. [17:42:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.75, 6.99, 6.80 [17:43:12] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [17:43:48] !log [agent@test131] starting deploy of {'folders': 'w/extensions/ImportDump'} to all [17:43:49] !log [agent@test131] finished deploy of {'folders': 'w/extensions/ImportDump'} to all - SUCCESS in 0s [17:43:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:44:46] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.42, 6.52, 6.67 [17:45:29] [02mw-config] 07AgentIsai closed pull request 03#5225: Userrights config for nycsubwaywiki - 13https://github.com/miraheze/mw-config/pull/5225 [17:45:30] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a844b0a79398...abde38285813 [17:45:31] [02miraheze/mw-config] 07Amanda-Catherine 03abde382 - Userrights config for nycsubwaywiki (#5225) [17:46:16] [02ImportDump] 07AgentIsai commented on pull request 03#42: Redirect logged out users directly to the login page - 13https://github.com/miraheze/ImportDump/pull/42#issuecomment-1540604662 [17:46:26] [02ImportDump] 07AgentIsai closed pull request 03#42: Redirect logged out users directly to the login page - 13https://github.com/miraheze/ImportDump/pull/42 [17:46:28] [02miraheze/ImportDump] 07AgentIsai deleted branch 03redirect-if-logged-out [17:46:31] [02ImportDump] 07AgentIsai deleted branch 03redirect-if-logged-out - 13https://github.com/miraheze/ImportDump [17:46:33] [02miraheze/ImportDump] 07AgentIsai pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/ImportDump/compare/8c17db4a61f4...0d0af464bcb0 [17:46:35] [02miraheze/ImportDump] 07AgentIsai 030d0af46 - Redirect logged out users directly to the login page (#42) [17:46:41] miraheze/mw-config - AgentIsai the build passed. [17:50:31] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:54:24] [02miraheze/ImportDump] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/0d0af464bcb0...9e98595d9fd4 [17:54:25] [02miraheze/ImportDump] 07AgentIsai 039e98595 - Fix capitalization in i18n [17:56:55] !log [@test131] starting deploy of {'config': True} to all [17:56:56] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [17:57:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:57:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:02:05] miraheze/ImportDump - AgentIsai the build passed. [18:02:22] [02miraheze/mediawiki] 07AgentIsai pushed 031 commit to 03REL1_40 [+0/-0/±6] 13https://github.com/miraheze/mediawiki/compare/9e21e9fa8585...8049cd5244a0 [18:02:25] [02miraheze/mediawiki] 07AgentIsai 038049cd5 - Updating [18:03:09] [02miraheze/mediawiki] 07AgentIsai pushed 031 commit to 03REL1_40 [+0/-0/±6] 13https://github.com/miraheze/mediawiki/compare/8049cd5244a0...1998b74046f2 [18:03:12] [02miraheze/mediawiki] 07AgentIsai 031998b74 - Update all extensions [18:06:39] [02miraheze/mediawiki] 07AgentIsai pushed 031 commit to 03REL1_40 [+0/-0/±12] 13https://github.com/miraheze/mediawiki/compare/1998b74046f2...acb7937c4c69 [18:06:41] [02miraheze/mediawiki] 07AgentIsai 03acb7937 - Update core extensions [18:07:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.22, 6.94, 6.81 [18:11:38] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.68, 5.97, 6.46 [18:12:00] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.32, 3.55, 2.68 [18:12:17] !log [@mwtask141] starting deploy of {'config': True} to all [18:12:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:12:33] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 15s [18:12:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:12:59] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:14:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.52, 3.52, 2.78 [18:17:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.78, 7.19, 6.83 [18:18:00] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.16, 3.35, 2.90 [18:19:00] Could somebody please merge https://github.com/miraheze/puppet/pull/3208 when they get a chance? [18:19:27] Um is there a way I can change the logo of my own wiki? You know have it not be the Mira logo? [18:19:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.04, 7.42, 6.94 [18:20:07] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [18:21:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.80, 7.21, 6.93 [18:22:37] @Reven yes, see https://meta.miraheze.org/wiki/ManageWiki#Frequently_asked_questions [18:22:52] The first thing under frequently asked questions [18:23:25] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.08, 6.61, 6.73 [18:27:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.08, 6.87, 6.80 [18:29:18] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.43, 6.17, 6.54 [18:36:46] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.01, 5.27, 2.87 [18:38:45] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.99, 5.91, 3.40 [18:40:45] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.23, 5.02, 3.39 [18:47:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.49, 3.52, 3.26 [18:50:48] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.13, 3.39, 3.27 [18:52:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.60, 9.89, 7.99 [18:54:24] [02puppet] 07Universal-Omega opened pull request 03#3213: test_mwscript: make tests pass without skipping - 13https://github.com/miraheze/puppet/pull/3213 [19:00:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.54, 4.75, 3.30 [19:00:33] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:01:57] [02ImportDump] 07Universal-Omega opened pull request 03#43: Fix comment format - 13https://github.com/miraheze/ImportDump/pull/43 [19:02:20] Thank you [19:02:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 7.65, 7.83 [19:04:06] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.04, 3.50, 3.31 [19:08:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.60, 3.73, 3.46 [19:09:09] [02puppet] 07Reception123 closed pull request 03#3213: test_mwscript: make tests pass without skipping - 13https://github.com/miraheze/puppet/pull/3213 [19:09:12] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/01f2c7523543...764cc44bf91d [19:09:15] [02miraheze/puppet] 07Universal-Omega 03764cc44 - test_mwscript: make tests pass without skipping (#3213) [19:10:43] [02ImportDump] 07AgentIsai closed pull request 03#43: Fix comment format - 13https://github.com/miraheze/ImportDump/pull/43 [19:10:46] [02miraheze/ImportDump] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/9e98595d9fd4...196f624b1ce3 [19:10:49] [02miraheze/ImportDump] 07Universal-Omega 03196f624 - Fix comment format (#43) [19:11:15] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1863437891 secs [19:12:00] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.11, 3.82, 3.55 [19:12:20] miraheze/ImportDump - Universal-Omega the build passed. [19:12:32] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [19:14:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.15, 3.63, 3.51 [19:16:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.09, 3.43, 3.75 [19:16:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.25, 7.78, 7.53 [19:18:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.67, 7.12, 7.30 [19:20:02] miraheze/ImportDump - AgentIsai the build passed. [19:20:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.89, 7.86, 7.54 [19:22:00] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.22, 3.26, 3.37 [19:22:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.64, 2.46, 3.22 [19:22:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.44, 6.95, 7.25 [19:25:33] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:26:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.01, 7.71, 7.46 [19:30:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.99, 7.91, 7.61 [19:36:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.79, 8.16, 7.71 [19:37:02] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 4.00, 2.96, 2.34 [19:39:01] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.18, 2.60, 2.29 [19:41:15] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.05811002851 secs [19:42:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.02, 7.86, 7.74 [19:46:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.38, 7.34, 7.49 [19:50:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.56, 8.00, 7.75 [19:51:45] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.61, 3.77, 3.21 [19:54:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.27, 8.10, 7.86 [19:55:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.32, 4.27, 3.48 [19:57:39] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.20, 3.63, 3.33 [19:59:38] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.30, 3.16, 3.19 [20:00:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.02, 7.58, 7.69 [20:10:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.49, 7.96, 7.62 [20:12:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.86, 7.36, 7.44 [20:18:08] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=creaturawiki --delete --all --ns 3000 (START) [20:18:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:46] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.17, 5.96, 6.67 [20:34:21] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=creaturawiki --delete --all --ns 3000 (END - exit=0) [20:34:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:38:04] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=creaturawiki --delete --all --ns 3002 (START) [20:38:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:46:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.43, 3.13, 2.83 [20:48:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.14, 3.11, 2.86 [20:58:16] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.68, 6.78, 6.34 [21:02:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.81, 4.07, 3.35 [21:02:14] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.50, 6.53, 6.36 [21:04:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.93, 3.20, 3.11 [21:09:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.55, 7.24, 6.71 [21:13:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.70, 7.71, 6.95 [21:16:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.70, 7.42, 6.97 [21:24:06] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.44, 4.34, 2.96 [21:26:05] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.84, 3.66, 2.87 [21:28:04] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.22, 2.88, 2.69 [21:30:46] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.39, 6.38, 6.77 [21:31:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.29, 11.42, 11.95 [21:32:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.39, 3.45, 2.89 [21:34:55] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.83, 3.72, 3.07 [21:35:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.34, 11.75, 11.94 [21:36:46] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.26, 6.27, 3.88 [21:36:54] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.15, 4.86, 3.58 [21:37:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.99, 11.27, 11.73 [21:38:45] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.75, 4.49, 3.51 [21:39:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.87, 12.83, 12.20 [21:48:43] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.11, 3.83, 3.90 [21:50:42] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.52, 4.14, 4.01 [22:14:45] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.10, 5.00, 2.97 [22:16:23] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1126888692 secs [22:16:45] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.18, 3.46, 2.65 [22:18:23] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08397242427 secs [22:20:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.51, 2.89, 3.97 [22:23:35] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.79, 3.96, 3.06 [22:26:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.20, 3.53, 3.88 [22:27:33] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.37, 2.83, 2.83 [22:34:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.44, 3.59, 3.91 [22:40:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.44, 4.91, 4.38 [22:47:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.49, 6.98, 6.51 [22:52:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.04, 3.38, 3.95 [22:53:36] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.47, 6.64, 6.52 [22:57:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.48, 7.89, 7.03 [22:59:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.29, 6.93, 6.79 [23:00:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.04, 3.51, 3.71 [23:01:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.32, 7.74, 7.07 [23:02:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.53, 3.09, 3.53 [23:03:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.48, 7.54, 7.08 [23:04:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.66, 4.15, 3.87 [23:06:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.83, 3.70, 3.75 [23:10:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.81, 2.87, 3.40 [23:11:10] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.99, 3.64, 2.87 [23:13:09] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.92, 3.18, 2.80 [23:21:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.50, 7.59, 7.23 [23:22:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.13, 4.50, 3.74 [23:23:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.61, 6.86, 7.01 [23:25:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.07, 7.27, 7.14 [23:26:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.84, 3.77, 3.62 [23:26:46] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.74, 5.91, 3.31 [23:27:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.50, 6.88, 7.01 [23:28:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.08, 4.60, 3.95 [23:28:45] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.22, 5.03, 3.33 [23:34:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.12, 3.34, 3.60 [23:34:58] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.41, 6.21, 6.75 [23:36:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.01, 3.80, 3.72 [23:36:46] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 12.61, 7.95, 4.82 [23:38:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.85, 3.42, 3.59 [23:38:45] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.39, 6.30, 4.62 [23:46:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.26, 4.92, 4.11 [23:50:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.85, 3.55, 3.76 [23:52:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.89, 4.28, 3.98 [23:54:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.26, 3.90, 3.88 [23:59:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.19, 7.07, 6.77