[00:02:59] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.92, 2.82, 3.12 [00:04:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 7.32, 9.02, 5.31 [00:04:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.61, 3.00, 2.23 [00:04:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.59, 3.97, 2.59 [00:06:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.19, 2.96, 2.38 [00:08:14] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.30, 3.22, 2.58 [00:08:49] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.12, 3.68, 3.38 [00:09:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.96, 3.84, 2.83 [00:10:46] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.46, 3.98, 3.55 [00:11:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.68, 3.21, 2.72 [00:12:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 1.52, 3.17, 3.88 [00:14:41] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.16, 3.77, 3.53 [00:15:28] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.04737374187 secs [00:16:39] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.56, 3.55, 3.47 [00:18:07] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 1.89, 2.42, 3.32 [00:18:36] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 3.06, 3.34, 3.40 [00:25:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.58, 3.52, 2.91 [00:27:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.42, 3.21, 2.87 [00:36:08] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.26, 3.61, 3.31 [00:38:06] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.02, 3.04, 3.14 [00:38:37] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.10, 4.50, 2.98 [00:40:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.97, 3.80, 2.90 [00:42:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.83, 3.00, 2.71 [00:51:34] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.41, 3.01, 2.28 [00:52:43] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.33, 3.49, 3.29 [00:53:30] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.25, 2.36, 2.13 [00:54:41] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.59, 3.21, 3.22 [00:58:33] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.06, 3.77, 3.45 [01:03:10] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 6.37, 3.79, 2.72 [01:05:06] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.98, 3.35, 2.67 [01:08:01] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.83, 5.19, 3.48 [01:08:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.73, 3.89, 3.76 [01:10:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 3.49, 4.13, 3.88 [01:11:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.83, 3.98, 3.43 [01:12:49] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.49, 2.84, 2.75 [01:13:55] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.33, 3.14, 3.19 [01:22:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.00, 3.64, 3.94 [01:25:13] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.15, 3.54, 2.44 [01:29:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.58, 3.91, 2.95 [01:30:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.81, 3.35, 3.54 [01:31:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.52, 3.21, 2.82 [01:34:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.64, 3.21, 3.49 [01:36:22] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 3.11, 3.09, 3.40 [01:42:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 6.00, 4.22, 3.70 [01:47:01] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.41, 3.79, 2.66 [01:53:13] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.60, 3.36, 2.54 [01:54:44] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 1.56, 3.73, 3.27 [01:56:40] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.64, 3.11, 3.10 [01:58:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 1.85, 2.99, 3.73 [01:59:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.69, 3.79, 3.05 [02:02:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.41, 3.25, 3.63 [02:03:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.67, 3.23, 3.06 [02:04:22] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.11, 2.38, 3.26 [02:06:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.69, 3.40, 2.46 [02:10:48] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.35, 3.98, 3.00 [02:12:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 11.85, 5.71, 3.68 [02:13:43] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.seamly.net. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:18:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.12, 3.09, 3.22 [02:24:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.57, 3.42, 3.13 [02:25:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.82, 3.29, 2.62 [02:26:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.48, 3.49, 3.19 [02:27:13] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.49, 6.99, 4.15 [02:28:22] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 3.21, 3.35, 3.17 [02:33:26] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 9.96, 5.67, 3.60 [02:34:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 3.93, 4.43, 3.67 [02:38:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.40, 3.71, 2.99 [02:40:13] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.08, 3.18, 2.89 [02:41:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.23, 3.15, 4.00 [02:42:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.41, 3.91, 3.83 [02:43:04] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.51, 3.78, 3.74 [02:43:05] RECOVERY - wiki.seamly.net - reverse DNS on sslhost is OK: SSL OK - wiki.seamly.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:44:09] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.34, 7.06, 4.61 [02:45:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.87, 1.79, 3.24 [02:46:56] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.96, 2.84, 3.34 [02:48:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.34, 4.09, 3.87 [02:50:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.18, 3.55, 3.75 [02:50:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.08, 3.25, 3.58 [02:53:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.02, 2.29, 3.18 [02:54:22] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.14, 2.79, 3.33 [03:03:50] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.57, 4.57, 2.51 [03:04:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.59, 4.05, 3.56 [03:05:50] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.96, 3.26, 2.28 [03:07:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.88, 4.63, 3.57 [03:09:46] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.99, 3.56, 3.30 [03:11:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.35, 3.14, 3.17 [03:14:08] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.44, 3.77, 3.34 [03:16:08] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 6.25, 4.87, 3.80 [03:20:08] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.56, 3.94, 3.67 [03:22:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.93, 3.85, 3.95 [03:24:08] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.57, 4.13, 3.73 [03:24:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 7.55, 4.73, 4.23 [03:34:08] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 1.26, 3.10, 3.53 [03:36:08] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.38, 2.50, 3.26 [03:37:50] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 13.17, 6.45, 3.79 [03:43:32] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1999721229 secs [03:44:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.05, 3.36, 3.97 [03:53:50] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.3361420929 secs [03:54:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.08, 3.41, 3.68 [03:58:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.82, 3.55, 3.67 [03:59:46] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.02, 1.85, 3.88 [04:01:50] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.90, 3.43, 2.02 [04:03:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.54, 1.58, 3.29 [04:03:50] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.49, 2.35, 1.80 [04:04:22] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.87, 2.85, 3.35 [04:09:50] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.01619011164 secs [04:13:28] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.06838074327 secs [04:24:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.71, 3.24, 2.99 [04:26:22] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.76, 3.27, 3.05 [04:33:50] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1281622946 secs [04:35:10] PROBLEM - wiki.erisly.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.erisly.com could not be found [04:39:14] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.35, 3.66, 3.31 [04:41:12] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.99, 3.02, 3.11 [04:43:50] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.09752458334 secs [04:45:04] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.18, 4.31, 3.61 [04:47:01] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.28, 3.72, 3.46 [04:48:59] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.32, 4.28, 3.68 [04:50:56] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.52, 3.76, 3.55 [04:54:51] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.19, 3.28, 3.35 [04:56:49] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.07, 2.89, 3.21 [05:03:50] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset -0.1171783507 secs [05:05:50] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.084602952 secs [05:06:31] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.71, 3.56, 3.42 [05:08:29] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.24, 3.10, 3.27 [05:17:01] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.48, 3.23, 1.71 [05:22:21] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.83, 2.80, 1.95 [05:24:16] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.02, 2.49, 1.93 [05:25:01] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.59, 3.04, 2.77 [05:28:10] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.94, 3.56, 2.50 [05:30:08] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.42, 3.20, 2.50 [05:53:28] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.102442205 secs [05:55:13] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.22, 3.11, 1.91 [05:58:34] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.95, 3.70, 3.00 [06:02:29] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.16, 3.97, 3.29 [06:04:59] PROBLEM - ping6 on cp33 is WARNING: PING WARNING - Packet loss = 0%, RTA = 138.55 ms [06:05:27] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 133.89 ms [06:05:28] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.09915328026 secs [06:05:57] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 138.39 ms [06:07:56] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 138.47 ms [06:08:22] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.55, 3.01, 3.08 [06:09:01] RECOVERY - ping6 on cp33 is OK: PING OK - Packet loss = 0%, RTA = 118.40 ms [06:09:33] RECOVERY - ping6 on cp32 is OK: PING OK - Packet loss = 0%, RTA = 102.88 ms [06:09:55] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 106.26 ms [06:15:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 0.87, 2.71, 3.99 [06:19:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.70, 1.70, 3.28 [06:27:50] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.99, 3.89, 1.84 [06:29:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 12.08, 10.06, 6.00 [06:29:50] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.37, 3.47, 1.95 [06:31:50] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.48, 2.41, 1.74 [06:39:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 0.62, 2.29, 3.68 [06:41:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.52, 1.67, 3.28 [06:50:31] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.98, 3.52, 3.04 [06:52:29] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 3.05, 3.21, 2.98 [06:55:13] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.87, 3.66, 2.97 [07:07:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 0.63, 3.03, 3.69 [07:09:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.34, 2.47, 3.40 [07:19:01] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.70, 3.43, 1.98 [07:21:01] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.95, 3.05, 2.03 [07:25:13] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.66, 4.71, 3.26 [07:33:30] PROBLEM - cp33 PowerDNS Recursor on cp33 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:35:30] RECOVERY - cp33 PowerDNS Recursor on cp33 is OK: DNS OK: 0.378 seconds response time. miraheze.org returns 108.175.15.182,2607:f1c0:1800:26f::1,2607:f1c0:1800:8100::1,74.208.203.152 [07:40:24] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.03, 3.43, 2.30 [07:40:38] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=itorawiki --no-updates /home/reception/ImportDump/itorawiki-20221129202908.xml (START) [07:40:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:40:47] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=gensekaiwiki --no-updates --username-prefix=wikia:gensekai /home/reception/ImportDump/gensekaiwiki-20221129230600.xml (START) [07:40:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:41:00] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=gensekaiwiki --no-updates --username-prefix=wikia:gensekai /home/reception/ImportDump/gensekaiwiki-20221129230600.xml (END - exit=0) [07:41:01] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=gensekaiwiki (START) [07:41:04] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=itorawiki --no-updates /home/reception/ImportDump/itorawiki-20221129202908.xml (END - exit=0) [07:41:05] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=itorawiki (START) [07:41:08] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=decodedocdigitizewiki --no-updates --username-prefix=mh:decodedocdigitize /home/reception/ImportDump/decodedocdigitizewiki-20221129233215.xml (START) [07:41:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:41:20] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=gensekaiwiki (END - exit=0) [07:41:21] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=gensekaiwiki --active --update (END - exit=0) [07:41:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:41:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:41:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:42:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:42:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:42:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:42:22] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.46, 2.68, 2.16 [07:43:53] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=itorawiki (END - exit=0) [07:43:54] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=itorawiki --active --update (END - exit=0) [07:43:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:44:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:53:42] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=decodedocdigitizewiki --no-updates --username-prefix=mh:decodedocdigitize /home/reception/ImportDump/decodedocdigitizewiki-20221129233215.xml (END - exit=0) [07:53:43] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=decodedocdigitizewiki (START) [07:53:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:53:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:59:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.08, 2.38, 3.79 [08:01:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.68, 1.74, 3.38 [08:02:06] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=decodedocdigitizewiki (END - exit=0) [08:02:07] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=decodedocdigitizewiki --active --update (END - exit=0) [08:02:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:02:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:25:13] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.32, 3.03, 2.23 [08:29:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.39, 3.06, 2.53 [08:41:50] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1280248761 secs [08:47:01] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.67, 3.30, 1.67 [08:49:01] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.41, 3.09, 1.80 [08:55:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.85, 2.56, 1.80 [08:57:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.37, 2.87, 2.03 [09:05:50] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.08042180538 secs [09:09:46] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.52, 3.83, 2.24 [09:11:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.24, 3.08, 2.17 [09:27:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.48, 2.65, 1.71 [09:29:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.16, 2.18, 1.66 [09:37:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.47, 2.59, 1.62 [09:39:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.22, 2.92, 1.88 [09:45:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.03, 3.88, 2.56 [09:53:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.72, 3.16, 3.09 [10:29:40] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.70, 4.04, 3.03 [10:30:08] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.96, 3.47, 2.53 [10:31:38] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.03, 3.72, 3.04 [10:32:08] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 3.00, 3.01, 2.45 [10:35:33] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.77, 3.09, 2.97 [10:36:08] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.19, 4.74, 3.36 [10:38:08] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.74, 3.95, 3.23 [10:41:23] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.86, 3.67, 3.20 [10:42:08] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.64, 2.96, 3.01 [10:43:20] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.87, 3.27, 3.11 [10:52:08] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.16, 3.40, 3.02 [10:54:08] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.79, 3.40, 3.09 [10:55:13] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.61, 4.06, 2.09 [10:57:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.16, 3.64, 2.21 [10:58:55] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.67, 3.42, 3.18 [10:59:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.71, 2.97, 2.13 [11:00:53] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.12, 2.89, 3.01 [11:04:45] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 6.97, 4.84, 3.76 [11:06:08] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.36, 3.88, 3.19 [11:06:43] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.52, 3.93, 3.56 [11:08:08] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.47, 3.12, 2.99 [11:08:40] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 6.09, 4.56, 3.81 [11:09:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.28, 3.66, 1.97 [11:10:38] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.46, 3.74, 3.60 [11:12:35] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.65, 4.55, 3.92 [11:17:15] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1781359613 secs [11:20:25] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.12, 3.70, 3.79 [11:23:46] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.73, 2.95, 3.64 [11:25:13] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.46, 2.80, 1.72 [11:25:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.79, 2.21, 3.29 [11:27:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.98, 3.09, 1.98 [11:27:14] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08136767149 secs [11:28:22] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.88, 2.96, 3.36 [11:34:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.76, 5.44, 4.31 [11:38:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.45, 3.72, 3.86 [11:44:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.64, 3.56, 3.68 [11:48:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.48, 3.47, 3.61 [11:52:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.33, 3.46, 3.55 [11:52:44] !log reception@mwtask141:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/dumpBackup.php --full --include-files --wiki obeymewiki > /home/reception/obeymewiki30112022.xml [11:52:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:54:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.14, 3.39, 3.51 [11:56:22] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.23, 2.94, 3.33 [11:58:41] !log reception@mwtask141:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/dumpBackup.php --full --include-files --wiki hololivewiki > /home/reception/hololivewiki30112022.xml [11:58:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:02:00] !log reception@mwtask141:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/dumpBackup.php --full --include-files --wiki archiopediawiki > /home/reception/archiopediawiki30112022.xml [12:02:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:09:09] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 6.36, 4.14, 3.54 [12:11:06] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.81, 3.58, 3.41 [12:13:04] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.00, 3.09, 3.25 [12:47:01] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.52, 2.83, 1.57 [12:53:01] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.01, 3.32, 2.37 [13:06:38] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.81, 2.82, 1.77 [13:08:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.40, 3.42, 2.11 [13:10:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.41, 2.63, 1.98 [13:35:04] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.62, 4.00, 3.33 [13:38:59] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.73, 3.26, 3.16 [13:57:13] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.23, 2.70, 1.73 [14:03:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.50, 2.84, 2.22 [14:08:23] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.08, 4.81, 2.61 [14:12:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.41, 3.18, 3.01 [14:14:22] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.32, 2.69, 2.84 [14:15:53] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:08] PROBLEM - cp32 PowerDNS Recursor on cp32 is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:17:50] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:18:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.50, 3.87, 3.33 [14:20:14] RECOVERY - cp32 PowerDNS Recursor on cp32 is OK: DNS OK: 0.121 seconds response time. miraheze.org returns 2607:f1c0:1800:26f::1,74.208.203.152 [14:20:22] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.84, 3.28, 3.19 [14:20:31] !log reboot cp32 - swapping intensingly causing high load/io [14:20:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:21:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 108.175.15.182/cpweb, 2607:f1c0:1800:8100::1/cpweb [14:21:50] PROBLEM - cp32 HTTPS on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:59] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 108.175.15.182/cpweb, 2607:f1c0:1800:8100::1/cpweb [14:22:07] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:16] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [14:23:22] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 96% [14:23:42] !log cp32: turn swapoff to see how it copes [14:23:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:23:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:23:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:24:01] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:24:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.47, 3.46, 3.32 [14:24:33] !log reboot cp33 & switch swap off [14:24:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:25:17] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.43, 0.28, 0.12 [14:26:12] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:26:22] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.80, 2.89, 3.12 [14:27:15] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [14:27:22] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 7% [14:27:41] !log reboot cp22 & switch swap off [14:27:41] RECOVERY - cp32 HTTPS on cp32 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.529 second response time [14:27:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:07] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.92, 0.35, 0.13 [14:28:28] PROBLEM - cp33 Puppet on cp33 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:29:54] !log reboot cp23 & switch swap off [14:30:19] PROBLEM - psychevos.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:20] PROBLEM - wiki.simorgh.me - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 11.81, 6.96, 4.68 [14:30:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:31:13] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 11.69, 5.67, 3.08 [14:33:47] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.77, 3.74, 2.97 [14:33:51] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:34:53] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:35:45] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.45, 3.82, 3.09 [14:36:58] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.362 second response time [14:37:43] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.38, 3.50, 3.05 [14:37:49] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.428 second response time [14:38:03] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.91, 3.62, 3.00 [14:38:18] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [14:39:42] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.87, 4.01, 3.29 [14:43:38] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.73, 3.96, 3.45 [14:43:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.35, 4.97, 3.66 [14:45:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.98, 4.96, 3.89 [14:48:46] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 5.49, 4.07, 2.99 [14:49:42] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.41, 3.61, 3.05 [14:50:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.13, 3.62, 2.95 [14:51:05] !log cp*: created /etc/dhcp/dhclient-enter-hooks.d/nodnsupdate to prevent overwritting resolv.conf [14:51:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:52:35] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.88, 3.37, 2.94 [14:53:39] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.58, 3.24, 3.04 [14:55:29] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.93, 3.97, 3.96 [14:59:44] RECOVERY - psychevos.org - LetsEncrypt on sslhost is OK: OK - Certificate 'psychevos.org' will expire on Fri 24 Feb 2023 01:43:05 GMT +0000. [15:00:13] RECOVERY - wiki.simorgh.me - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.simorgh.me' will expire on Fri 24 Feb 2023 05:16:29 GMT +0000. [15:00:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6200fae3a299...dbc512dda927 [15:00:40] [url] Comparing 6200fae3a299...dbc512dda927 · miraheze/puppet · GitHub | github.com [15:00:42] [02miraheze/puppet] 07paladox 03dbc512d - puppetdb: fix firewall rule for icinga [15:01:17] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 3.97, 4.04, 4.01 [15:01:39] RECOVERY - puppet141 puppetdb on puppet141 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:507 port 8081 [15:02:05] RECOVERY - mon141 PowerDNS Recursor on mon141 is OK: DNS OK: 0.024 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [15:05:10] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.73, 3.89, 3.97 [15:11:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/14cfb9b6bb1a [15:11:43] [02miraheze/puppet] 07paladox 0314cfb9b - powerdns: set pdns-distributes-queries to no and reuseport to yes [15:11:46] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [15:11:47] [url] Page not found · GitHub · GitHub | github.com [15:11:47] [02puppet] 07paladox opened pull request 03#3038: powerdns: set pdns-distributes-queries to no and reuseport to yes - 13https://github.com/miraheze/puppet/pull/3038 [15:11:48] [url] Page not found · GitHub · GitHub | github.com [15:14:21] [02puppet] 07paladox synchronize pull request 03#3038: powerdns: set pdns-distributes-queries to no and reuseport to yes - 13https://github.com/miraheze/puppet/pull/3038 [15:14:21] [url] Page not found · GitHub · GitHub | github.com [15:14:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/14cfb9b6bb1a...baa67ea29a8f [15:14:24] [url] Comparing 14cfb9b6bb1a...baa67ea29a8f · miraheze/puppet · GitHub | github.com [15:14:25] [02miraheze/puppet] 07paladox 03baa67ea - Update recursor.conf [15:14:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 7.97, 4.59, 4.06 [15:16:12] [02puppet] 07paladox edited pull request 03#3038: powerdns: set some configs - 13https://github.com/miraheze/puppet/pull/3038 [15:16:12] ... [15:16:17] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.07, 3.41, 3.23 [15:16:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.07, 3.99, 3.92 [15:20:14] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.03, 3.00, 3.13 [15:22:35] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 6.19, 4.25, 3.95 [15:33:05] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.30, 3.34, 3.93 [15:34:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.81, 3.77, 4.00 [15:35:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 6.35, 4.25, 4.17 [15:36:45] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.62, 2.93, 2.61 [15:38:40] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.37, 2.64, 2.54 [15:39:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.65, 3.83, 4.00 [15:40:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 6.53, 4.38, 4.11 [15:42:38] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.18, 9.15, 6.53 [15:43:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.78, 8.86, 6.80 [15:44:35] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 5.32, 7.84, 6.37 [15:45:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.42, 7.56, 6.58 [15:46:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.03, 3.57, 3.92 [15:50:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.32, 3.59, 3.81 [15:51:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.16, 4.41, 4.11 [15:54:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.13, 3.62, 3.80 [15:56:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.26, 4.08, 3.95 [15:58:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.58, 3.84, 3.88 [15:59:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.53, 3.53, 3.90 [16:02:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.26, 3.70, 3.78 [16:04:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.68, 3.39, 3.66 [16:05:05] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 1.93, 2.63, 3.39 [16:06:32] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 1.38, 2.75, 3.40 [16:10:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.52, 3.32, 3.51 [16:12:33] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.61, 3.02, 3.38 [16:13:58] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.17, 3.34, 3.41 [16:15:56] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.11, 3.33, 3.40 [16:20:58] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/b9b5e46101f9 [16:21:00] [02miraheze/mw-config] 07paladox 03b9b5e46 - swift: use redis for file locking [16:21:02] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [16:21:03] [url] Page not found · GitHub · GitHub | github.com [16:21:05] [02mw-config] 07paladox opened pull request 03#4996: swift: use redis for file locking - 13https://github.com/miraheze/mw-config/pull/4996 [16:21:05] ... [16:22:08] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 1.44, 2.74, 3.88 [16:22:14] miraheze/mw-config - paladox the build passed. [16:22:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.65, 4.78, 3.77 [16:24:46] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.60, 3.74, 3.59 [16:26:08] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.29, 1.99, 3.30 [16:26:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.57, 3.88, 3.68 [16:26:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.79, 4.28, 3.81 [16:28:42] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.23, 3.98, 3.76 [16:30:41] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 3.93, 4.07, 3.83 [16:32:39] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.52, 3.48, 3.64 [16:34:33] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.83, 3.08, 3.35 [16:38:34] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 1.84, 2.55, 3.20 [16:42:57] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.65, 3.39, 3.11 [16:44:53] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.46, 2.60, 2.85 [16:46:51] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.71, 3.50, 2.84 [16:48:00] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 7.17, 4.23, 3.19 [16:48:24] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.74, 4.12, 3.58 [16:49:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.73, 3.48, 3.04 [16:50:23] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.69, 3.86, 3.55 [16:50:40] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.14, 3.43, 2.94 [16:51:56] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.90, 3.19, 2.97 [16:52:21] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.13, 3.21, 3.35 [16:52:35] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.51, 3.02, 2.84 [16:53:36] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 8.86, 5.12, 3.89 [17:07:49] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.49, 3.53, 3.16 [17:11:42] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 3.15, 3.37, 3.18 [17:15:59] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.59, 3.73, 3.24 [17:17:58] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.64, 3.58, 3.24 [17:18:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.57, 3.79, 3.39 [17:22:32] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 1.76, 2.97, 3.76 [17:22:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.35, 3.71, 3.45 [17:23:52] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.39, 3.17, 3.19 [17:24:32] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.02, 3.10, 3.25 [17:26:23] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.69, 2.27, 3.29 [17:31:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.80, 3.81, 3.44 [17:32:33] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.01, 3.62, 3.36 [17:33:42] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.84, 3.40, 3.33 [17:34:15] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1036990285 secs [17:34:32] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.58, 3.21, 3.24 [17:37:38] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.08, 4.39, 3.75 [17:38:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.52, 3.89, 3.55 [17:41:34] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.52, 4.00, 3.73 [17:42:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 6.52, 4.80, 3.96 [17:45:31] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.73, 4.70, 4.04 [17:46:09] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.27, 3.42, 3.22 [17:47:27] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:28] PROBLEM - swiftproxy111 SSH on swiftproxy111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.54, 3.87, 3.82 [17:48:07] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 3.35, 3.29, 3.18 [17:48:08] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.32, 3.78, 3.20 [17:48:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 1.87, 3.92, 3.98 [17:48:36] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:48:40] PROBLEM - swiftproxy111 conntrack_table_size on swiftproxy111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:49:23] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:59] PROBLEM - swiftproxy111 ferm_active on swiftproxy111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:50:05] PROBLEM - swiftproxy111 Current Load on swiftproxy111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:50:08] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 6.30, 4.45, 3.50 [17:50:09] !log restart swiftproxy111, OOMs [17:50:15] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09960436821 secs [17:50:39] PROBLEM - swiftproxy111 Disk Space on swiftproxy111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:50:39] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 37 minutes ago with 0 failures [17:50:40] RECOVERY - swiftproxy111 conntrack_table_size on swiftproxy111 is OK: OK: nf_conntrack is 0 % full [17:50:40] PROBLEM - swiftproxy111 NTP time on swiftproxy111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:50:47] PROBLEM - swiftproxy111 PowerDNS Recursor on swiftproxy111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:51:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:51:20] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 400 bytes in 0.025 second response time [17:51:26] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.48, 3.68, 3.70 [17:51:32] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 367 bytes in 0.035 second response time [17:51:33] RECOVERY - swiftproxy111 SSH on swiftproxy111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:51:55] RECOVERY - swiftproxy111 ferm_active on swiftproxy111 is OK: OK ferm input default policy is set [17:52:03] RECOVERY - swiftproxy111 Current Load on swiftproxy111 is OK: OK - load average: 4.15, 2.60, 1.02 [17:52:33] RECOVERY - swiftproxy111 Disk Space on swiftproxy111 is OK: DISK OK - free space: / 16002 MB (56% inode=94%); [17:52:40] RECOVERY - swiftproxy111 NTP time on swiftproxy111 is OK: NTP OK: Offset 0.001239597797 secs [17:52:44] RECOVERY - swiftproxy111 PowerDNS Recursor on swiftproxy111 is OK: DNS OK: 0.072 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [17:53:24] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.46, 3.20, 3.53 [17:55:23] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.52, 2.86, 3.35 [17:56:08] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.51, 3.68, 3.56 [17:56:32] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 1.76, 2.64, 3.35 [17:58:08] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.57, 3.18, 3.38 [18:00:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/d12f67fc9c9a [18:00:31] [02miraheze/puppet] 07paladox 03d12f67f - swiftproxy111: change some memcache params [18:00:33] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [18:00:34] [url] Page not found · GitHub · GitHub | github.com [18:00:34] [02puppet] 07paladox opened pull request 03#3039: swiftproxy111: change some memcache params - 13https://github.com/miraheze/puppet/pull/3039 [18:00:35] ... [18:01:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d12f67fc9c9a...2034736af59c [18:01:30] [url] Comparing d12f67fc9c9a...2034736af59c · miraheze/puppet · GitHub | github.com [18:01:31] [02miraheze/puppet] 07paladox 032034736 - Update memcached.pp [18:01:33] [02puppet] 07paladox synchronize pull request 03#3039: swiftproxy111: change some memcache params - 13https://github.com/miraheze/puppet/pull/3039 [18:01:34] [url] Page not found · GitHub · GitHub | github.com [18:01:47] [02puppet] 07paladox closed pull request 03#3039: swiftproxy111: change some memcache params - 13https://github.com/miraheze/puppet/pull/3039 [18:01:48] [url] Page not found · GitHub · GitHub | github.com [18:01:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/dbc512dda927...438702e07bf9 [18:01:50] [url] Comparing dbc512dda927...438702e07bf9 · miraheze/puppet · GitHub | github.com [18:01:51] [02miraheze/puppet] 07paladox 03438702e - swiftproxy111: change some memcache params (#3039) [18:01:54] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [18:01:57] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [18:01:58] [url] Page not found · GitHub · GitHub | github.com [18:03:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/438702e07bf9...31e80df51591 [18:03:37] [url] Comparing 438702e07bf9...31e80df51591 · miraheze/puppet · GitHub | github.com [18:03:39] [02miraheze/puppet] 07paladox 0331e80df - fix [18:04:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/31e80df51591...ec3602e256a3 [18:04:34] [url] Comparing 31e80df51591...ec3602e256a3 · miraheze/puppet · GitHub | github.com [18:04:35] [02miraheze/puppet] 07paladox 03ec3602e - swiftproxy111: reduce memcached size to 128M [18:05:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ec3602e256a3...13a6c2a01794 [18:05:01] [url] Comparing ec3602e256a3...13a6c2a01794 · miraheze/puppet · GitHub | github.com [18:05:02] [02miraheze/puppet] 07paladox 0313a6c2a - memcached: revert changes [18:06:08] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.10, 3.77, 3.59 [18:06:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/13a6c2a01794...b027b05ae390 [18:06:29] [url] Comparing 13a6c2a01794...b027b05ae390 · miraheze/puppet · GitHub | github.com [18:06:29] [02miraheze/puppet] 07paladox 03b027b05 - Update memcached.pp [18:08:08] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.17, 2.87, 3.28 [18:14:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.02, 3.86, 3.36 [18:19:05] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.66, 3.50, 3.07 [18:20:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.13, 3.53, 3.48 [18:21:05] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.65, 3.04, 2.95 [18:24:32] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.76, 3.10, 3.31 [18:30:27] !log [salt-user@db101] Test Log [18:31:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:31:11] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.64, 3.53, 2.84 [18:33:01] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.12, 4.40, 3.53 [18:33:09] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.91, 4.36, 3.22 [18:37:04] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.75, 3.97, 3.34 [18:38:04] !log [salt-user@db101] Test Log [18:38:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:41:22] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 12.64, 5.98, 4.12 [18:41:38] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5796 bytes in 0.016 second response time [18:42:55] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.41, 5.96, 3.78 [18:43:34] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.581 second response time [18:44:09] !log [universalomega@db101] Test Log [18:44:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:44:56] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.486 second response time [18:45:46] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.85, 3.60, 2.18 [18:45:54] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.75, 3.96, 3.33 [18:46:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.63, 3.70, 3.37 [18:47:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.41, 2.86, 2.08 [18:48:27] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 1.86, 3.48, 3.82 [18:48:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.13, 3.20, 3.22 [18:49:13] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.41, 3.54, 3.90 [18:49:27] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.82, 3.93, 3.38 [18:53:19] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.57, 3.20, 3.20 [18:53:48] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.64, 3.98, 3.66 [18:54:14] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.82, 3.85, 3.88 [18:55:22] !log [universalomega@puppet141] sudo -u www-data rm /srv/mediawiki/cache/*darkangelwiki* on mw*, test131, and mwtask141 [18:55:29] !log [universalomega@db101] MariaDB [mhglobal] FROM cw_wikis WHERE wiki_dbname=darkangelwiki [18:55:35] !log [universalomega@db101] MariaDB [mhglobal] FROM mw_settings WHERE s_dbname=darkangelwiki [18:55:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:55:41] !log [universalomega@db101] MariaDB [mhglobal] FROM mw_namespaces WHERE ns_dbname=darkangelwiki [18:55:49] !log [universalomega@db101] MariaDB [mhglobal] FROM mw_permissions WHERE perm_dbname=darkangelwiki [18:55:54] !log [universalomega@db101] MariaDB [mhglobal] FROM localnames WHERE ln_wiki=darkangelwiki [18:56:01] !log [universalomega@db101] MariaDB [mhglobal] FROM localuser WHERE lu_wiki=darkangelwiki [18:56:02] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 801 bytes in 0.124 second response time [18:56:08] !log [universalomega@db101] MariaDB [metawiki] FROM echo_unread_wikis WHERE euw_wiki=darkangelwiki [18:56:10] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 1.64, 3.08, 3.60 [18:56:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:03] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.66, 2.43, 3.26 [18:57:30] !log [universalomega@db101] MariaDB [mhglobal]> test [18:57:42] PROBLEM - mw122 MediaWiki Rendering on mw122 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 6576 bytes in 0.048 second response time [18:57:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.14, 3.62, 3.55 [18:57:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:08] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.81, 2.58, 3.35 [19:01:06] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.71, 3.81, 3.48 [19:01:33] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.330 second response time [19:01:41] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.95, 3.84, 3.67 [19:03:10] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:04:58] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.21, 3.08, 3.27 [19:07:19] !log [universalomega@puppet141] sudo -u www-data rm /srv/mediawiki/cache/*darkangelwiki* on mw*, test131, and mwtask141 [19:07:19] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM cw_wikis WHERE wiki_dbname='darkangelwiki' [19:07:19] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_settings WHERE s_dbname='darkangelwiki' [19:07:19] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_namespaces WHERE ns_dbname='darkangelwiki' [19:07:19] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_permissions WHERE perm_dbname='darkangelwiki' [19:07:19] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localnames WHERE ln_wiki='darkangelwiki' [19:07:19] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localuser WHERE lu_wiki='darkangelwiki' [19:07:20] !log [universalomega@db101] MariaDB [metawiki]> DELETE FROM echo_unread_wikis WHERE euw_wiki='darkangelwiki' [19:07:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:08] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 3.92, 4.21, 3.62 [19:08:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:09:35] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.25, 3.01, 3.36 [19:09:39] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5796 bytes in 0.373 second response time [19:09:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:09:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:10:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:11:33] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.444 second response time [19:14:41] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.98, 4.08, 3.56 [19:15:30] !log [universalomega@puppet141] sudo -u www-data rm /srv/mediawiki/cache/*unitedbookwiki* on mw*, test131, and mwtask141 [19:15:36] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM cw_wikis WHERE wiki_dbname='unitedbookwiki' [19:15:43] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_settings WHERE s_dbname='unitedbookwiki' [19:15:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:15:49] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_namespaces WHERE ns_dbname='unitedbookwiki' [19:15:54] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_permissions WHERE perm_dbname='unitedbookwiki' [19:15:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:15:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:16:01] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localnames WHERE ln_wiki='unitedbookwiki' [19:16:07] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localuser WHERE lu_wiki='unitedbookwiki' [19:16:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:16:13] !log [universalomega@db101] MariaDB [metawiki]> DELETE FROM echo_unread_wikis WHERE euw_wiki='unitedbookwiki' [19:16:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:16:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:16:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 7.02, 5.04, 4.07 [19:16:32] [discord] Something's happened to the Viva Pinata, Scooby Doo and Wii Sports Wikis: https://wiisports.miraheze.org/wiki/ [19:16:33] [url] Action required | wiisports.miraheze.org [19:16:33] [discord] https://scoobydoo.miraheze.org/wiki/ [19:16:33] [url] Action required | scoobydoo.miraheze.org [19:16:34] [discord] https://vivapinata.miraheze.org/wiki/ [19:16:34] [url] Action required | vivapinata.miraheze.org [19:16:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:16:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:17:33] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.09, 3.90, 3.30 [19:17:51] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.59, 3.53, 3.07 [19:18:08] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 1.55, 3.45, 3.83 [19:18:33] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.46, 3.92, 3.64 [19:19:05] @DarkMatterMan4500 those were on db141 [19:19:30] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.80, 3.45, 3.20 [19:19:49] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 1.92, 2.92, 2.90 [19:20:25] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.78, 3.65, 3.71 [19:20:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.28, 4.01, 3.70 [19:21:28] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.75, 4.02, 3.45 [19:22:08] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.15, 2.63, 3.38 [19:22:12] [discord] Didn't the database somehow update lately? [19:23:47] @DarkMatterMan4500 old wikis are unrecoverable at this time unless wikis had external backups generated unofficially. See #announcements [19:24:29] I am processing recreation requests which resets the wikis entirely at requests. [19:24:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.46, 3.26, 3.47 [19:24:49] !log [universalomega@puppet141] sudo -u www-data rm /srv/mediawiki/cache/*unioflaamwiki* on mw*, test131, and mwtask141 [19:24:57] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM cw_wikis WHERE wiki_dbname='unioflaamwiki' [19:25:02] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_settings WHERE s_dbname='unioflaamwiki' [19:25:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:10] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_namespaces WHERE ns_dbname='unioflaamwiki' [19:25:17] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_permissions WHERE perm_dbname='unioflaamwiki' [19:25:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:22] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localnames WHERE ln_wiki='unioflaamwiki' [19:25:28] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localuser WHERE lu_wiki='unioflaamwiki' [19:25:34] !log [universalomega@db101] MariaDB [metawiki]> DELETE FROM echo_unread_wikis WHERE euw_wiki='unioflaamwiki' [19:25:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.03, 3.85, 3.67 [19:27:21] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.96, 3.81, 3.57 [19:28:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.54, 3.64, 3.61 [19:29:01] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.80, 3.50, 3.56 [19:29:19] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.38, 4.21, 3.72 [19:29:25] !log [universalomega@puppet141] sudo -u www-data rm /srv/mediawiki/cache/*wikimediabotswanawiki* on mw*, test131, and mwtask141 [19:29:31] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM cw_wikis WHERE wiki_dbname='wikimediabotswanawiki' [19:29:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:29:36] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_settings WHERE s_dbname='wikimediabotswanawiki' [19:29:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:29:42] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_namespaces WHERE ns_dbname='wikimediabotswanawiki' [19:29:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:29:49] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_permissions WHERE perm_dbname='wikimediabotswanawiki' [19:29:55] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localnames WHERE ln_wiki='wikimediabotswanawiki' [19:30:01] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localuser WHERE lu_wiki='wikimediabotswanawiki' [19:30:07] !log [universalomega@db101] MariaDB [metawiki]> DELETE FROM echo_unread_wikis WHERE euw_wiki='wikimediabotswanawiki' [19:30:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:30:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:31:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:31:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:31:17] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.68, 3.55, 3.54 [19:31:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:15] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.58, 3.99, 3.72 [19:32:53] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.89, 3.12, 3.37 [19:33:04] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [19:33:14] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.64, 4.30, 3.81 [19:34:13] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.25, 3.75, 3.67 [19:34:26] !log [universalomega@puppet141] sudo -u www-data rm /srv/mediawiki/cache/*projectdiablo2wiki* on mw*, test131, and mwtask141 [19:34:32] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM cw_wikis WHERE wiki_dbname='projectdiablo2wiki' [19:34:32] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.73, 3.09, 3.38 [19:34:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:38] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_settings WHERE s_dbname='projectdiablo2wiki' [19:34:44] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_namespaces WHERE ns_dbname='projectdiablo2wiki' [19:34:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:50] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_permissions WHERE perm_dbname='projectdiablo2wiki' [19:34:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:59] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localnames WHERE ln_wiki='projectdiablo2wiki' [19:35:07] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localuser WHERE lu_wiki='projectdiablo2wiki' [19:35:12] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.34, 3.89, 3.72 [19:35:14] !log [universalomega@db101] MariaDB [metawiki]> DELETE FROM echo_unread_wikis WHERE euw_wiki='projectdiablo2wiki' [19:35:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:35:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:35:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:35:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:36:11] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.27, 3.94, 3.74 [19:36:46] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.86, 3.52, 3.48 [19:38:10] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.96, 3.66, 3.67 [19:38:41] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 6.10, 4.46, 3.84 [19:39:07] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.04, 2.93, 3.37 [19:39:58] !log [universalomega@puppet141] sudo -u www-data rm /srv/mediawiki/cache/*hkrailwiki* on mw*, test131, and mwtask141 [19:40:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:05] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM cw_wikis WHERE wiki_dbname='hkrailwiki' [19:40:08] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.31, 3.69, 3.66 [19:40:10] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_settings WHERE s_dbname='hkrailwiki' [19:40:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:17] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_namespaces WHERE ns_dbname='hkrailwiki' [19:40:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:23] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_permissions WHERE perm_dbname='hkrailwiki' [19:40:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:31] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localnames WHERE ln_wiki='hkrailwiki' [19:40:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:37] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.30, 3.61, 3.60 [19:40:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:39] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localuser WHERE lu_wiki='hkrailwiki' [19:40:47] !log [universalomega@db101] MariaDB [metawiki]> DELETE FROM echo_unread_wikis WHERE euw_wiki='hkrailwiki' [19:40:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:41:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:44:29] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.20, 2.47, 3.16 [19:44:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 6.20, 5.37, 4.27 [19:48:01] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.76, 3.90, 3.92 [19:48:34] !log [universalomega@puppet141] sudo -u www-data rm /srv/mediawiki/cache/*dwspeedrunningwiki* on mw*, test131, and mwtask141 [19:48:41] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM cw_wikis WHERE wiki_dbname='dwspeedrunningwiki' [19:48:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:48] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_settings WHERE s_dbname='dwspeedrunningwiki' [19:48:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:54] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_namespaces WHERE ns_dbname='dwspeedrunningwiki' [19:48:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:00] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_permissions WHERE perm_dbname='dwspeedrunningwiki' [19:49:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:07] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localnames WHERE ln_wiki='dwspeedrunningwiki' [19:49:13] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localuser WHERE lu_wiki='dwspeedrunningwiki' [19:49:20] !log [universalomega@db101] MariaDB [metawiki]> DELETE FROM echo_unread_wikis WHERE euw_wiki='dwspeedrunningwiki' [19:49:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:32] !log [universalomega@puppet141] sudo -u www-data rm /srv/mediawiki/cache/*ucroniaswiki* on mw*, test131, and mwtask141 [19:51:38] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM cw_wikis WHERE wiki_dbname='ucroniaswiki' [19:51:44] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_settings WHERE s_dbname='ucroniaswiki' [19:51:50] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_namespaces WHERE ns_dbname='ucroniaswiki' [19:51:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:56] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_permissions WHERE perm_dbname='ucroniaswiki' [19:52:04] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localnames WHERE ln_wiki='ucroniaswiki' [19:52:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:09] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localuser WHERE lu_wiki='ucroniaswiki' [19:52:15] !log [universalomega@db101] MariaDB [metawiki]> DELETE FROM echo_unread_wikis WHERE euw_wiki='ucroniaswiki' [19:52:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.45, 3.67, 3.91 [19:55:55] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.73, 2.78, 3.35 [19:56:09] !log [universalomega@puppet141] sudo -u www-data rm /srv/mediawiki/cache/*fohsarchivewiki* on mw*, test131, and mwtask141 [19:56:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:18] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM cw_wikis WHERE wiki_dbname='fohsarchivewiki' [19:56:27] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_settings WHERE s_dbname='fohsarchivewiki' [19:56:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:35] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_namespaces WHERE ns_dbname='fohsarchivewiki' [19:56:48] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_permissions WHERE perm_dbname='fohsarchivewiki' [19:57:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:07] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localnames WHERE ln_wiki='fohsarchivewiki' [19:57:15] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localuser WHERE lu_wiki='fohsarchivewiki' [19:57:37] !log [universalomega@db101] MariaDB [metawiki]> DELETE FROM echo_unread_wikis WHERE euw_wiki='fohsarchivewiki' [19:58:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:48] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:01:35] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.05, 5.44, 4.03 [20:02:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.15, 3.10, 3.47 [20:02:51] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 9.33, 5.16, 3.73 [20:02:55] !log [universalomega@puppet141] sudo -u www-data rm /srv/mediawiki/cache/*xomniwiki* on mw*, test131, and mwtask141 [20:03:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:01] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM cw_wikis WHERE wiki_dbname='xomniwiki' [20:03:08] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_settings WHERE s_dbname='xomniwiki' [20:03:14] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_namespaces WHERE ns_dbname='xomniwiki' [20:03:19] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_permissions WHERE perm_dbname='xomniwiki' [20:03:25] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localnames WHERE ln_wiki='xomniwiki' [20:03:31] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localuser WHERE lu_wiki='xomniwiki' [20:03:37] !log [universalomega@db101] MariaDB [metawiki]> DELETE FROM echo_unread_wikis WHERE euw_wiki='xomniwiki' [20:04:32] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.31, 2.83, 3.33 [20:05:40] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.324 second response time [20:05:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:07] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:06:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:07:41] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw141 [20:07:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.05, 7.92, 3.97 [20:08:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:08:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.81, 3.73, 3.59 [20:08:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:09:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:09:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:09:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.34, 2.72 [20:10:18] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.518 second response time [20:10:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.00, 3.68, 3.61 [20:11:41] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [20:11:52] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:11:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.93, 3.61, 2.90 [20:13:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.75, 3.16, 3.10 [20:13:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.77, 3.87, 3.08 [20:16:33] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.31, 3.76, 3.66 [20:17:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.48, 3.25, 3.07 [20:18:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.26, 3.77, 3.17 [20:18:33] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.36, 3.57, 3.14 [20:45:23] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.67, 3.61, 3.48 [20:46:16] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=xomniwiki --no-updates --username-prefix=mh:xomni /home/reception/ImportDump/xomniwiki-20221130202129.xml (END - exit=0) [20:46:17] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=xomniwiki (START) [20:46:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:46:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:46:42] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.11, 2.49, 3.30 [20:47:36] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildImages.php --wiki=forecastingwiki --missing (END - exit=0) [20:48:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:50:44] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b027b05ae390...89c962cbb96c [20:50:45] [url] Comparing b027b05ae390...89c962cbb96c · miraheze/puppet · GitHub | github.com [20:50:45] [02miraheze/puppet] 07Universal-Omega 0389c962c - mwscript: add rebuildImages.php to long scripts [20:51:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 7.61, 4.75, 3.91 [20:55:15] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildImages.php --wiki=darkangelwiki --missing (END - exit=0) [20:55:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:55:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/15340adea052 [20:55:49] [02miraheze/mw-config] 07Universal-Omega 0315340ad - Re-enable polcomwiki foreign file repos [20:55:51] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [20:55:51] [url] Page not found · GitHub · GitHub | github.com [20:55:53] [02mw-config] 07Universal-Omega opened pull request 03#4997: Re-enable polcomwiki foreign file repos - 13https://github.com/miraheze/mw-config/pull/4997 [20:55:53] [url] Page not found · GitHub · GitHub | github.com [20:56:53] miraheze/mw-config - Universal-Omega the build passed. [21:00:08] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.02, 2.82, 3.78 [21:01:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.32, 3.44, 3.38 [21:02:48] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.23, 3.54, 3.81 [21:03:05] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.01, 3.38, 3.37 [21:04:08] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.19, 2.37, 3.37 [21:04:55] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=xomniwiki (END - exit=0) [21:04:56] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=xomniwiki --active --update (END - exit=0) [21:05:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:05:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.32, 3.58, 3.63 [21:13:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.41, 3.89, 3.54 [21:18:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.34, 3.42, 3.61 [21:19:42] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildImages.php --wiki=polcomwiki --missing (START) [21:19:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:24:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.30, 3.43, 3.53 [21:25:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.02, 3.87, 3.81 [21:26:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.06, 3.27, 3.46 [21:26:37] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.21, 4.97, 3.86 [21:28:32] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.99, 3.12, 3.37 [21:31:05] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.32, 4.52, 3.53 [21:32:24] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.67, 3.30, 3.43 [21:33:04] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.83, 2.87, 3.35 [21:36:16] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 6.00, 4.45, 3.86 [21:36:58] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.76, 3.31, 3.40 [21:40:51] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.85, 3.75, 3.60 [21:42:49] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.29, 4.06, 3.71 [21:47:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 3.45, 4.01, 3.51 [21:47:05] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=lgbtawiki --no-updates --username-prefix=mh:lgbta /home/universalomega/lgbtamirahezeorg_w-20220626-history.xml (START) [21:47:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.17, 3.92, 3.45 [21:47:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.99, 3.63, 3.43 [21:49:08] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.54, 3.78, 3.45 [21:49:18] [02mw-config] 07Universal-Omega closed pull request 03#4997: Re-enable polcomwiki foreign file repos - 13https://github.com/miraheze/mw-config/pull/4997 [21:49:19] [url] Page not found · GitHub · GitHub | github.com [21:49:19] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [21:49:21] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [21:49:22] [url] Page not found · GitHub · GitHub | github.com [21:49:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9ece3a7b09c3...32ee58d51f2c [21:49:25] [url] Comparing 9ece3a7b09c3...32ee58d51f2c · miraheze/mw-config · GitHub | github.com [21:49:27] [02miraheze/mw-config] 07Universal-Omega 0332ee58d - Re-enable polcomwiki foreign file repos (#4997) [21:50:24] miraheze/mw-config - Universal-Omega the build passed. [21:51:04] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 1.63, 3.09, 3.24 [21:51:04] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.36, 3.12, 3.26 [21:51:20] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildImages.php --wiki=lgbtawiki --missing (START) [21:51:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:52:44] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.25, 3.99, 3.95 [21:57:14] !log [@test131] starting deploy of {'config': True} to all [21:57:15] !log [@test131] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [21:57:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:57:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:00:28] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [22:01:20] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=lgbtawiki --no-updates --username-prefix=mh:lgbta /home/universalomega/lgbtamirahezeorg_w-20220626-history.xml (END - exit=0) [22:01:21] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=lgbtawiki (START) [22:01:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:01:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:02:33] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.10, 3.37, 3.58 [22:03:47] [02miraheze/ErrorPages] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/commit/575f519ab405 [22:03:50] [02miraheze/ErrorPages] 07paladox 03575f519 - fix "Undefined variable: wgCommandLineMode" [22:03:53] [02ErrorPages] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/ErrorPages [22:03:54] [url] Page not found · GitHub · GitHub | github.com [22:03:56] [02ErrorPages] 07paladox opened pull request 03#21: fix "Undefined variable: wgCommandLineMode" - 13https://github.com/miraheze/ErrorPages/pull/21 [22:03:57] ... [22:04:30] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 4.00, 3.97, 3.79 [22:04:46] miraheze/ErrorPages - paladox the build passed. [22:05:47] [02ErrorPages] 07Universal-Omega closed pull request 03#21: fix "Undefined variable: wgCommandLineMode" - 13https://github.com/miraheze/ErrorPages/pull/21 [22:05:48] ... [22:05:49] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/3adc23d9cfec...b8933a8206e3 [22:05:50] [url] Comparing 3adc23d9cfec...b8933a8206e3 · miraheze/ErrorPages · GitHub | github.com [22:05:51] [02miraheze/ErrorPages] 07paladox 03b8933a8 - fix "Undefined variable: wgCommandLineMode" (#21) [22:06:24] !log [paladox@mwtask141] starting deploy of {'pull': 'errorpages', 'errorpages': True} to all [22:06:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:33] !log [paladox@mwtask141] finished deploy of {'pull': 'errorpages', 'errorpages': True} to all - SUCCESS in 8s [22:06:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:40] miraheze/ErrorPages - Universal-Omega the build passed. [22:06:40] !log [paladox@mwtask141] starting deploy of {'errorpages': True} to all [22:06:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:46] !log [paladox@mwtask141] finished deploy of {'errorpages': True} to all - SUCCESS in 6s [22:06:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:07:05] !log [paladox@test131] starting deploy of {'pull': 'errorpages', 'errorpages': True} to test131 [22:07:07] !log [paladox@test131] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [22:07:17] !log [paladox@test131] starting deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True} to test131 [22:07:18] !log [paladox@test131] finished deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True} to test131 - SUCCESS in 0s [22:07:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:07:26] !log [paladox@test131] starting deploy of {'errorpages': True, 'force': True} to test131 [22:07:27] !log [paladox@test131] finished deploy of {'errorpages': True, 'force': True} to test131 - SUCCESS in 0s [22:07:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:07:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:07:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:07:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:08:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:09:57] !log [@mwtask141] starting deploy of {'config': True} to all [22:10:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:10:09] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 12s [22:10:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:10:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.68, 2.81, 3.35 [22:16:14] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.76, 4.62, 3.90 [22:18:11] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.94, 3.55, 3.59 [22:22:10] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.73, 3.12, 3.39 [22:24:08] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=lgbtawiki (END - exit=0) [22:24:09] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=lgbtawiki --active --update (END - exit=0) [22:24:11] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.14, 3.20, 2.61 [22:24:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:24:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 1.99, 3.27, 3.96 [22:24:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:26:09] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.86, 3.13, 2.67 [22:26:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.94, 3.88, 3.35 [22:28:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.46, 3.72, 3.95 [22:28:47] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:30:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.73, 3.42, 3.80 [22:32:22] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.29, 3.66, 3.84 [22:32:32] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.92, 3.34, 3.26 [22:42:36] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.97, 3.34, 2.85 [22:44:35] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.11, 3.17, 2.84 [22:52:55] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:54:40] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 1 backends are down. mw121 [22:55:15] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5796 bytes in 1.832 second response time [22:56:40] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [22:56:47] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.337 second response time [22:57:11] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.248 second response time [23:05:36] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.41, 2.71, 2.29 [23:07:34] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.38, 2.52, 2.27 [23:15:27] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.41, 3.82, 3.35 [23:16:52] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=polandballwiki --no-updates --username-prefix=wikia:polandball --namespaces=0 (START) [23:16:53] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=polandballwiki --no-updates --username-prefix=wikia:polandball --namespaces=0 (END - exit=65280) [23:16:54] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=polandballwiki (START) [23:16:58] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=polandballwiki (END - exit=0) [23:17:00] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=polandballwiki --active --update (END - exit=0) [23:17:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:17:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:17:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:17:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:06] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=polandballwiki --no-updates --username-prefix=wikia:polandball --namespaces=0|8|10|14 /home/universalomega/polandball_pages_full.xml (START) [23:18:07] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=polandballwiki --no-updates --username-prefix=wikia:polandball --namespaces=0|8|10|14 /home/universalomega/polandball_pages_full.xml (END - exit=32512) [23:18:08] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=polandballwiki (START) [23:18:09] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=polandballwiki (END - exit=0) [23:18:10] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=polandballwiki --active --update (END - exit=0) [23:18:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:20] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 3.24, 3.37, 3.25 [23:19:53] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=polandballwiki --no-updates --username-prefix=wikia:polandball --namespaces=0 (START) [23:19:54] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=polandballwiki --no-updates --username-prefix=wikia:polandball --namespaces=0 (END - exit=65280) [23:20:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:20:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:23:52] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=polandballwiki --no-updates --username-prefix=wikia:polandball --namespaces="0|8|10|14" /home/universalomega/polandball_pages_full.xml [23:24:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:01] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.19, 3.76, 3.47 [23:31:36] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildImages.php --wiki=polandballwiki --missing (START) [23:31:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:32:57] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.72, 3.59, 3.43 [23:34:54] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 1.80, 2.96, 3.22 [23:41:43] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildImages.php --wiki=lgbtawiki --missing (END - exit=0) [23:41:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:44:36] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 7.30, 5.09, 3.91 [23:46:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.85, 3.38, 2.87 [23:48:07] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 3.17, 3.14, 2.84 [23:54:55] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.30, 4.13, 3.32 [23:56:30] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 3.98, 4.35, 3.43 [23:58:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.33, 3.71, 3.32 [23:58:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.30, 3.79, 3.99 [23:58:51] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.39, 3.55, 3.30