[00:07:10] PROBLEM - cp32 Disk Space on cp32 is CRITICAL: DISK CRITICAL - free space: / 4438 MB (5% inode=98%); [00:11:10] RECOVERY - cp32 Disk Space on cp32 is OK: DISK OK - free space: / 13222 MB (17% inode=98%); [00:15:54] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 9.77, 6.20, 3.19 [00:17:31] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.93, 3.64, 3.06 [00:19:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.54, 3.19 [00:19:28] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.89, 3.03, 2.91 [00:19:51] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.57, 3.10, 2.60 [00:21:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.46, 2.79, 2.96 [00:29:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.18, 2.99 [00:31:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.23, 2.78, 2.87 [00:31:20] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.05, 3.12, 2.09 [00:33:20] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.47, 2.55, 2.02 [01:31:20] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.49, 3.00, 2.00 [01:35:20] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.42, 3.43, 2.47 [01:37:20] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.48, 2.34, 2.18 [03:06:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.61, 9.53, 7.93 [03:08:17] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.43, 8.41, 7.70 [03:11:04] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 5.50, 4.17, 2.99 [03:13:04] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.78, 3.98, 3.07 [03:23:04] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.05, 3.70, 3.34 [03:25:04] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.00, 3.00, 3.13 [03:29:04] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.59, 3.77, 3.39 [03:31:04] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.91, 3.68, 3.40 [03:33:04] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.67, 3.89, 3.50 [03:35:13] PROBLEM - test131 NTP time on test131 is CRITICAL: NTP CRITICAL: Offset 0.6798706353 secs [03:37:04] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.95, 3.92, 3.59 [03:39:04] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 5.62, 4.40, 3.80 [03:45:04] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.87, 3.55, 3.71 [03:45:31] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 314 bytes in 0.017 second response time [03:49:04] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.10, 2.58, 3.26 [03:49:18] RECOVERY - test131 NTP time on test131 is OK: NTP OK: Offset -0.02714955807 secs [04:33:25] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.25, 4.40, 2.64 [04:39:20] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.63, 2.85, 2.67 [05:01:20] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.88, 2.81, 2.10 [05:03:20] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.87, 2.65, 2.15 [05:30:04] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.17, 3.13, 3.83 [05:34:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.46, 3.56, 3.80 [05:42:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.01, 3.57, 3.91 [05:44:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.33, 4.23, 4.10 [05:50:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.60, 3.09, 3.75 [05:56:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.68, 2.41, 3.23 [06:03:01] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.85, 5.37, 4.14 [06:06:55] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.41, 3.78, 3.76 [06:08:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.19, 3.78, 3.75 [06:10:50] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.27, 3.79, 3.77 [06:15:51] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.03, 3.66, 2.21 [06:17:51] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.25, 2.86, 2.10 [06:20:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.41, 2.80, 3.31 [06:26:15] PROBLEM - smashbros.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:29] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.29, 6.56, 4.58 [06:28:54] PROBLEM - cp22 PowerDNS Recursor on cp22 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:29:21] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 9.55, 9.31, 4.95 [06:29:56] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 9.93, 10.46, 5.66 [06:30:52] RECOVERY - cp22 PowerDNS Recursor on cp22 is OK: DNS OK: 0.269 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [06:37:20] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.41, 3.12, 3.75 [06:37:51] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.47, 2.84, 3.69 [06:39:20] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.37, 2.18, 3.32 [06:40:12] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.37, 3.49, 3.92 [06:41:51] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.84, 2.01, 3.17 [06:44:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.83, 3.39, 3.72 [06:46:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.59, 3.27, 3.62 [06:50:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.11, 4.00, 3.81 [06:52:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.33, 3.62, 3.69 [06:56:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.22, 4.15, 3.89 [06:56:05] PROBLEM - smashbros.miraheze.org - Sectigo on sslhost is WARNING: WARNING - Certificate '*.miraheze.org' expires in 15 day(s) (Wed 09 Nov 2022 23:59:59 GMT +0000). [06:59:20] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.83, 3.61, 3.03 [07:02:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.57, 3.35, 3.71 [07:03:20] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.85, 2.89, 2.95 [07:06:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.70, 2.50, 3.28 [07:55:05] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.41, 3.48, 2.84 [07:57:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.59, 2.81, 2.68 [08:31:18] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.80, 3.43, 2.46 [08:39:06] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.75, 3.93, 3.13 [08:43:00] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.61, 2.81, 2.86 [08:44:17] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.85, 2.76, 1.80 [08:46:13] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.43, 2.26, 1.73 [10:42:24] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1643195748 secs [10:59:20] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.54, 3.70, 2.35 [11:01:20] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.88, 2.98, 2.26 [11:06:23] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.05019286275 secs [11:20:53] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1108870208 secs [11:24:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.80, 3.20, 2.48 [11:26:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.42, 2.76, 2.39 [11:43:08] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.57, 12.16, 7.60 [11:43:35] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 10.45, 12.59, 8.15 [11:44:34] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 7.25, 10.48, 7.05 [11:45:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.73, 10.32, 7.46 [11:45:30] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 6.70, 10.29, 7.81 [11:46:34] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 4.34, 8.32, 6.68 [11:47:05] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.95, 8.58, 7.17 [11:47:25] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.54, 8.75, 7.54 [11:50:53] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08111318946 secs [11:53:23] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.70, 3.57, 3.02 [11:57:17] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.13, 4.31, 3.42 [11:59:15] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.16, 3.36, 3.17 [12:01:51] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.09, 3.05, 2.05 [12:03:51] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.63, 2.35, 1.91 [12:05:09] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.33, 4.16, 3.63 [12:09:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.77, 3.84, 3.67 [12:11:01] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.84, 4.76, 4.06 [12:18:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.36, 3.51, 3.90 [12:24:41] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.38, 2.53, 3.39 [12:33:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.30, 3.48, 3.47 [12:35:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.88, 4.21, 3.72 [12:39:25] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.03, 3.49, 3.59 [12:43:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.86, 2.81, 3.29 [12:43:58] PROBLEM - mwtask141 Puppet on mwtask141 is WARNING: WARNING: Puppet last ran 1 hour ago [12:49:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.38, 3.83, 2.32 [12:50:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.28, 4.17, 2.44 [12:51:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.49, 3.46, 2.12 [12:52:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.90, 2.56 [12:53:12] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.39, 3.80, 3.55 [12:54:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.26, 4.42, 2.94 [12:55:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.42, 3.64, 2.51 [12:58:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.04, 3.55, 2.95 [12:59:03] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.95, 3.26, 2.61 [12:59:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.92, 3.57, 3.57 [12:59:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.31, 3.77, 3.23 [13:00:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.63, 3.00, 2.83 [13:01:00] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.83, 2.97, 3.35 [13:03:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.16, 3.20, 3.15 [13:06:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.88, 4.83, 3.91 [13:14:45] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.02, 3.51, 3.73 [13:16:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.49, 3.25 [13:18:24] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1858763993 secs [13:18:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.97, 2.87, 3.05 [13:20:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.38, 3.58, 3.60 [13:22:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.22, 3.46, 3.56 [13:26:28] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.10, 2.89, 3.29 [13:34:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.07, 3.25, 3.04 [13:36:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.41, 3.31, 3.09 [13:36:17] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.88, 3.35, 3.19 [13:38:13] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.24, 3.09, 3.05 [13:40:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.41, 3.01, 3.10 [13:40:24] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.04690614343 secs [13:43:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.19, 9.67, 8.71 [13:45:04] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.12, 9.51, 8.77 [13:47:38] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.15, 3.14, 2.30 [13:49:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.51, 4.61, 3.89 [13:49:33] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 3.17, 3.32, 2.48 [14:00:24] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1904608309 secs [14:06:52] !log rerun wikibackups due to error [14:06:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:08:42] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wolvdenwiki --no-updates --username-prefix=wikia:wolvden /mnt/mediawiki-static/metawiki/ImportDump/wolvdenwiki-20221024031355.xml (START) [14:08:43] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wolvdenwiki --no-updates --username-prefix=wikia:wolvden /mnt/mediawiki-static/metawiki/ImportDump/wolvdenwiki-20221024031355.xml (END - exit=65280) [14:08:44] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=wolvdenwiki (START) [14:08:45] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=wolvdenwiki (END - exit=65280) [14:08:46] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=wolvdenwiki --active --update (END - exit=65280) [14:08:49] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spidersonamultiversewiki --no-updates --username-prefix=wikia:spidersona-multiverse /mnt/mediawiki-static/metawiki/ImportDump/spidersonamultiversewiki-20221024230019.xml (START) [14:08:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:08:50] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spidersonamultiversewiki --no-updates --username-prefix=wikia:spidersona-multiverse /mnt/mediawiki-static/metawiki/ImportDump/spidersonamultiversewiki-20221024230019.xml (END - exit=65280) [14:08:51] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=spidersonamultiversewiki (START) [14:08:52] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=spidersonamultiversewiki (END - exit=65280) [14:08:53] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=spidersonamultiversewiki --active --update (END - exit=65280) [14:08:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:09:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:09:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:09:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:09:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:09:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:09:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:09:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:09:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:12:23] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.05110165477 secs [14:17:30] PROBLEM - forecasting.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query forecasting.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:19:58] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:20:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.08, 3.19, 3.92 [14:21:32] RECOVERY - mwtask141 Disk Space on mwtask141 is OK: DISK OK - free space: / 19592 MB (27% inode=90%); [14:22:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.47, 5.81, 4.88 [14:22:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.27, 2.77 [14:23:21] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wolvdenwiki --no-updates --username-prefix=wikia:wolvden /mnt/mediawiki-static/metawiki/ImportDump/wolvdenwiki-20221024031355.xml (START) [14:23:22] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wolvdenwiki --no-updates --username-prefix=wikia:wolvden /mnt/mediawiki-static/metawiki/ImportDump/wolvdenwiki-20221024031355.xml (END - exit=65280) [14:23:23] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=wolvdenwiki (START) [14:23:24] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=wolvdenwiki (END - exit=65280) [14:23:25] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=wolvdenwiki --active --update (END - exit=65280) [14:23:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:23:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:23:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:24:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:24:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:24:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.71, 3.03, 2.74 [14:26:53] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wolvdenwiki --no-updates --username-prefix=wikia:wolvden /mnt/mediawiki-static/metawiki/ImportDump/wolvdenwiki-20221024031355.xml (START) [14:26:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:27:04] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spidersonamultiversewiki --no-updates --username-prefix=wikia:spidersona-multiverse /mnt/mediawiki-static/metawiki/ImportDump/spidersonamultiversewiki-20221024230019.xml (START) [14:27:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:04] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spidersonamultiversewiki --no-updates --username-prefix=wikia:spidersona-multiverse /mnt/mediawiki-static/metawiki/ImportDump/spidersonamultiversewiki-20221024230019.xml (END - exit=0) [14:28:05] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=spidersonamultiversewiki (START) [14:28:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:12] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=spidersonamultiversewiki (END - exit=0) [14:28:13] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=spidersonamultiversewiki --active --update (END - exit=0) [14:28:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:33:05] [02ssl] 07paladox synchronize pull request 03#610: renew miraheze.org wildcard - 13https://github.com/miraheze/ssl/pull/610 [14:33:08] [02miraheze/ssl] 07paladox pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3c0ffa400162...dfc22411c683 [14:33:10] [02miraheze/ssl] 07paladox 03dfc2241 - Update wildcard.miraheze.org-2020-2.crt [14:40:43] PROBLEM - es131 Puppet on es131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [14:40:44] PROBLEM - bast121 Puppet on bast121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 31 minutes ago with 0 failures [14:40:48] PROBLEM - es141 Puppet on es141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [14:40:51] PROBLEM - cloud13 Puppet on cloud13 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 20 minutes ago with 0 failures [14:40:52] PROBLEM - swiftac111 Puppet on swiftac111 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 18 minutes ago with 0 failures [14:41:04] PROBLEM - mon141 Puppet on mon141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 18 minutes ago with 0 failures [14:41:07] PROBLEM - mw131 Puppet on mw131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 28 minutes ago with 0 failures [14:41:10] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [14:41:11] PROBLEM - mw141 Puppet on mw141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [14:41:15] PROBLEM - db131 Puppet on db131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [14:41:16] PROBLEM - cp33 Puppet on cp33 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 16 minutes ago with 0 failures [14:41:17] PROBLEM - cp32 Puppet on cp32 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [14:41:20] PROBLEM - phab121 Puppet on phab121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [14:41:20] [02ssl] 07paladox closed pull request 03#610: renew miraheze.org wildcard - 13https://github.com/miraheze/ssl/pull/610 [14:41:23] [02miraheze/ssl] 07paladox deleted branch 03Reception123-patch-1 [14:41:25] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c108848f05d9...ebd29d2ca3e0 [14:41:26] [02miraheze/ssl] 07Reception123 03ebd29d2 - renew miraheze.org wildcard (#610) [14:41:27] [02ssl] 07paladox deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/ssl [14:41:32] PROBLEM - swiftobject112 Puppet on swiftobject112 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 26 minutes ago with 0 failures [14:41:34] PROBLEM - db121 Puppet on db121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 22 minutes ago with 0 failures [14:41:34] PROBLEM - bast101 Puppet on bast101 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [14:41:35] PROBLEM - puppet141 Puppet on puppet141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [14:41:35] PROBLEM - gluster102 Puppet on gluster102 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [14:41:35] PROBLEM - db112 Puppet on db112 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [14:41:36] PROBLEM - cloud14 Puppet on cloud14 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 21 minutes ago with 0 failures [14:41:36] PROBLEM - cloud11 Puppet on cloud11 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 21 minutes ago with 0 failures [14:41:37] PROBLEM - cloud11 Puppet on cloud11 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 21 minutes ago with 0 failures [14:41:37] PROBLEM - mw142 Puppet on mw142 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [14:41:37] PROBLEM - mem101 Puppet on mem101 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 22 minutes ago with 0 failures [14:41:41] PROBLEM - mw132 Puppet on mw132 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [14:41:42] PROBLEM - swiftobject113 Puppet on swiftobject113 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 28 minutes ago with 0 failures [14:41:44] PROBLEM - mem131 Puppet on mem131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 17 minutes ago with 0 failures [14:41:46] PROBLEM - db101 Puppet on db101 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 21 minutes ago with 0 failures [14:41:49] PROBLEM - cp23 Puppet on cp23 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [14:41:55] PROBLEM - test131 Puppet on test131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [14:41:55] PROBLEM - gluster122 Puppet on gluster122 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 32 minutes ago with 0 failures [14:41:58] PROBLEM - mwtask141 Puppet on mwtask141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [14:41:59] PROBLEM - swiftobject115 Puppet on swiftobject115 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [14:42:00] PROBLEM - swiftproxy131 Puppet on swiftproxy131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [14:42:00] PROBLEM - cloud12 Puppet on cloud12 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [14:42:03] PROBLEM - graylog121 Puppet on graylog121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [14:42:07] PROBLEM - swiftobject114 Puppet on swiftobject114 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 23 minutes ago with 0 failures [14:42:08] PROBLEM - mw122 Puppet on mw122 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [14:42:12] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 30 minutes ago with 0 failures [14:42:20] PROBLEM - gluster121 Puppet on gluster121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [14:42:21] PROBLEM - mail121 Puppet on mail121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 30 minutes ago with 0 failures [14:42:22] PROBLEM - gluster101 Puppet on gluster101 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 minutes ago with 0 failures [14:42:29] PROBLEM - jobchron121 Puppet on jobchron121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 26 minutes ago with 0 failures [14:42:32] PROBLEM - cp22 Puppet on cp22 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 28 minutes ago with 0 failures [14:42:33] PROBLEM - db141 Puppet on db141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [14:42:34] PROBLEM - ns2 Puppet on ns2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 24 minutes ago with 0 failures [14:42:36] PROBLEM - ldap141 Puppet on ldap141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [14:42:36] PROBLEM - matomo131 Puppet on matomo131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 17 minutes ago with 0 failures [14:42:37] PROBLEM - reports121 Puppet on reports121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [14:42:40] PROBLEM - swiftobject111 Puppet on swiftobject111 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [14:42:41] PROBLEM - cloud10 Puppet on cloud10 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [14:42:48] PROBLEM - prometheus131 Puppet on prometheus131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 19 minutes ago with 0 failures [14:42:52] PROBLEM - mw121 Puppet on mw121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 31 minutes ago with 0 failures [14:43:46] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:46:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.26, 2.82 [14:46:30] RECOVERY - forecasting.wiki - reverse DNS on sslhost is OK: SSL OK - forecasting.wiki reverse DNS resolves to cp23.miraheze.org - CNAME FLAT [14:48:26] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.76, 2.93, 2.75 [14:48:46] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wolvdenwiki --no-updates --username-prefix=wikia:wolvden /mnt/mediawiki-static/metawiki/ImportDump/wolvdenwiki-20221024031355.xml (END - exit=0) [14:48:47] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=wolvdenwiki (START) [14:49:35] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:50:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.66, 3.39, 3.93 [14:51:03] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=wolvdenwiki (END - exit=0) [14:51:04] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=wolvdenwiki --active --update (END - exit=0) [14:51:14] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:51:33] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:52:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.00, 4.32, 4.25 [14:52:33] RECOVERY - db141 Puppet on db141 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [14:52:52] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:53:41] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:54:08] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:55:07] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [14:55:36] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:56:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.04, 3.43, 3.88 [14:57:11] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [14:57:59] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:58:20] RECOVERY - gluster121 Puppet on gluster121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:58:22] RECOVERY - gluster101 Puppet on gluster101 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:59:35] RECOVERY - gluster102 Puppet on gluster102 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:59:56] RECOVERY - gluster122 Puppet on gluster122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:02:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.16, 2.56, 3.34 [15:02:31] RECOVERY - cp22 Puppet on cp22 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:03:15] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:03:18] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:03:52] RECOVERY - cp23 Puppet on cp23 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:06:06] RECOVERY - cp23 Puppet on cp23 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [15:06:21] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [15:06:22] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures [15:06:22] RECOVERY - cp22 Puppet on cp22 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:48] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [15:06:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.01, 4.00, 3.75 [15:07:59] RECOVERY - prometheus131 Puppet on prometheus131 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [15:08:26] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:44] RECOVERY - bast101 Puppet on bast101 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [15:08:46] RECOVERY - puppet141 Puppet on puppet141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:49] RECOVERY - matomo131 Puppet on matomo131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:54] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:09] RECOVERY - es141 Puppet on es141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:16] RECOVERY - es131 Puppet on es131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:50] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.93, 3.57, 3.63 [15:12:48] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.02, 4.12, 3.83 [15:14:21] RECOVERY - graylog121 Puppet on graylog121 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [15:15:58] RECOVERY - taswin.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:15:59] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:16:09] RECOVERY - animefeet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:16:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.97, 4.05, 2.99 [15:16:19] RECOVERY - yokaiwatch.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:16:35] RECOVERY - schizoidnightmares.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:16:47] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.52, 5.55, 3.44 [15:17:00] RECOVERY - metroid.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:17:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.89, 9.88, 8.59 [15:18:13] RECOVERY - cnt.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:18:17] RECOVERY - hellointernet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:18:22] RECOVERY - sims.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:18:44] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.09, 3.91, 3.10 [15:19:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.06, 3.69, 2.76 [15:19:35] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.51, 4.03, 2.76 [15:19:44] RECOVERY - civclassic.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:19:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.45, 9.28, 8.52 [15:19:50] RECOVERY - unmade.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:20:03] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [15:20:25] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:20:41] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.66, 2.82, 2.79 [15:20:50] RECOVERY - piwik.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:21:30] RECOVERY - test3.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:21:30] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.57, 3.68, 2.77 [15:21:37] RECOVERY - finalfantasy.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:22:04] RECOVERY - animatedfeet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:22:21] RECOVERY - mem131 Puppet on mem131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:22:31] RECOVERY - mem101 Puppet on mem101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:22:38] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [15:22:47] RECOVERY - erawikisource.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:22:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.91, 3.10 [15:23:01] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.96, 3.68, 2.69 [15:23:18] RECOVERY - allthetropes.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:23:25] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.70, 3.76, 2.89 [15:23:47] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [15:23:48] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:04] RECOVERY - croc.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:24:26] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:28] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:41] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:42] RECOVERY - test2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:24:54] RECOVERY - nintendo.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:24:55] RECOVERY - pokemon2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:25:13] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [15:25:20] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.89, 3.85, 3.03 [15:25:25] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [15:25:35] RECOVERY - swiftobject115 Puppet on swiftobject115 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:25:50] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [15:25:51] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [15:25:53] RECOVERY - swiftproxy131 Puppet on swiftproxy131 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [15:25:54] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [15:25:57] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:25:58] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [15:26:01] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [15:26:05] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.39, 3.18, 2.33 [15:26:05] RECOVERY - smashbros.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:26:12] RECOVERY - swiftobject114 Puppet on swiftobject114 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [15:26:24] RECOVERY - recaptime.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:27:05] !log restart varnish on cp33, seemed to have crashed on the 22nd of ocotober? [15:27:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:27:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.31, 11.03, 9.73 [15:27:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:27:16] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 3.61, 4.13, 3.25 [15:27:23] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 4% [15:27:26] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:27:38] RECOVERY - uncyclopediakr.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:27:41] RECOVERY - dc.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:28:00] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.61, 2.87, 2.31 [15:28:01] RECOVERY - cp33 HTTPS on cp33 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.184 second response time [15:28:22] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [15:29:10] RECOVERY - mario.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:29:12] RECOVERY - projectsekai.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:29:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.69, 10.40, 9.31 [15:30:11] RECOVERY - beta.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:30:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.83, 3.34 [15:30:30] RECOVERY - dragonquest2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:31:17] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 29415 bytes in 1.864 second response time [15:31:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.81, 11.10, 9.65 [15:32:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.66, 3.34 [15:34:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.64, 3.35 [15:34:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.35, 11.88, 10.67 [15:34:55] RECOVERY - rare.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:34:56] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.53, 3.57, 3.42 [15:35:12] RECOVERY - commonswiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:35:15] RECOVERY - dcmultiverse.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:35:36] RECOVERY - template.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:35:42] RECOVERY - wiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:35:53] RECOVERY - closinglogosgroup.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:36:17] RECOVERY - crashspyro.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:36:32] RECOVERY - polandball.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:36:51] RECOVERY - test1.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:36:52] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.60, 2.98, 3.22 [15:37:29] RECOVERY - donkeykong.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:37:33] RECOVERY - intp.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:37:39] RECOVERY - houkai2nd.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [15:38:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.60, 11.86, 10.87 [15:39:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.64, 3.56, 3.63 [15:39:39] PROBLEM - cp32 Disk Space on cp32 is WARNING: DISK WARNING - free space: / 8227 MB (10% inode=98%); [15:39:42]