[00:00:10] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.78, 4.12, 3.20 [00:02:05] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.22, 3.68, 3.15 [00:03:31] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.79, 3.66, 2.82 [00:05:25] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.54, 3.21, 2.75 [00:05:55] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.39, 3.30, 3.09 [00:09:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.21, 4.13, 2.97 [00:13:36] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.64, 3.53, 2.28 [00:19:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.18, 3.25, 3.26 [00:23:49] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 32.12, 15.03, 8.40 [00:24:59] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 14.64, 13.63, 8.47 [00:26:19] ugh i just found out my personal wiki is affected by the db141 outages [00:26:59] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 6.93, 10.93, 8.10 [00:27:42] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.26, 10.19, 7.90 [00:28:58] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 4.59, 8.84, 7.69 [00:29:03] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.50, 3.14, 2.73 [00:31:01] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.30, 2.73, 2.62 [00:31:30] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.97, 3.92, 3.02 [00:32:48] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 0.52, 1.73, 3.74 [00:33:26] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.25, 2.94, 2.77 [00:34:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.60, 1.35, 3.35 [00:57:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.91, 3.89, 3.07 [00:59:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.08, 3.22, 2.92 [01:10:24] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.61, 3.56, 3.18 [01:14:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.73, 2.82, 2.99 [01:25:49] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.40, 4.36, 3.52 [01:27:44] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.47, 3.36, 3.26 [01:32:38] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.28, 3.42, 2.14 [01:34:37] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.69, 2.41, 1.93 [01:55:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.26, 3.62, 3.08 [01:57:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.79, 3.31, 3.03 [02:03:38] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.57, 3.69, 3.26 [02:05:30] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.82, 3.99, 2.93 [02:07:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.18, 3.16, 2.75 [02:09:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.55, 3.72, 3.50 [02:13:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.56, 3.34, 3.39 [02:16:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 9.16, 4.43, 2.35 [02:17:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.06, 3.38, 3.43 [02:19:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.03, 2.46, 3.08 [02:25:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.63, 3.87, 3.43 [02:26:48] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 0.30, 3.66, 3.87 [02:27:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.80, 3.53, 3.37 [02:28:45] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.29, 3.58, 3.14 [02:30:07] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.46, 2.90, 1.83 [02:30:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.54, 2.13, 3.18 [02:31:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.49, 2.83, 3.15 [02:32:06] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.06, 3.44, 2.16 [02:34:05] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.08, 2.55, 1.99 [02:34:39] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.19, 3.98, 3.41 [02:35:00] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.21, 3.73, 3.10 [02:36:54] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.74, 3.75, 3.19 [02:38:49] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 1.82, 3.04, 2.99 [02:40:33] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 1.90, 3.34, 3.41 [02:42:31] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.75, 3.20, 3.35 [02:59:49] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.92, 3.72, 3.21 [02:59:53] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.59, 2.63, 1.71 [03:02:09] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 41.64, 16.19, 7.68 [03:03:00] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.20, 3.76, 2.70 [03:03:53] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.60, 3.92, 2.45 [03:04:56] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.87, 3.13, 2.60 [03:05:53] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.03, 2.75, 2.20 [03:27:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.98, 4.45, 3.28 [03:29:53] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.54, 3.00, 2.26 [03:33:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.35, 3.05, 3.90 [03:33:53] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.82, 2.53, 2.32 [03:37:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.70, 2.16, 3.33 [03:51:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.23, 3.74, 3.42 [03:52:18] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 3.53, 4.39, 3.28 [03:53:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.45, 4.05, 3.57 [03:54:15] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 1.92, 3.54, 3.10 [03:56:13] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.31, 3.12, 2.99 [03:57:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.47, 2.68, 3.98 [04:03:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.63, 3.29, 3.72 [04:07:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.23, 3.75, 3.83 [04:09:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.45, 3.47, 3.91 [04:13:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.80, 2.50, 3.27 [04:15:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.15, 2.33, 3.22 [04:16:49] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1120811701 secs [04:22:49] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09606820345 secs [04:25:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.73, 3.63, 3.28 [04:27:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.63, 2.83, 3.03 [04:29:55] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.92, 3.55, 2.21 [04:31:54] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.03, 4.06, 2.56 [04:45:53] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.63, 3.23, 3.85 [04:49:53] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.35, 1.65, 3.05 [05:04:49] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1088986695 secs [05:10:48] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.08415287733 secs [05:25:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.09, 3.13, 2.41 [05:27:15] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 3.31, 4.17, 2.73 [05:27:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.83, 2.75, 2.37 [05:29:15] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 0.99, 3.06, 2.50 [05:34:00] PROBLEM - microworldwiki.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'microworldwiki.ga' expires in 15 day(s) (Fri 09 Dec 2022 05:10:59 GMT +0000). [05:37:45] PROBLEM - wiki.jkrstudios.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.jkrstudios.org' expires in 15 day(s) (Fri 09 Dec 2022 05:09:18 GMT +0000). [05:40:30] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=raamasamgebhyahwiki --no-updates --username-prefix=mh:kranditri /home/reception/ImportDump/raamasamgebhyahwiki-20221122213229.xml (START) [05:40:31] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=raamasamgebhyahwiki --no-updates --username-prefix=mh:kranditri /home/reception/ImportDump/raamasamgebhyahwiki-20221122213229.xml (END - exit=256) [05:40:32] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=raamasamgebhyahwiki (START) [05:40:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:40:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:40:48] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=raamasamgebhyahwiki --no-updates --username-prefix=mh:kranditri /home/reception/ImportDump/raamasamgebhyahwiki-20221122213229.xml (START) [05:40:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:40:49] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=raamasamgebhyahwiki --no-updates --username-prefix=mh:kranditri /home/reception/ImportDump/raamasamgebhyahwiki-20221122213229.xml (END - exit=256) [05:40:50] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=raamasamgebhyahwiki (START) [05:40:55] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=raamasamgebhyahwiki (END - exit=2) [05:40:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:40:57] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=raamasamgebhyahwiki --active --update (END - exit=0) [05:41:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:41:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:41:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:41:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:41:25] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=raamasamgebhyahwiki (END - exit=0) [05:41:26] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=raamasamgebhyahwiki --active --update (END - exit=0) [05:41:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:41:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:43:43] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=raamasamgebhyahwiki --no-updates --username-prefix=mh:kranditri /home/reception/ImportDump/raamasamgebhyahwiki-20221122213229.xml (START) [05:43:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:44:48] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.50, 2.43, 1.45 [05:45:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.54, 2.88, 2.25 [05:46:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.65, 4.60, 2.41 [05:46:53] !log ran bash ./reception/wikibackups.sh /home/reception/publicwikis.txt /srv/mediawiki/w/maintenance/dumpBackup.php (wikibackups) yesterday (db141 not included); running separate instance now for larger wikis [05:47:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:47:15] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=raamasamgebhyahwiki --no-updates --username-prefix=mh:kranditri /home/reception/ImportDump/raamasamgebhyahwiki-20221122213229.xml (END - exit=0) [05:47:16] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=raamasamgebhyahwiki (START) [05:47:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:47:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:47:36] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.10, 2.23, 2.09 [05:48:35] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=raamasamgebhyahwiki (END - exit=0) [05:48:36] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=raamasamgebhyahwiki --active --update (END - exit=0) [05:48:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:48:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:52:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.19, 3.33, 2.64 [05:59:01] PROBLEM - wiki.wilderyogi.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.wilderyogi.eu' expires in 15 day(s) (Fri 09 Dec 2022 05:50:57 GMT +0000). [06:00:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.73, 3.71, 2.93 [06:02:38] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.72, 3.07, 2.79 [06:09:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.71, 5.09, 3.27 [06:11:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.65, 3.83, 3.02 [06:13:48] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.18, 2.92, 2.78 [06:21:51] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.33, 4.06, 2.77 [06:25:42] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.89, 3.00, 2.64 [06:28:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.01, 7.39, 3.96 [06:29:12] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.29, 11.83, 6.37 [06:31:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 13.04, 7.46, 4.23 [06:34:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.92, 3.17, 3.16 [06:37:07] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.30, 2.61, 3.92 [06:41:05] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.86, 1.65, 3.22 [06:45:15] PROBLEM - cp32 HTTPS on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:45:50] PROBLEM - cp32 PowerDNS Recursor on cp32 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:45:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 108.175.15.182/cpweb, 2607:f1c0:1800:8100::1/cpweb [06:45:58] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:36] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:47:52] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:47:54] PROBLEM - cp32 conntrack_table_size on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:48:01] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:49:13] RECOVERY - cp32 HTTPS on cp32 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3424 bytes in 0.417 second response time [06:49:32] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [06:49:47] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 39 minutes ago with 0 failures [06:49:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:49:53] RECOVERY - cp32 conntrack_table_size on cp32 is OK: OK: nf_conntrack is 2 % full [06:50:04] RECOVERY - cp32 PowerDNS Recursor on cp32 is OK: DNS OK: 0.123 seconds response time. miraheze.org returns 2607:f1c0:1800:26f::1,74.208.203.152 [06:57:41] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.07, 3.26, 2.46 [06:59:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.94, 2.85, 2.41 [07:35:56] RECOVERY - pubwiki.lab.co.pl - reverse DNS on sslhost is OK: SSL OK - pubwiki.lab.co.pl reverse DNS resolves to cp23.miraheze.org - CNAME OK [07:52:26] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.56, 3.50, 2.46 [07:54:25] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.51, 2.95, 2.37 [07:55:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.55, 3.44, 2.67 [07:57:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.81, 3.61, 2.83 [07:59:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.32, 4.25, 3.14 [08:07:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.62, 3.82, 3.43 [08:08:51] PROBLEM - cp32 HTTPS on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:08:52] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:09:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 108.175.15.182/cpweb, 2607:f1c0:1800:8100::1/cpweb [08:09:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 108.175.15.182/cpweb, 2607:f1c0:1800:8100::1/cpweb [08:09:34] PROBLEM - cp32 PowerDNS Recursor on cp32 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:10:49] RECOVERY - cp32 HTTPS on cp32 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3424 bytes in 3.880 second response time [08:10:58] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:11:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:11:17] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:11:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.14, 2.65, 3.07 [08:12:01] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:13:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:15:16] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 7.22, 4.08, 2.41 [08:15:21] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:17:04] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 34 minutes ago with 0 failures [08:17:04] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [08:17:20] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:18:06] RECOVERY - cp32 PowerDNS Recursor on cp32 is OK: DNS OK: 0.153 seconds response time. miraheze.org returns 108.175.15.182,2607:f1c0:1800:26f::1,2607:f1c0:1800:8100::1,74.208.203.152 [08:23:02] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.159075588 secs [08:24:55] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 0.65, 3.50, 3.52 [08:26:49] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.71, 2.63, 3.20 [08:37:02] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07391387224 secs [09:04:48] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.54, 2.73, 2.05 [09:08:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.59, 2.67, 2.27 [09:15:01] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.46, 2.60, 3.82 [09:16:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.31, 3.61, 2.71 [09:17:31] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 8191 bytes in 0.209 second response time [09:18:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.13, 2.63, 2.46 [09:19:01] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.86, 1.76, 3.20 [09:19:20] PROBLEM - mw141 nutcracker process on mw141 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 115 (nutcracker), command name 'nutcracker' [09:19:20] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 5.80, 10.68, 7.36 [09:20:01] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw141 [09:21:21] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 2.76, 7.83, 6.71 [09:22:01] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [09:30:34] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.34, 3.79, 2.38 [09:32:33] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.82, 3.39, 2.41 [09:37:20] RECOVERY - mw141 nutcracker process on mw141 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [09:37:25] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.210 second response time [09:56:18] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.10, 3.18, 2.22 [09:58:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.01, 2.73, 2.17 [10:10:50] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1377810538 secs [10:16:49] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08222511411 secs [10:19:11] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.87, 2.58, 1.88 [10:21:10] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.07, 1.93, 1.72 [10:24:49] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1822988391 secs [10:37:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.75, 4.61, 2.44 [10:41:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.54, 3.41, 2.47 [10:45:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.64, 2.48, 2.37 [10:50:49] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.07552006841 secs [10:54:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.91, 3.18, 2.28 [10:56:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.07, 2.81, 2.25 [10:58:53] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.63, 3.21, 2.07 [11:05:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.85, 4.46, 2.78 [11:07:50] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.58, 3.45, 2.61 [11:09:45] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.67, 2.87, 2.50 [11:10:48] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.99, 3.20, 3.18 [11:10:49] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1240415871 secs [11:18:49] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09227100015 secs [11:19:03] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.26, 3.44, 3.13 [11:21:03] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.50, 2.77, 2.92 [11:26:10] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.96, 3.30, 2.30 [11:28:05] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.08, 3.79, 2.59 [11:30:00] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.29, 3.87, 2.78 [11:30:37] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.03, 3.39, 3.10 [11:32:06] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.78, 3.70, 3.41 [11:33:51] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.91, 2.90, 2.63 [11:34:35] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.23, 3.70, 3.37 [11:34:48] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.67, 3.23, 2.46 [11:36:34] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.76, 2.65, 3.02 [11:36:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.98, 5.56, 3.46 [11:38:08] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.17, 4.10, 3.65 [11:38:48] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.30, 3.83, 3.07 [11:40:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.87, 2.89, 2.83 [11:46:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.97, 4.31, 3.33 [11:48:49] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1299433112 secs [11:52:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.56, 3.28, 3.33 [11:53:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.48, 2.82, 2.25 [11:57:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.61, 2.41, 2.23 [12:00:49] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.07100322843 secs [12:04:03] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.24, 3.31, 3.87 [12:08:02] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.64, 2.36, 3.34 [12:17:24] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.58, 3.27, 2.72 [12:19:24] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.29, 2.91, 2.66 [12:43:53] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.18, 3.85, 2.96 [12:44:48] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.55, 2.39, 1.65 [12:45:53] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.92, 3.26, 2.84 [12:48:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.27, 2.52, 1.96 [12:49:52] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.91, 3.62, 3.08 [12:51:32] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 13.05, 10.47, 6.70 [12:51:51] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.73, 3.22, 3.00 [12:53:32] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.53, 8.07, 6.27 [13:06:41] PROBLEM - privacywiki.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'privacywiki.xyz' expires in 15 day(s) (Fri 09 Dec 2022 12:54:53 GMT +0000). [13:06:47] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 8.61, 4.78, 3.57 [13:10:46] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.84, 3.99, 3.52 [13:14:45] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.35, 3.39, 3.39 [13:14:48] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.96, 2.49, 1.72 [13:16:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.35, 3.08, 2.03 [13:20:44] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.76, 3.78, 3.50 [13:20:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.10, 3.12, 2.42 [13:24:43] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.52, 3.54, 3.47 [13:28:35] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.68, 3.09, 1.96 [13:28:42] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.53, 3.30, 3.40 [13:30:34] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 3.11, 3.16, 2.13 [13:41:38] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.70, 3.42, 3.41 [13:43:38] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 3.03, 3.32, 3.37 [13:55:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.96, 4.07, 2.58 [13:56:35] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.78, 3.53, 3.37 [13:57:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.64, 4.00, 2.75 [13:58:34] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.26, 3.86, 3.50 [14:00:34] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.14, 3.41, 3.37 [14:01:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.42, 3.06, 2.68 [14:02:33] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.24, 2.91, 3.18 [14:14:30] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.38, 3.46, 3.23 [14:18:29] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.60, 3.66, 3.37 [14:19:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.59, 3.91, 3.14 [14:20:29] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.06, 3.87, 3.48 [14:21:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.95, 2.85, 2.84 [14:22:29] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.56, 3.30, 3.32 [14:25:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.06, 4.27, 3.39 [14:30:06] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.39, 3.57, 2.48 [14:31:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.88, 3.19, 3.27 [14:32:04] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.97, 3.00, 2.41 [14:39:24] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.64, 3.05, 3.00 [14:41:24] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.85, 3.10, 3.04 [14:41:50] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1710412502 secs [14:49:15] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.76, 3.45, 2.43 [14:52:20] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.57, 4.58, 3.78 [14:53:15] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.03, 3.06, 2.59 [14:55:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.14, 3.69, 2.98 [14:57:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.19, 3.89, 3.17 [14:59:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.39, 2.96, 2.91 [15:06:17] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.60, 3.70, 3.92 [15:08:17] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.02, 4.49, 4.22 [15:09:01] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.02, 3.08, 2.18 [15:09:50] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.06268197298 secs [15:11:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.45, 2.84, 2.21 [15:14:16] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.15, 3.44, 3.88 [15:16:15] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.06, 3.94, 4.00 [15:22:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.66, 2.96, 2.43 [15:24:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.81, 3.37, 2.65 [15:25:47] !log add 1g of swap to mon141 [15:25:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:26:07] PROBLEM - mem101 APT on mem101 is UNKNOWN: [15:27:48] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 1.86, 3.44, 3.85 [15:29:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4d95697c89a0...cc73124e61a5 [15:29:16] [url] Comparing 4d95697c89a0...cc73124e61a5 · miraheze/puppet · GitHub | github.com [15:29:17] [02miraheze/puppet] 07paladox 03cc73124 - graylog: fix firewall rule for port 12201 [15:29:43] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 6.72, 4.58, 4.20 [15:31:10] PROBLEM - cp32 APT on cp32 is UNKNOWN: [15:33:18] PROBLEM - cp32 APT on cp32 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (26 critical updates). [15:33:43] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.68, 3.85, 3.03 [15:35:37] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.65, 4.57, 3.41 [15:39:25] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.99, 3.37, 3.21 [15:42:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cc73124e61a5...65eba220497b [15:42:45] [url] Comparing cc73124e61a5...65eba220497b · miraheze/puppet · GitHub | github.com [15:42:46] [02miraheze/puppet] 07paladox 0365eba22 - grafana: set root as /usr/share/grafana/public [15:46:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/65eba220497b...23fd0d241f8c [15:46:35] [url] Comparing 65eba220497b...23fd0d241f8c · miraheze/puppet · GitHub | github.com [15:46:35] [02miraheze/puppet] 07paladox 0323fd0d2 - grafana: tweek nginx config [15:51:41] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.27, 3.54, 3.96 [15:51:58] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.65, 2.88, 2.35 [15:53:59] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.26, 3.29, 2.57 [15:56:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/0ca8c3d44856 [15:56:23] [02miraheze/puppet] 07paladox 030ca8c3d - varnish: add support websockets [15:56:24] [02puppet] 07paladox opened pull request 03#3014: varnish: add support websockets - 13https://github.com/miraheze/puppet/pull/3014 [15:56:24] [url] Page not found · GitHub · GitHub | github.com [15:56:27] [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [15:56:27] [url] Page not found · GitHub · GitHub | github.com [15:57:35] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.21, 3.53, 3.78 [15:59:14] [02puppet] 07paladox synchronize pull request 03#3014: varnish: add support websockets - 13https://github.com/miraheze/puppet/pull/3014 [15:59:15] [url] Page not found · GitHub · GitHub | github.com [15:59:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0ca8c3d44856...972c13465099 [15:59:18] [url] Comparing 0ca8c3d44856...972c13465099 · miraheze/puppet · GitHub | github.com [15:59:19] [02miraheze/puppet] 07paladox 03972c134 - Update default.vcl [15:59:33] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.90, 3.63, 3.82 [16:01:30] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 6.24, 4.09, 3.93 [16:01:58] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.70, 3.40, 3.19 [16:04:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.02, 5.94, 3.92 [16:09:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.40, 3.74, 3.95 [16:10:37] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.94, 3.48, 3.47 [16:12:37] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.54, 2.81, 3.22 [16:13:18] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.08, 3.47, 3.75 [16:15:21] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.79, 3.01, 3.53 [16:16:00] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.05, 3.30, 2.31 [16:17:23] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.78, 3.36, 3.58 [16:18:18] !log [paladox@test131] starting deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True} to test131 [16:18:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:19:23] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.98, 3.38, 3.55 [16:23:02] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1332 bytes in 0.014 second response time [16:26:58] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.499 second response time [16:27:13] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [16:27:30] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.76, 3.03, 3.20 [16:31:20] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.12, 3.37, 3.43 [16:31:55] !log [paladox@test131] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True} to test131 - SUCCESS in 816s [16:32:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:33:20] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.18, 3.39, 3.44 [16:35:19] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.98, 3.30, 3.39 [16:35:49] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.76, 3.51, 2.73 [16:37:47] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.18, 2.67, 2.52 [16:38:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/d34ba070bb37 [16:38:05] [02miraheze/puppet] 07paladox 03d34ba07 - varnish: flush vm more steadily [16:38:08] [02puppet] 07paladox opened pull request 03#3015: varnish: flush vm more steadily - 13https://github.com/miraheze/puppet/pull/3015 [16:38:09] [url] Page not found · GitHub · GitHub | github.com [16:38:10] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:38:11] [url] Page not found · GitHub · GitHub | github.com [16:38:12] [02puppet] 07paladox closed pull request 03#3015: varnish: flush vm more steadily - 13https://github.com/miraheze/puppet/pull/3015 [16:38:13] ... [16:38:13] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [16:38:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/23fd0d241f8c...5b9b771e59ca [16:38:16] [url] Comparing 23fd0d241f8c...5b9b771e59ca · miraheze/puppet · GitHub | github.com [16:38:18] [02miraheze/puppet] 07paladox 035b9b771 - varnish: flush vm more steadily (#3015) [16:38:20] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:38:21] ... [16:38:36] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.33, 3.80, 2.37 [16:40:35] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.62, 3.28, 2.35 [16:43:21] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 8.61, 4.95, 3.80 [16:43:38] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 3.41, 4.33, 3.35 [16:44:31] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.17, 3.72, 2.86 [16:45:35] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.38, 3.85, 3.30 [16:46:30] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.64, 3.05, 2.71 [16:47:32] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.01, 2.85, 3.00 [16:48:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.22, 4.00, 2.90 [16:50:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.36, 2.90, 2.63 [16:52:26] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 9.86, 5.96, 3.67 [16:53:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.34, 4.65, 3.13 [16:54:30] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.98, 3.89, 3.10 [16:55:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.39, 3.39, 2.84 [16:56:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b564d3838fa8 [16:56:09] [02miraheze/puppet] 07paladox 03b564d38 - varnish: don't set malloc [16:56:12] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:56:12] [url] Page not found · GitHub · GitHub | github.com [16:56:14] [02puppet] 07paladox opened pull request 03#3016: varnish: don't set malloc - 13https://github.com/miraheze/puppet/pull/3016 [16:56:14] [url] Page not found · GitHub · GitHub | github.com [16:56:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.23, 3.75, 3.26 [16:56:30] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.67, 3.02, 2.87 [16:56:53] [02puppet] 07paladox closed pull request 03#3016: varnish: don't set malloc - 13https://github.com/miraheze/puppet/pull/3016 [16:56:54] [url] Page not found · GitHub · GitHub | github.com [16:56:56] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [16:56:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5b9b771e59ca...134cdcdbad7c [16:56:59] [url] Comparing 5b9b771e59ca...134cdcdbad7c · miraheze/puppet · GitHub | github.com [16:57:01] [02miraheze/puppet] 07paladox 03134cdcd - varnish: don't set malloc (#3016) [16:57:02] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:57:04] [url] Page not found · GitHub · GitHub | github.com [16:58:22] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.00, 2.76, 2.95 [16:59:03] !log restart varnish on cp22 to pickup changes [17:01:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 3.71, 3.07 [17:02:18] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 2.41, 4.03, 3.56 [17:03:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.26, 2.87, 2.84 [17:04:17] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.47, 3.11, 3.27 [17:05:31] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.92, 3.85, 2.68 [17:05:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/1a56cc7213e8 [17:05:59] [02miraheze/puppet] 07paladox 031a56cc7 - varnish: add limits to Transient storage [17:06:02] [02puppet] 07paladox opened pull request 03#3017: varnish: add limits to Transient storage - 13https://github.com/miraheze/puppet/pull/3017 [17:06:02] [url] Page not found · GitHub · GitHub | github.com [17:06:05] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [17:06:05] ... [17:06:14] [02puppet] 07paladox closed pull request 03#3017: varnish: add limits to Transient storage - 13https://github.com/miraheze/puppet/pull/3017 [17:06:15] ... [17:06:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/134cdcdbad7c...790a8cd471e9 [17:06:17] [url] Comparing 134cdcdbad7c...790a8cd471e9 · miraheze/puppet · GitHub | github.com [17:06:18] [02miraheze/puppet] 07paladox 03790a8cd - varnish: add limits to Transient storage (#3017) [17:06:20] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [17:06:21] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [17:06:22] [url] Page not found · GitHub · GitHub | github.com [17:07:31] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.63, 3.89, 2.85 [17:08:43] !log restart varnish everywhere to pickup config changes [17:13:31] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.51, 4.50, 3.39 [17:14:09] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.75, 3.32, 3.08 [17:16:08] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.72, 3.37, 3.13 [17:16:48] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.30, 3.78, 3.25 [17:18:07] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.34, 3.20, 3.08 [17:18:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.82, 3.80, 3.07 [17:20:18] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.59, 3.67, 3.11 [17:22:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.62, 4.60, 3.62 [17:22:14] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.67, 4.16, 3.34 [17:22:50] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.66, 4.25, 3.27 [17:24:10] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.88, 3.70, 3.27 [17:24:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.94, 3.28 [17:26:06] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.70, 4.28, 3.52 [17:26:12] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.25, 3.91, 2.93 [17:26:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.32, 3.34, 3.13 [17:28:12] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.80, 3.87, 3.04 [17:29:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.76, 3.64, 2.91 [17:30:12] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.79, 4.38, 3.31 [17:31:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.38, 3.82, 3.06 [17:32:12] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.00, 3.93, 3.28 [17:32:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.26, 4.13, 3.60 [17:32:49] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:34:12] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.76, 4.30, 3.50 [17:34:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.71, 3.65, 3.49 [17:34:55] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.383 second response time [17:35:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.69, 10.75, 8.65 [17:36:36] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.87, 3.20, 3.33 [17:37:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.73, 10.05, 8.66 [17:41:50] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.66, 3.67, 3.98 [17:44:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.24, 9.94, 8.58 [17:45:31] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.26, 3.57, 3.96 [17:45:48] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.22, 4.38, 4.19 [17:46:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.20, 10.11, 8.85 [17:47:31] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.38, 4.30, 4.18 [17:47:47] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.29, 3.61, 3.93 [17:55:47] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.14, 3.12, 3.50 [17:57:31] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.63, 3.38, 3.86 [17:57:47] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.35, 2.43, 3.20 [18:01:52] !log reception@mwtask141:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/dumpBackup.php --full --include-files --wiki libertygamewiki > /home/reception/libertygamewiki22112022.xml [18:02:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:03:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.80, 3.44, 4.00 [18:05:31] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.76, 2.67, 3.30 [18:11:03] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.39, 2.88, 3.88 [18:13:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 5.30, 3.98, 3.84 [18:15:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.08, 3.75, 3.78 [18:17:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.47, 3.83, 3.79 [18:19:01] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.90, 3.50, 3.71 [18:19:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.70, 3.42, 3.65 [18:21:01] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.41, 3.63, 3.75 [18:21:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 7.10, 4.68, 4.06 [18:22:21] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.91, 3.60, 3.36 [18:23:00] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.44, 3.76, 3.77 [18:24:15] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 3.03, 3.26, 3.25 [18:28:05] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 3.28, 4.05, 3.68 [18:28:28] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.31, 3.70, 3.08 [18:29:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.72, 3.65, 3.99 [18:30:00] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.33, 3.57, 3.55 [18:31:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 3.91, 3.83, 4.02 [18:33:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.04, 3.52, 3.88 [18:33:49] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.88, 2.94, 3.32 [18:37:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.66, 4.38, 4.15 [18:37:39] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.59, 3.51, 3.48 [18:39:33] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.83, 3.19, 3.36 [18:42:18] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.71, 3.18, 3.54 [18:43:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.89, 3.68, 3.94 [18:43:31] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.66, 3.35, 3.41 [18:44:17] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 16.71, 7.30, 4.93 [18:45:31] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 0.91, 2.41, 3.06 [18:45:50] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:45:56] PROBLEM - gluster101 Disk Space on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:45:59] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [18:46:12] PROBLEM - gluster102 Disk Space on gluster102 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:46:15] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 217.174.247.33/cpweb, 109.228.51.216/cpweb, 2a00:da00:1800:326::1/cpweb, 2a00:da00:1800:328::1/cpweb, 108.175.15.182/cpweb, 74.208.203.152/cpweb, 2607:f1c0:1800:8100::1/cpweb, 2607:f1c0:1800:26f::1/cpweb [18:46:42] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:47:00] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 0.72, 2.63, 3.76 [18:47:07] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 1.65, 2.39, 3.33 [18:47:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [18:47:21] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:47:42] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [18:47:45] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.010 second response time [18:47:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.28, 11.00, 6.56 [18:47:51] PROBLEM - gluster101 Disk Space on gluster101 is WARNING: DISK WARNING - free space: / 103577 MB (10% inode=82%); [18:47:59] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 9% [18:48:11] PROBLEM - gluster102 Disk Space on gluster102 is WARNING: DISK WARNING - free space: / 55327 MB (10% inode=70%); [18:48:12] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.18, 2.85, 3.89 [18:48:15] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:48:38] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.363 second response time [18:49:07] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 36% [18:49:22] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.373 second response time [18:49:27] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 5 backends are down. mw121 mw122 mw132 mw141 mw142 [18:49:29] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [18:49:42] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 56% [18:50:00] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 15.15, 9.22, 6.82 [18:50:07] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [18:50:08] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [18:50:59] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.49, 1.97, 3.23 [18:51:27] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [18:51:28] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [18:51:42] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 4% [18:52:00] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 6.33, 7.87, 6.62 [18:52:07] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [18:52:07] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [18:52:12] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.13, 2.21, 3.37 [18:52:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.54, 2.74, 3.85 [18:53:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.92, 2.66, 3.94 [18:55:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.97, 4.13, 4.34 [18:56:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.35, 3.34, 3.81 [18:57:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 0.68, 2.37, 3.92 [18:58:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.11, 3.35, 3.76 [19:03:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.71, 2.09, 3.32 [19:03:55] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.55, 3.90, 3.53 [19:05:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.08, 3.45, 3.92 [19:05:55] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.10, 3.42, 3.39 [19:07:54] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.03, 3.00, 3.24 [19:07:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.39, 3.65, 3.92 [19:09:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.44, 3.61, 3.88 [19:10:08] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.52, 3.10, 4.00 [19:10:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.14, 3.72, 3.67 [19:12:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.83, 3.49, 3.60 [19:13:53] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.76, 3.60, 3.44 [19:14:05] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.68, 4.13, 4.10 [19:15:55] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.26, 3.69, 3.79 [19:16:04] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.78, 3.53, 3.89 [19:17:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.37, 3.54, 3.73 [19:19:51] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 3.11, 3.28, 3.34 [19:19:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.45, 3.83, 3.81 [19:21:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.57, 3.80, 3.81 [19:23:58] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.63, 4.20, 3.99 [19:24:44] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 1.83, 2.78, 3.28 [19:25:50] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.35, 3.57, 3.39 [19:25:57] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.75, 3.72, 3.85 [19:29:49] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.03, 3.13, 3.27 [19:29:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.71, 3.80, 3.68 [19:29:55] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.63, 3.89, 3.81 [19:31:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.41, 3.73, 3.67 [19:33:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.24, 3.99, 3.77 [19:34:29] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.25, 4.02, 3.49 [19:36:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.64, 3.93, 3.56 [19:37:50] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.87, 3.24, 3.58 [19:38:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.01, 3.66, 3.51 [19:39:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.89, 3.26, 2.88 [19:40:12] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.54, 3.43, 3.43 [19:41:07] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 1.70, 2.68, 2.71 [19:41:47] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.23, 2.84, 3.36 [19:42:12] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.57, 3.10, 3.30 [19:42:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.22, 4.00, 3.70 [19:44:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.24, 3.53, 3.55 [19:45:47] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.23, 3.66, 3.51 [19:45:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.23, 3.34, 3.79 [19:48:44] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.52, 3.06, 3.36 [19:49:47] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.68, 3.91, 3.66 [19:49:55] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.07, 3.45, 3.68 [19:51:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.22, 3.25, 3.57 [19:52:12] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.89, 2.64, 2.00 [19:53:47] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.78, 2.94, 3.33 [19:53:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.37, 3.83, 3.75 [19:54:12] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.88, 2.44, 2.01 [19:55:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.38, 3.22, 3.53 [19:57:54] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.18, 2.92, 3.39 [20:01:55] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.16, 3.91, 3.70 [20:07:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.55, 3.56, 3.66 [20:16:39] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.66, 3.82, 3.37 [20:19:54] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.23, 3.03, 3.38 [20:22:35] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.84, 3.30, 3.25 [20:23:50] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.25, 8.86, 6.95 [20:25:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.86, 8.05, 6.90 [20:31:28] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.79, 4.28, 3.58 [20:33:26] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.79, 3.34, 3.32 [20:37:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.79, 3.20, 3.04 [20:38:57] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.74, 3.67, 2.98 [20:39:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.74, 4.01, 3.34 [20:41:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.10, 3.68, 3.31 [20:42:12] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.59, 3.20, 2.66 [20:42:49] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.75, 3.43, 3.05 [20:43:54] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.76, 3.35, 3.23 [20:44:12] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 3.07, 3.22, 2.73 [20:44:45] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.30, 3.58, 3.14 [20:47:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.78, 3.70, 3.39 [20:48:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.49, 3.63, 3.32 [20:49:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.65, 4.25, 3.61 [20:50:08] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03create-iaupload [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/87e7cd1db3fa [20:50:11] [02miraheze/puppet] 07Universal-Omega 0387e7cd1 - Create python script for uploading to archive.org [20:50:17] [02puppet] 07Universal-Omega created branch 03create-iaupload - 13https://github.com/miraheze/puppet [20:50:18] [url] Page not found · GitHub · GitHub | github.com [20:50:18] [02puppet] 07Universal-Omega opened pull request 03#3018: Create python script for uploading to archive.org - 13https://github.com/miraheze/puppet/pull/3018 [20:50:18] [url] Page not found · GitHub · GitHub | github.com [20:51:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.48, 2.97, 2.50 [20:51:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.15, 3.93, 3.59 [20:52:44] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.42, 3.31, 3.28 [20:53:07] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.80, 2.84, 2.51 [20:53:22] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03create-iaupload [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/87e7cd1db3fa...fc009cd90a7b [20:53:22] [url] Comparing 87e7cd1db3fa...fc009cd90a7b · miraheze/puppet · GitHub | github.com [20:53:23] [02miraheze/puppet] 07Universal-Omega 03fc009cd - Update init.pp [20:53:24] [02puppet] 07Universal-Omega synchronize pull request 03#3018: Create python script for uploading to archive.org - 13https://github.com/miraheze/puppet/pull/3018 [20:53:24] [url] Page not found · GitHub · GitHub | github.com [20:53:37] [02puppet] 07Universal-Omega edited pull request 03#3018: T10000: create python script for uploading to archive.org - 13https://github.com/miraheze/puppet/pull/3018 [20:53:37] [url] Page not found · GitHub · GitHub | github.com [20:53:54] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.04, 3.26, 3.38 [20:55:06] [02puppet] 07Universal-Omega synchronize pull request 03#3018: T10000: create python script for uploading to archive.org - 13https://github.com/miraheze/puppet/pull/3018 [20:55:07] ... [20:55:08] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03create-iaupload [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fc009cd90a7b...380c9f4a0997 [20:55:09] [url] Comparing fc009cd90a7b...380c9f4a0997 · miraheze/puppet · GitHub | github.com [20:55:10] [02miraheze/puppet] 07Universal-Omega 03380c9f4 - Update iaupload.py [20:56:39] [02puppet] 07Universal-Omega synchronize pull request 03#3018: T10000: create python script for uploading to archive.org - 13https://github.com/miraheze/puppet/pull/3018 [20:56:40] ... [20:56:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03create-iaupload [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/380c9f4a0997...2699f682beb4 [20:56:42] [url] Comparing 380c9f4a0997...2699f682beb4 · miraheze/puppet · GitHub | github.com [20:56:42] [02miraheze/puppet] 07Universal-Omega 032699f68 - Update iaupload.py [20:59:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.64, 3.69, 3.54 [21:03:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.13, 3.92, 3.62 [21:05:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.22, 3.87, 3.65 [21:09:29] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03create-iaupload [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2699f682beb4...8cbbb59cb8a0 [21:09:29] [url] Comparing 2699f682beb4...8cbbb59cb8a0 · miraheze/puppet · GitHub | github.com [21:09:31] [02miraheze/puppet] 07Universal-Omega 038cbbb59 - Update iaupload.py [21:09:33] [02puppet] 07Universal-Omega synchronize pull request 03#3018: T10000: create python script for uploading to archive.org - 13https://github.com/miraheze/puppet/pull/3018 [21:09:33] [url] Page not found · GitHub · GitHub | github.com [21:09:54] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.10, 2.81, 3.25 [21:17:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.50, 3.85, 3.26 [21:19:17] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.71, 3.35, 3.15 [21:28:59] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/790a8cd471e9...b2b862328f3f [21:28:59] [url] Comparing 790a8cd471e9...b2b862328f3f · miraheze/puppet · GitHub | github.com [21:29:00] [02miraheze/puppet] 07Universal-Omega 03b2b8623 - Fix typo [21:29:22] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b2b862328f3f...eb8cefd68425 [21:29:23] [url] Comparing b2b862328f3f...eb8cefd68425 · miraheze/puppet · GitHub | github.com [21:29:24] [02miraheze/puppet] 07Universal-Omega 03eb8cefd - Fix typo [21:37:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.51, 3.26, 3.02 [21:38:27] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03fix-puppet-lint [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/37b5747d23f9 [21:38:30] [02miraheze/puppet] 07Universal-Omega 0337b5747 - Fix puppet lint issues [21:38:31] [02puppet] 07Universal-Omega opened pull request 03#3019: Fix puppet lint issues - 13https://github.com/miraheze/puppet/pull/3019 [21:38:31] [url] Page not found · GitHub · GitHub | github.com [21:38:33] [02puppet] 07Universal-Omega created branch 03fix-puppet-lint - 13https://github.com/miraheze/puppet [21:38:34] [url] Page not found · GitHub · GitHub | github.com [21:39:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03fix-puppet-lint [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/37b5747d23f9...07c5e3fc205b [21:39:42] [url] Comparing 37b5747d23f9...07c5e3fc205b · miraheze/puppet · GitHub | github.com [21:39:42] [02miraheze/puppet] 07Universal-Omega 0307c5e3f - Update init.pp [21:39:44] [02puppet] 07Universal-Omega synchronize pull request 03#3019: Fix puppet lint issues - 13https://github.com/miraheze/puppet/pull/3019 [21:39:45] [url] Page not found · GitHub · GitHub | github.com [21:39:54] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.43, 3.00, 2.96 [21:40:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03fix-puppet-lint [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/07c5e3fc205b...4ae1e438d074 [21:40:42] [url] Comparing 07c5e3fc205b...4ae1e438d074 · miraheze/puppet · GitHub | github.com [21:40:43] [02miraheze/puppet] 07Universal-Omega 034ae1e43 - Update swift.pp [21:40:44] [02puppet] 07Universal-Omega synchronize pull request 03#3019: Fix puppet lint issues - 13https://github.com/miraheze/puppet/pull/3019 [21:40:45] [url] Page not found · GitHub · GitHub | github.com [21:42:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.24, 3.55, 3.10 [21:43:07] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03fix-puppet-lint [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4ae1e438d074...4647cdb5e40b [21:43:08] [url] Comparing 4ae1e438d074...4647cdb5e40b · miraheze/puppet · GitHub | github.com [21:43:10] [02miraheze/puppet] 07Universal-Omega 034647cdb - Update ac.pp [21:43:12] [02puppet] 07Universal-Omega synchronize pull request 03#3019: Fix puppet lint issues - 13https://github.com/miraheze/puppet/pull/3019 [21:43:13] [url] Page not found · GitHub · GitHub | github.com [21:43:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.19, 3.86, 3.33 [21:44:03] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03fix-puppet-lint [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4647cdb5e40b...7b253ee27da7 [21:44:04] [url] Comparing 4647cdb5e40b...7b253ee27da7 · miraheze/puppet · GitHub | github.com [21:44:05] [02miraheze/puppet] 07Universal-Omega 037b253ee - Update init.pp [21:44:08] [02puppet] 07Universal-Omega synchronize pull request 03#3019: Fix puppet lint issues - 13https://github.com/miraheze/puppet/pull/3019 [21:44:09] [url] Page not found · GitHub · GitHub | github.com [21:44:44] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.91, 3.25, 3.04 [21:44:53] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03fix-puppet-lint [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7b253ee27da7...16473e54b4cc [21:44:54] [url] Comparing 7b253ee27da7...16473e54b4cc · miraheze/puppet · GitHub | github.com [21:44:56] [02miraheze/puppet] 07Universal-Omega 0316473e5 - Update proxy.pp [21:44:58] [02puppet] 07Universal-Omega synchronize pull request 03#3019: Fix puppet lint issues - 13https://github.com/miraheze/puppet/pull/3019 [21:44:58] [url] Page not found · GitHub · GitHub | github.com [21:45:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.82, 3.86, 3.40 [21:49:16] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03fix-puppet-lint [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/16473e54b4cc...e743020f67e8 [21:49:17] [url] Comparing 16473e54b4cc...e743020f67e8 · miraheze/puppet · GitHub | github.com [21:49:19] [02miraheze/puppet] 07Universal-Omega 03e743020 - Update init.pp [21:49:21] [02puppet] 07Universal-Omega synchronize pull request 03#3019: Fix puppet lint issues - 13https://github.com/miraheze/puppet/pull/3019 [21:49:22] [url] Page not found · GitHub · GitHub | github.com [21:49:54] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 1.90, 2.88, 3.11 [21:51:21] [02puppet] 07Universal-Omega closed pull request 03#3019: Fix puppet lint issues - 13https://github.com/miraheze/puppet/pull/3019 [21:51:22] [url] Page not found · GitHub · GitHub | github.com [21:51:24] [02miraheze/puppet] 07Universal-Omega deleted branch 03fix-puppet-lint [21:51:26] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/eb8cefd68425...fd878ade1d0c [21:51:27] [url] Comparing eb8cefd68425...fd878ade1d0c · miraheze/puppet · GitHub | github.com [21:51:27] [02miraheze/puppet] 07Universal-Omega 03fd878ad - Fix puppet lint issues [21:51:30] [02puppet] 07Universal-Omega deleted branch 03fix-puppet-lint - 13https://github.com/miraheze/puppet [21:51:31] [url] Page not found · GitHub · GitHub | github.com [21:52:54] [02puppet] 07Universal-Omega closed pull request 03#2999: mediawiki-landing: redirect `/en` to `/` - 13https://github.com/miraheze/puppet/pull/2999 [21:52:55] ... [21:52:55] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fd878ade1d0c...a9ff61f38ae5 [21:52:56] [url] Comparing fd878ade1d0c...a9ff61f38ae5 · miraheze/puppet · GitHub | github.com [21:52:57] [02miraheze/puppet] 07Universal-Omega 03a9ff61f - mediawiki-landing: redirect `/en` to `/` (#2999) [21:52:59] [02miraheze/puppet] 07Universal-Omega deleted branch 03mediawiki-landing-redirect-en [21:53:01] [02puppet] 07Universal-Omega deleted branch 03mediawiki-landing-redirect-en - 13https://github.com/miraheze/puppet [21:53:02] [url] Page not found · GitHub · GitHub | github.com [21:53:29] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.67, 3.52, 3.14 [21:55:28] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.82, 3.06, 3.02 [21:56:14] !log create table chat_users on islamipediawiki [21:56:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:59:24] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.99, 3.50, 3.18 [22:01:22] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 3.82, 4.20, 3.53 [22:03:21] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.02, 3.35, 3.30 [22:04:42] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:07:17] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.57, 4.33, 3.66 [22:08:42] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:09:45] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:10:12] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:11:15] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.62, 3.25, 3.41 [22:12:45] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:13:13] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.57, 3.01, 3.30 [22:14:12] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:15:10] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:17:10] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.88, 3.65, 3.61 [22:19:42] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a9ff61f38ae5...dc5f862064f9 [22:19:43] [url] Comparing a9ff61f38ae5...dc5f862064f9 · miraheze/puppet · GitHub | github.com [22:19:45] [02miraheze/puppet] 07Universal-Omega 03dc5f862 - Fix syntax [22:21:07] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.86, 2.68, 3.22 [22:21:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/c49e5da8efb2 [22:21:30] [02miraheze/puppet] 07paladox 03c49e5da - varnish: stop mounting /var/lib/varnish [22:21:32] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [22:21:33] [url] Page not found · GitHub · GitHub | github.com [22:21:35] [02puppet] 07paladox opened pull request 03#3020: varnish: stop mounting /var/lib/varnish - 13https://github.com/miraheze/puppet/pull/3020 [22:21:36] [url] Page not found · GitHub · GitHub | github.com [22:21:47] [02puppet] 07paladox closed pull request 03#3020: varnish: stop mounting /var/lib/varnish - 13https://github.com/miraheze/puppet/pull/3020 [22:21:47] [url] Page not found · GitHub · GitHub | github.com [22:21:50] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [22:21:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dc5f862064f9...8e386baa193d [22:21:51] [url] Comparing dc5f862064f9...8e386baa193d · miraheze/puppet · GitHub | github.com [22:21:53] [02miraheze/puppet] 07paladox 038e386ba - varnish: stop mounting /var/lib/varnish (#3020) [22:21:55] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [22:21:55] [url] Page not found · GitHub · GitHub | github.com [22:22:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8e386baa193d...97deab786568 [22:22:54] [url] Comparing 8e386baa193d...97deab786568 · miraheze/puppet · GitHub | github.com [22:22:55] [02miraheze/puppet] 07paladox 0397deab7 - Update init.pp [22:25:16] PROBLEM - cp33 Puppet on cp33 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:26:45] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/73ba1f64a854...af7f96adf8a0 [22:26:46] [url] Comparing 73ba1f64a854...af7f96adf8a0 · miraheze/dns · GitHub | github.com [22:26:47] [02miraheze/dns] 07paladox 03af7f96a - depool cp22/23 [22:29:01] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.77, 3.39, 3.22 [22:29:16] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:30:15] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2a00:da00:1800:328::1/cpweb, 2607:f1c0:1800:8100::1/cpweb [22:31:01] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.64, 3.20, 3.18 [22:31:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 217.174.247.33/cpweb, 2a00:da00:1800:326::1/cpweb, 2607:f1c0:1800:8100::1/cpweb [22:32:38] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:32:52] PROBLEM - ping6 on cp32 is CRITICAL: PING CRITICAL - Packet loss = 100% [22:32:54] PROBLEM - cp32 HTTPS on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:32:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:33:00] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:33:00] PROBLEM - cp32 Stunnel for puppet141 on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:33:00] PROBLEM - cp32 Stunnel for mwtask141 on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:33:01] PROBLEM - cp32 Stunnel for test131 on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:33:04] PROBLEM - cp32 Stunnel for reports121 on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:34:03] PROBLEM - Host cp32 is DOWN: PING CRITICAL - Packet loss = 100% [22:36:01] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/af7f96adf8a0...78941f0946bd [22:36:02] [url] Comparing af7f96adf8a0...78941f0946bd · miraheze/dns · GitHub | github.com [22:36:03] [02miraheze/dns] 07paladox 0378941f0 - Revert "depool cp22/23" [22:36:39] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [22:38:03] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:38:08] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [22:40:53] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [22:42:11] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [22:43:31] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:44:43] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.89, 3.41, 2.99 [22:45:11] [02puppet] 07Universal-Omega synchronize pull request 03#3018: T10000: create python script for uploading to archive.org - 13https://github.com/miraheze/puppet/pull/3018 [22:45:11] [url] Page not found · GitHub · GitHub | github.com [22:45:14] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03create-iaupload [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8cbbb59cb8a0...d970c9d0dae7 [22:45:14] [url] Comparing 8cbbb59cb8a0...d970c9d0dae7 · miraheze/puppet · GitHub | github.com [22:45:16] [02miraheze/puppet] 07Universal-Omega 03d970c9d - Update iaupload.py [22:46:37] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 1.82, 2.77, 2.80 [22:53:49] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/78941f0946bd...c58d858f2623 [22:53:50] [url] Comparing 78941f0946bd...c58d858f2623 · miraheze/dns · GitHub | github.com [22:53:51] [02miraheze/dns] 07paladox 03c58d858 - Depool cp32/33 [22:57:52] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.102452606 secs [22:58:41] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.52, 3.53, 3.05 [22:59:19] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.32, 4.01, 3.12 [22:59:38] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.57, 3.61, 3.20 [23:01:34] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.44, 4.26, 3.48 [23:01:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.07, 4.05, 3.26 [23:02:38] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.00, 3.56, 3.27 [23:03:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.11, 3.78, 3.23 [23:03:30] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.39, 3.96, 3.46 [23:03:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.24, 3.74, 3.25 [23:04:37] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.78, 3.26, 3.19 [23:07:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:07:20] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1339473426 secs [23:07:37] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.0885514617 secs [23:08:15] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:08:18] RECOVERY - Host cp32 is UP: PING OK - Packet loss = 0%, RTA = 102.85 ms [23:08:43] RECOVERY - ping6 on cp32 is OK: PING OK - Packet loss = 0%, RTA = 102.53 ms [23:08:45] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [23:08:51] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.65, 3.34, 3.25 [23:09:04] RECOVERY - cp32 Stunnel for test131 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8180 [23:09:04] RECOVERY - cp32 Stunnel for mwtask141 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8150 [23:09:09] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [23:09:19] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:09:29] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.07, 0.13, 0.08 [23:09:39] RECOVERY - cp32 Stunnel for reports121 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8205 [23:09:49] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:09:54] RECOVERY - cp32 Stunnel for puppet141 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8204 [23:09:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.48, 3.85, 3.43 [23:10:59] RECOVERY - cp32 HTTPS on cp32 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.744 second response time [23:11:14] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.80, 3.92, 3.56 [23:11:15] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.05809468031 secs [23:11:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.79, 3.51, 3.36 [23:12:36] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [23:13:07] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 8% [23:13:10] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.47, 3.28, 3.36 [23:13:31] PROBLEM - cp33 HTTPS on cp33 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 8191 bytes in 7.646 second response time [23:14:16] PROBLEM - cp33 Puppet on cp33 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:14:43] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/c58d858f2623...79116d8b8390 [23:14:44] [url] Comparing c58d858f2623...79116d8b8390 · miraheze/dns · GitHub | github.com [23:14:45] [02miraheze/dns] 07paladox 0379116d8 - Revert "Depool cp32/33" [23:15:36] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [23:15:36] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 94% [23:15:54] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.60, 3.22, 3.29 [23:17:27] RECOVERY - cp33 HTTPS on cp33 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 0.546 second response time [23:17:31] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 6% [23:17:35] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [23:22:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/0e673e874ded [23:22:43] [02miraheze/puppet] 07paladox 030e673e8 - varnish: experiment with max threads being 1000 [23:22:45] [02puppet] 07paladox opened pull request 03#3021: varnish: experiment with max threads being 1000 - 13https://github.com/miraheze/puppet/pull/3021 [23:22:46] [url] Page not found · GitHub · GitHub | github.com [23:22:47] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [23:22:48] [url] Page not found · GitHub · GitHub | github.com [23:23:41] [02puppet] 07paladox closed pull request 03#3021: varnish: experiment with max threads being 1000 - 13https://github.com/miraheze/puppet/pull/3021 [23:23:42] [url] Page not found · GitHub · GitHub | github.com [23:23:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/97deab786568...821eb6b5e683 [23:23:44] [url] Comparing 97deab786568...821eb6b5e683 · miraheze/puppet · GitHub | github.com [23:23:44] [02miraheze/puppet] 07paladox 03821eb6b - varnish: experiment with max threads being 1000 (#3021) [23:23:47] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [23:23:50] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [23:23:51] [url] Page not found · GitHub · GitHub | github.com [23:23:53] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.85, 3.91, 3.63 [23:23:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.04, 3.42, 3.35 [23:24:12] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 3.69, 4.17, 3.60 [23:25:49] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.06, 3.92, 3.66 [23:25:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.00, 4.13, 3.62 [23:26:01] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [23:26:12] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.32, 3.98, 3.61 [23:26:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e1f80444b966 [23:26:59] [02miraheze/puppet] 07paladox 03e1f8044 - varnish: up the max threads to 5000 - still experimenting [23:27:00] [02puppet] 07paladox opened pull request 03#3022: varnish: up the max threads to 5000 - still experimenting - 13https://github.com/miraheze/puppet/pull/3022 [23:27:01] [url] Page not found · GitHub · GitHub | github.com [23:27:01] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [23:27:02] ... [23:27:04] [02puppet] 07paladox closed pull request 03#3022: varnish: up the max threads to 5000 - still experimenting - 13https://github.com/miraheze/puppet/pull/3022 [23:27:05] ... [23:27:05] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [23:27:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/821eb6b5e683...5f90346b63b8 [23:27:08] [url] Comparing 821eb6b5e683...5f90346b63b8 · miraheze/puppet · GitHub | github.com [23:27:08] [02miraheze/puppet] 07paladox 035f90346 - varnish: up the max threads to 5000 - still experimenting (#3022) [23:27:10] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [23:27:11] [url] Page not found · GitHub · GitHub | github.com [23:27:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.43, 3.95, 3.62 [23:29:41] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.72, 3.70, 3.62 [23:32:12] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 3.04, 3.20, 3.39 [23:32:17] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.62, 3.42, 3.11 [23:33:33] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.45, 2.94, 3.31 [23:33:54] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.60, 3.04, 3.34 [23:34:15] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.80, 2.79, 2.92 [23:36:12] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.98, 3.41, 3.45 [23:38:13] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.91, 3.86, 3.60 [23:39:03] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:23] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.77, 3.70, 3.53 [23:39:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.04, 3.44, 3.42 [23:40:12] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.39, 3.25, 3.40 [23:41:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.79, 3.93, 3.60 [23:43:15] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.57, 3.18, 3.35 [23:45:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.17, 3.95, 3.72 [23:47:09] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.00, 3.45, 3.44 [23:51:00] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.04, 2.97, 3.26 [23:55:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.46, 3.97, 3.77 [23:56:52] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.12, 3.51, 3.44 [23:57:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.56, 3.81, 3.73 [23:58:47] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.16, 3.38, 3.40