[00:07:16] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 3.86, 4.68, 3.32 [00:14:43] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:16:41] RECOVERY - cp22 APT on cp22 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:17:53] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:21:11] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.52, 3.46, 3.88 [00:21:56] RECOVERY - cp23 APT on cp23 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:25:10] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.92, 2.48, 3.37 [00:51:47] PROBLEM - ping6 on cp32 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 105.94 ms [00:53:47] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 105.90 ms [01:05:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-ceph [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f1f25e427f2b...a5ae85863db1 [01:05:58] [url] Comparing f1f25e427f2b...a5ae85863db1 · miraheze/puppet · GitHub | github.com [01:05:59] [02miraheze/puppet] 07paladox 03a5ae858 - Update ceph.conf.epp [01:06:00] [02puppet] 07paladox synchronize pull request 03#2875: Add ceph module - 13https://github.com/miraheze/puppet/pull/2875 [01:06:01] [url] Page not found · GitHub · GitHub | github.com [01:06:48] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [01:14:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-ceph [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a5ae85863db1...69e0d8147e8d [01:14:36] [url] Comparing a5ae85863db1...69e0d8147e8d · miraheze/puppet · GitHub | github.com [01:14:37] [02miraheze/puppet] 07paladox 0369e0d81 - Update common.yaml [01:14:39] [02puppet] 07paladox synchronize pull request 03#2875: Add ceph module - 13https://github.com/miraheze/puppet/pull/2875 [01:14:39] [url] Page not found · GitHub · GitHub | github.com [01:23:19] PROBLEM - swiftobject111 Puppet on swiftobject111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:23:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-ceph [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/69e0d8147e8d...5ebfd2333123 [01:23:52] [url] Comparing 69e0d8147e8d...5ebfd2333123 · miraheze/puppet · GitHub | github.com [01:23:53] [02miraheze/puppet] 07paladox 035ebfd23 - Update osd.pp [01:23:55] [02puppet] 07paladox synchronize pull request 03#2875: Add ceph module - 13https://github.com/miraheze/puppet/pull/2875 [01:23:55] [url] Page not found · GitHub · GitHub | github.com [01:24:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-ceph [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5ebfd2333123...10a1d70de92e [01:24:40] [url] Comparing 5ebfd2333123...10a1d70de92e · miraheze/puppet · GitHub | github.com [01:24:41] [02miraheze/puppet] 07paladox 0310a1d70 - Update swiftobject111.yaml [01:24:42] [02puppet] 07paladox synchronize pull request 03#2875: Add ceph module - 13https://github.com/miraheze/puppet/pull/2875 [01:24:43] [url] Page not found · GitHub · GitHub | github.com [01:27:19] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [01:34:43] RECOVERY - swiftobject114 Puppet on swiftobject114 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:34:57] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:35:12] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:35:39] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:36:34] RECOVERY - swiftobject115 Puppet on swiftobject115 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:44:57] PROBLEM - swiftobject113 Puppet on swiftobject113 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:47:39] PROBLEM - swiftobject112 Puppet on swiftobject112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:50:43] PROBLEM - swiftobject114 Puppet on swiftobject114 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:59:29] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 6.09, 3.47, 1.60 [02:02:34] PROBLEM - swiftobject115 Puppet on swiftobject115 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:11:29] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 0.98, 0.54, 0.20 [02:19:52] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [02:23:58] RECOVERY - cp23 APT on cp23 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [02:28:21] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [02:28:46] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.40, 5.13, 3.46 [02:30:16] RECOVERY - cp23 APT on cp23 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [02:30:43] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.00, 3.88, 3.21 [02:32:39] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.16, 2.96, 2.94 [02:55:17] RECOVERY - ping6 on cp32 is OK: PING OK - Packet loss = 0%, RTA = 105.62 ms [02:55:29] RECOVERY - ping6 on cp33 is OK: PING OK - Packet loss = 0%, RTA = 107.58 ms [02:56:28] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [02:56:55] PROBLEM - cp33 APT on cp33 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [02:57:08] PROBLEM - cp33 HTTPS on cp33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:57:10] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [02:59:32] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.13, 3.61, 1.58 [02:59:54] RECOVERY - swiftobject115 Puppet on swiftobject115 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:01:03] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [03:01:08] RECOVERY - cp33 HTTPS on cp33 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3361 bytes in 0.424 second response time [03:01:32] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.63, 3.60, 1.86 [03:02:22] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 89% [03:03:32] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.06, 2.61, 1.70 [03:05:05] !log reboot cp33 [03:05:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:07:23] PROBLEM - cp33 HTTPS on cp33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:07:47] PROBLEM - cp32 APT on cp32 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [03:07:55] PROBLEM - ping6 on cp33 is CRITICAL: PING CRITICAL - Packet loss = 100% [03:08:29] PROBLEM - cp33 SSH on cp33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:08:55] PROBLEM - cp33 Stunnel for mwtask141 on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:09:08] PROBLEM - Host cp33 is DOWN: PING CRITICAL - Packet loss = 100% [03:11:05] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [03:11:50] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [03:12:09] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.84, 3.22, 2.24 [03:13:03] RECOVERY - cp22 APT on cp22 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [03:13:42] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:14:05] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.42, 2.52, 2.10 [03:17:08] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:18:17] !log reboot cp33 again (this time from 1&1 as I could not SSH in) [03:18:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:19:05] RECOVERY - swiftobject114 Puppet on swiftobject114 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:26:08] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 3 backends are down. mw122 mw141 mw142 [03:36:55] PROBLEM - cp32 APT on cp32 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [03:37:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.89, 5.68, 3.26 [03:38:55] !log reboot cp32 [03:39:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:41:22] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/26341604edb1 [03:41:23] [url] Page not found · GitHub · GitHub | github.com [03:41:24] [02miraheze/dns] 07Universal-Omega 032634160 - Depool CA [03:41:25] [02dns] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/dns [03:41:26] ... [03:41:27] [02dns] 07Universal-Omega opened pull request 03#361: Depool CA - 13https://github.com/miraheze/dns/pull/361 [03:41:27] [url] Page not found · GitHub · GitHub | github.com [03:41:31] [02dns] 07Universal-Omega closed pull request 03#361: Depool CA - 13https://github.com/miraheze/dns/pull/361 [03:41:32] ... [03:41:33] [02dns] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/dns [03:41:33] [url] Page not found · GitHub · GitHub | github.com [03:41:34] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/50c52fb1823e...4baf99b1a8f2 [03:41:35] [url] Comparing 50c52fb1823e...4baf99b1a8f2 · miraheze/dns · GitHub | github.com [03:41:36] [02miraheze/dns] 07Universal-Omega 034baf99b - Depool CA (#361) [03:41:37] [02miraheze/dns] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [03:41:54] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:42:10] PROBLEM - ping6 on cp32 is CRITICAL: PING CRITICAL - Packet loss = 100% [03:42:18] PROBLEM - cp32 HTTPS on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:42:35] PROBLEM - Host cp32 is DOWN: PING CRITICAL - Packet loss = 100% [04:01:19] PROBLEM - wiki.ieg123.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ieg123.com could not be found [04:28:22] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [04:29:19] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.42, 3.96, 2.49 [04:32:32] RECOVERY - cp23 APT on cp23 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:35:19] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.93, 3.43, 3.00 [04:37:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.92, 2.87, 2.84 [05:51:03] PROBLEM - aresrocket.ml - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'aresrocket.ml' expires in 15 day(s) (Fri 14 Oct 2022 05:26:01 GMT +0000). [05:51:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/523bb328e54f...384a6da2cda8 [05:51:55] [url] Comparing 523bb328e54f...384a6da2cda8 · miraheze/ssl · GitHub | github.com [05:51:56] [02miraheze/ssl] 07MirahezeSSLBot 03384a6da - Bot: Update SSL cert for aresrocket.ml [05:52:47] [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+2/-0/±3] 13https://github.com/miraheze/ssl/compare/384a6da2cda8...670cddb7aaf9 [05:52:48] [url] Comparing 384a6da2cda8...670cddb7aaf9 · miraheze/ssl · GitHub | github.com [05:52:49] [02miraheze/ssl] 07Reception123 03bbae675 - add minescape.wiki cert [05:52:50] [02miraheze/ssl] 07Reception123 0344f1c21 - add wiki.templeofludus.org cert [05:52:52] [02miraheze/ssl] 07Reception123 03670cddb - Merge branch 'master' of https://github.com/miraheze/ssl [05:52:53] [url] GitHub - miraheze/ssl: A repository for storing and managing SSL certificates for Miraheze | github.com [05:58:21] !log RENAMED sparktheelectricjesterwikiwiki to sparktheelectricjesterwiki (T9774) [05:58:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:00:27] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c36669878c0c...7dd3847b5bea [06:00:28] [url] Comparing c36669878c0c...7dd3847b5bea · miraheze/mw-config · GitHub | github.com [06:00:29] [02miraheze/mw-config] 07Reception123 037dd3847 - readonly for CLGwiki for rename [06:00:33] !log [reception@mwtask141] starting deploy of {'config': True} to all [06:00:39] !log [reception@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [06:00:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:00:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:01:30] miraheze/mw-config - Reception123 the build passed. [06:04:06] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/670cddb7aaf9...5a0e860ef4f3 [06:04:06] [url] Comparing 670cddb7aaf9...5a0e860ef4f3 · miraheze/ssl · GitHub | github.com [06:04:07] [02miraheze/ssl] 07Reception123 035a0e860 - redirect closinglogosgroup -> avid [06:10:22] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.13, 1.71, 1.34 [06:10:28] !log [@mwtask141] starting deploy of {'config': True} to all [06:10:33] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 5s [06:10:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:10:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:16:22] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.97, 1.94, 1.57 [06:18:54] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 44.05, 16.52, 8.19 [06:19:47] RECOVERY - aresrocket.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'aresrocket.ml' will expire on Tue 27 Dec 2022 04:51:48 GMT +0000. [06:20:22] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.23, 2.05, 1.70 [06:28:10] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 24.10, 10.05, 4.50 [06:28:22] !log [@test131] starting deploy of {'config': True} to all [06:28:23] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [06:28:25] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 7.18, 6.18, 3.44 [06:28:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:28:41] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:28:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:28:47] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.08, 7.27, 7.21 [06:30:47] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.13, 5.92, 6.73 [06:32:19] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.38, 3.69, 3.04 [06:34:19] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.84, 2.75, 2.78 [06:38:24] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.04, 3.98, 3.28 [06:39:14] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.85, 2.00, 2.00 [06:40:19] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.56, 3.06, 3.02 [06:43:08] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.04, 1.99, 2.00 [06:43:15] RECOVERY - cp23 APT on cp23 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:43:55] !log RENAMED closinglogosgroupwiki to avidwiki (T9771) [06:44:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:44:37] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7dd3847b5bea...923220bd90c5 [06:44:37] [url] Comparing 7dd3847b5bea...923220bd90c5 · miraheze/mw-config · GitHub | github.com [06:44:38] [02miraheze/mw-config] 07Reception123 03923220b - rmv clgwiki readonly [06:45:38] miraheze/mw-config - Reception123 the build passed. [06:47:02] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.94, 1.99, 2.00 [06:50:56] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.05, 1.99, 2.00 [06:58:49] !log [@test131] starting deploy of {'config': True} to all [06:58:50] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [06:58:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:59:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:24] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [07:04:19] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.10, 3.62, 2.77 [07:04:24] RECOVERY - cp22 APT on cp22 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:06:19] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.38, 2.70, 2.53 [07:07:55] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.30, 1.41, 3.74 [07:10:16] !log [@mwtask141] starting deploy of {'config': True} to all [07:10:23] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [07:10:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:10:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:13:55] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.25, 1.89, 3.24 [07:14:27] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 0.95, 1.49, 1.91 [07:18:27] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.04, 1.17, 1.68 [08:14:19] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.19, 3.14, 2.22 [08:15:45] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [08:17:44] RECOVERY - cp22 APT on cp22 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [08:20:19] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.75, 3.85, 3.07 [08:22:19] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.12, 2.95, 2.83 [08:28:53] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [08:29:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.38, 3.74, 2.55 [08:30:48] RECOVERY - cp23 APT on cp23 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [08:31:55] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.63, 3.00, 2.43 [08:42:22] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=avidwiki --user Camenati (END - exit=0) [08:42:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:42:38] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=avidwiki --user Logohub (END - exit=0) [08:42:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:42:52] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=avidwiki --user Mr._Gann (END - exit=0) [08:42:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:43:01] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=avidwiki --user pantsmode (END - exit=0) [08:43:06] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=avidwiki --user Hifisara9007 (END - exit=0) [08:43:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:43:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:43:16] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=avidwiki --user Nico234 (END - exit=0) [08:43:22] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=avidwiki --user Pudsp (END - exit=0) [08:43:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:43:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:43:33] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=avidwiki --user BrandonH1 (END - exit=0) [08:43:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:43:42] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=avidwiki --user Metronome (END - exit=0) [08:43:49] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=avidwiki --user Tmsl01 (END - exit=0) [08:43:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:43:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:44:52] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=avidwiki --user Trevor807 (END - exit=0) [08:44:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:00:23] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=avidwiki --user HifiSara9000seven (END - exit=0) [09:00:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:12:43] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [09:14:43] RECOVERY - cp22 APT on cp22 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [09:43:09] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=avidwiki --user Nico234 (END - exit=0) [09:43:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:44:19] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [09:46:19] RECOVERY - cp22 APT on cp22 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [09:55:38] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.2106679976 secs [10:07:38] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -0.02420645952 secs [10:12:09] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [10:13:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.21, 4.34, 2.73 [10:14:04] RECOVERY - cp23 APT on cp23 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [10:15:55] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.87, 3.39, 2.58 [10:25:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.61, 3.68, 2.86 [10:29:55] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.00, 3.35, 2.96 [10:32:03] PROBLEM - lore.checkit.ws - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - lore.checkit.ws All nameservers failed to answer the query. [10:59:29] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [11:00:47] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.68, 4.94, 3.25 [11:01:23] PROBLEM - lore.checkit.ws - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lore.checkit.ws could not be found [11:01:24] RECOVERY - cp23 APT on cp23 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:02:42] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.14, 3.98, 3.08 [11:04:37] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.97, 3.21, 2.90 [11:17:38] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset -0.1335978508 secs [11:21:38] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -0.07538095117 secs [11:42:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.90, 3.48, 2.50 [11:44:44] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.79, 3.31, 2.57 [11:57:10] !log manually added ipv6 address to cp32, change hostname to cp32 (seems it changed to localhost) and reboot. [11:57:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:58:08] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [11:58:45] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.09, 4.93, 2.98 [12:00:07] RECOVERY - cp23 APT on cp23 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:02:35] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.26, 3.87, 3.02 [12:04:30] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.10, 3.33, 2.92 [12:08:48] RECOVERY - Host cp32 is UP: PING OK - Packet loss = 0%, RTA = 105.39 ms [12:08:59] RECOVERY - ping6 on cp32 is OK: PING OK - Packet loss = 0%, RTA = 105.49 ms [12:09:35] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:09:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.01, 0.05, 0.06 [12:09:35] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:10:05] RECOVERY - cp32 HTTPS on cp32 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3337 bytes in 0.740 second response time [12:10:05] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:11:36] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [12:11:53] !log manually added ipv6 address to cp33, change hostname to cp33 (seems it changed to localhost) and reboot. [12:12:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:14:01] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:14:07] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.09, 3.13, 2.84 [12:15:42] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [12:16:02] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.16, 2.67, 2.70 [12:16:20] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.88, 4.32, 2.84 [12:17:41] RECOVERY - cp22 APT on cp22 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:18:19] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.33, 3.17, 2.60 [12:28:12] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [12:28:46] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.71, 3.88, 2.98 [12:30:07] RECOVERY - cp23 APT on cp23 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:30:41] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.06, 3.20, 2.85 [12:44:09] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [12:48:07] RECOVERY - cp23 APT on cp23 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:53:46] RECOVERY - Host cp33 is UP: PING OK - Packet loss = 0%, RTA = 106.51 ms [12:53:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:53:59] RECOVERY - ping6 on cp33 is OK: PING OK - Packet loss = 0%, RTA = 106.10 ms [12:54:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:55:11] PROBLEM - cp33 Puppet on cp33 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:55:11] RECOVERY - cp33 SSH on cp33 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:55:13] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 7% [12:55:20] RECOVERY - cp33 Stunnel for mwtask141 on cp33 is OK: TCP OK - 0.001 second response time on localhost port 8150 [12:57:03] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [12:57:54] RECOVERY - cp33 HTTPS on cp33 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3337 bytes in 1.143 second response time [12:58:34] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [12:59:07] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/4baf99b1a8f2...c8d66ca63da6 [12:59:08] [url] Comparing 4baf99b1a8f2...c8d66ca63da6 · miraheze/dns · GitHub | github.com [12:59:08] [02miraheze/dns] 07paladox 03c8d66ca - Revert "Depool CA (#361)" [13:03:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 3.27, 2.16 [13:04:59] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [13:05:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.51, 2.40 [13:06:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/c2e236a3a250 [13:06:20] [02miraheze/puppet] 07paladox 03c2e236a - varnish: limit malloc to 256M [13:06:22] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [13:06:22] [url] Page not found · GitHub · GitHub | github.com [13:06:23] [02puppet] 07paladox opened pull request 03#2879: varnish: limit malloc to 256M - 13https://github.com/miraheze/puppet/pull/2879 [13:06:24] [url] Page not found · GitHub · GitHub | github.com [13:06:59] RECOVERY - cp22 APT on cp22 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:07:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.45, 3.25, 2.45 [13:11:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/39831313d9bf [13:11:51] [02miraheze/puppet] 07paladox 033983131 - ssl: disable diff for private keys [13:11:53] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [13:11:53] [url] Page not found · GitHub · GitHub | github.com [13:11:54] [02puppet] 07paladox opened pull request 03#2880: ssl: disable diff for private keys - 13https://github.com/miraheze/puppet/pull/2880 [13:11:55] [url] Page not found · GitHub · GitHub | github.com [13:12:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/39831313d9bf...aa2a0546120c [13:12:28] [url] Comparing 39831313d9bf...aa2a0546120c · miraheze/puppet · GitHub | github.com [13:12:28] [02miraheze/puppet] 07paladox 03aa2a054 - Update wildcard.pp [13:12:30] [02puppet] 07paladox synchronize pull request 03#2880: ssl: disable diff for private keys - 13https://github.com/miraheze/puppet/pull/2880 [13:12:31] [url] Page not found · GitHub · GitHub | github.com [13:13:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/aa2a0546120c...4629b552ff66 [13:13:01] [url] Comparing aa2a0546120c...4629b552ff66 · miraheze/puppet · GitHub | github.com [13:13:02] [02miraheze/puppet] 07paladox 034629b55 - Update cert.pp [13:13:03] [02puppet] 07paladox synchronize pull request 03#2880: ssl: disable diff for private keys - 13https://github.com/miraheze/puppet/pull/2880 [13:13:04] ... [13:13:54] [02puppet] 07paladox closed pull request 03#2880: ssl: disable diff for private keys - 13https://github.com/miraheze/puppet/pull/2880 [13:13:54] ... [13:13:55] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [13:13:57] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [13:13:57] [url] Page not found · GitHub · GitHub | github.com [13:13:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/51b5e299aecd...10ebce45c440 [13:13:59] [url] Comparing 51b5e299aecd...10ebce45c440 · miraheze/puppet · GitHub | github.com [13:14:00] [02miraheze/puppet] 07paladox 0310ebce4 - ssl: disable diff for private keys (#2880) [13:15:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.51, 2.84 [13:18:43] PROBLEM - cp33 Puppet on cp33 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:21:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c2e236a3a250...95551e009ec9 [13:21:32] [url] Comparing c2e236a3a250...95551e009ec9 · miraheze/puppet · GitHub | github.com [13:21:33] [02miraheze/puppet] 07paladox 0395551e0 - Update init.pp [13:21:36] [02puppet] 07paladox synchronize pull request 03#2879: varnish: limit malloc to 256M - 13https://github.com/miraheze/puppet/pull/2879 [13:21:37] [url] Page not found · GitHub · GitHub | github.com [13:22:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/95551e009ec9...6c981591b73c [13:22:03] [url] Comparing 95551e009ec9...6c981591b73c · miraheze/puppet · GitHub | github.com [13:22:03] [02miraheze/puppet] 07paladox 036c98159 - Update varnish.systemd.erb [13:22:05] [02puppet] 07paladox synchronize pull request 03#2879: varnish: limit malloc to 256M - 13https://github.com/miraheze/puppet/pull/2879 [13:22:06] [url] Page not found · GitHub · GitHub | github.com [13:22:33] [02puppet] 07paladox edited pull request 03#2879: varnish: limit malloc to 256M - 13https://github.com/miraheze/puppet/pull/2879 [13:22:33] [url] Page not found · GitHub · GitHub | github.com [13:23:03] [02puppet] 07paladox edited pull request 03#2879: varnish: dynamically set malloc based on the ram available - 13https://github.com/miraheze/puppet/pull/2879 [13:23:04] [url] Page not found · GitHub · GitHub | github.com [13:23:39] [02puppet] 07paladox closed pull request 03#2879: varnish: dynamically set malloc based on the ram available - 13https://github.com/miraheze/puppet/pull/2879 [13:23:40] ... [13:23:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/10ebce45c440...3f1ea4411460 [13:23:41] [url] Comparing 10ebce45c440...3f1ea4411460 · miraheze/puppet · GitHub | github.com [13:23:42] [02miraheze/puppet] 07paladox 033f1ea44 - varnish: dynamically set malloc based on the ram available (#2879) [13:23:44] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [13:23:45] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [13:23:46] [url] Page not found · GitHub · GitHub | github.com [13:24:20] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/c8d66ca63da6...fcf323e15d94 [13:24:21] [url] Comparing c8d66ca63da6...fcf323e15d94 · miraheze/dns · GitHub | github.com [13:24:21] [02miraheze/dns] 07paladox 03fcf323e - Depool gb [13:26:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3f1ea4411460...2975e00f9a4c [13:26:43] [url] Comparing 3f1ea4411460...2975e00f9a4c · miraheze/puppet · GitHub | github.com [13:26:44] [02miraheze/puppet] 07paladox 032975e00 - Fix [13:27:47] PROBLEM - cp22 Puppet on cp22 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:29:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.02, 4.68, 3.36 [13:31:44] RECOVERY - cp22 Puppet on cp22 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:33:09] [02miraheze/dns] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/fcf323e15d94...95989c1ea2a7 [13:33:10] [url] Comparing fcf323e15d94...95989c1ea2a7 · miraheze/dns · GitHub | github.com [13:33:11] [02miraheze/dns] 07paladox 03456fac0 - Revert "Depool gb" [13:33:12] [02miraheze/dns] 07paladox 0395989c1 - Depool ca [13:33:14] PROBLEM - cp32 Puppet on cp32 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 minutes ago with 0 failures [13:35:13] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:46] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.03, 3.64, 2.78 [13:37:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.54, 9.98, 8.37 [13:39:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.16, 8.78, 8.12 [13:39:53] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:01] PROBLEM - cp32 HTTPS on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:05] PROBLEM - ping6 on cp33 is CRITICAL: PING CRITICAL - Packet loss = 100% [13:40:13] PROBLEM - cp33 SSH on cp33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2607:f1c0:1800:8100::1/cpweb, 2607:f1c0:1800:26f::1/cpweb [13:40:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2607:f1c0:1800:8100::1/cpweb, 2607:f1c0:1800:26f::1/cpweb [13:41:07] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:41:11] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:41:19] PROBLEM - cp33 HTTPS on cp33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:41:20] PROBLEM - ping6 on cp32 is CRITICAL: PING CRITICAL - Packet loss = 100% [13:41:54] PROBLEM - Host cp32 is DOWN: PING CRITICAL - Packet loss = 100% [13:41:54] PROBLEM - Host cp33 is DOWN: PING CRITICAL - Packet loss = 100% [13:42:25] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [13:42:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:42:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:42:31] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.00, 3.68, 3.10 [13:43:56] RECOVERY - Host cp33 is UP: PING OK - Packet loss = 0%, RTA = 106.78 ms [13:43:56] RECOVERY - Host cp32 is UP: PING OK - Packet loss = 0%, RTA = 105.83 ms [13:44:07] RECOVERY - ping6 on cp33 is OK: PING OK - Packet loss = 0%, RTA = 105.96 ms [13:44:07] RECOVERY - cp33 SSH on cp33 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:44:07] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [13:44:25] RECOVERY - cp23 APT on cp23 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:44:26] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.75, 3.31, 3.03 [13:44:57] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 80% [13:44:57] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 89% [13:45:04] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/95989c1ea2a7...5b94071c31f4 [13:45:04] [url] Comparing 95989c1ea2a7...5b94071c31f4 · miraheze/dns · GitHub | github.com [13:45:05] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:45:05] [02miraheze/dns] 07paladox 035b94071 - Revert "Depool ca" [13:45:27] RECOVERY - ping6 on cp32 is OK: PING OK - Packet loss = 0%, RTA = 105.29 ms [13:45:52] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:46:01] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:46:03] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [13:46:06] RECOVERY - cp32 HTTPS on cp32 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3341 bytes in 0.517 second response time [13:46:52] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 3% [13:46:55] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 3% [13:47:10] RECOVERY - cp33 HTTPS on cp33 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3342 bytes in 0.886 second response time [13:47:11] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [13:59:21]