[00:01:21] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:03:21] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:04:08] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 20715MiB (23% inode=98%); [00:05:02] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 20711MiB (23% inode=98%); [00:05:46] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 23327MiB (25% inode=98%); [00:11:38] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:31:28] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/e2228da52a09...f05a0fdd3078 [00:31:30] [02miraheze/dns] 07Universal-Omega 03f05a0fd - wikitide.net: add ldap cname [00:31:45] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/f05a0fdd3078...b814ee7e048f [00:31:47] [02miraheze/dns] 07Universal-Omega 03b814ee7 - Fix [00:37:05] [02puppet] 07Universal-Omega closed pull request 03#3779: ldap: use ldap.wikitide.net - 13https://github.com/miraheze/puppet/pull/3779 [00:37:07] [02miraheze/puppet] 07Universal-Omega deleted branch 03ldap [00:37:08] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/0bf0420d88e9...f6e04bf5c483 [00:37:09] [02miraheze/puppet] 07Universal-Omega 03f6e04bf - ldap: use ldap.wikitide.net (#3779) [00:37:10] [02puppet] 07Universal-Omega deleted branch 03ldap - 13https://github.com/miraheze/puppet [01:07:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ee9f67b8a9ab...acec0b9e2f0f [01:07:17] [02miraheze/mw-config] 07Universal-Omega 03acec0b9 - Update LDAP host [01:08:06] miraheze/mw-config - Universal-Omega the build passed. [01:10:37] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:14:56] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=elysiumwiki (END - exit=0) [01:15:00] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=elysiumwiki --update (END - exit=0) [01:15:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:15:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:15:35] that was quick [01:22:57] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/affc2582ad20...e41f9bd146ae [01:22:58] [02miraheze/ssl] 07WikiTideSSLBot 03e41f9bd - Bot: Add SSL cert for wiki.ate42.ru [01:23:01] !log [@test151] starting deploy of {'config': True} to test151 [01:23:02] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [01:23:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:29:32] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/e41f9bd146ae...7fab44aec9a2 [01:29:35] [02miraheze/ssl] 07WikiTideSSLBot 037fab44a - Bot: Add SSL cert for wiki.maribelhearn.com [01:32:41] !log [@mwtask181] starting deploy of {'config': True} to all [01:32:42] !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [01:32:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:32:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:28] [02miraheze/puppet] 07Universal-Omega created branch 03opensearch 13https://github.com/miraheze/puppet/commit/f6e04bf5c483a0e495266196af81f209b0755880 [01:34:30] [02puppet] 07Universal-Omega created branch 03opensearch - 13https://github.com/miraheze/puppet [01:37:00] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03opensearch [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/f6e04bf5c483...b1b1e45c8ef2 [01:37:01] [02miraheze/puppet] 07Universal-Omega 03b1b1e45 - opensearch: use opensearch.wikitide.net [01:37:29] !log [@mwtask171] starting deploy of {'config': True} to all [01:37:38] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [01:37:55] [02puppet] 07Universal-Omega opened pull request 03#3809: opensearch: use opensearch.wikitide.net - 13https://github.com/miraheze/puppet/pull/3809 [01:38:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:38:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:38:44] [02puppet] 07Universal-Omega closed pull request 03#3809: opensearch: use opensearch.wikitide.net - 13https://github.com/miraheze/puppet/pull/3809 [01:38:46] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/f6e04bf5c483...261d6031e013 [01:38:48] [02miraheze/puppet] 07Universal-Omega 03261d603 - opensearch: use opensearch.wikitide.net (#3809) [01:38:50] [02miraheze/puppet] 07Universal-Omega deleted branch 03opensearch [01:38:53] [02puppet] 07Universal-Omega deleted branch 03opensearch - 13https://github.com/miraheze/puppet [01:40:37] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/b814ee7e048f...9a9884ece8dd [01:40:39] [02miraheze/dns] 07Universal-Omega 039a9884e - wikitide.ne: add opensearch cname [01:41:55] PROBLEM - os151 Puppet on os151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [01:42:18] PROBLEM - os162 Puppet on os162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [01:46:36] [02dns] 07MacFan4000 opened pull request 03#504: Add 3 zones - 13https://github.com/miraheze/dns/pull/504 [01:48:18] [02dns] 07Universal-Omega closed pull request 03#504: Add 3 zones - 13https://github.com/miraheze/dns/pull/504 [01:48:20] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+3/-0/±0] 13https://github.com/miraheze/dns/compare/9a9884ece8dd...851754071b42 [01:48:23] [02miraheze/dns] 07MacFan4000 038517540 - Add 3 zones (#504) [02:01:31] PROBLEM - graylog161 HTTPS on graylog161 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 113 bytes in 0.012 second response time [02:02:08] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [02:10:09] RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [02:28:05] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/261d6031e013...2f155997f47c [02:28:08] [02miraheze/puppet] 07Universal-Omega 032f15599 - Remove miraheze.org from resolver [02:31:31] [02puppet] 07Universal-Omega closed pull request 03#3805: jobrunner: add some apache config - 13https://github.com/miraheze/puppet/pull/3805 [02:31:34] [02miraheze/puppet] 07Universal-Omega deleted branch 03jobrunner-apache [02:31:37] [02puppet] 07Universal-Omega deleted branch 03jobrunner-apache - 13https://github.com/miraheze/puppet [02:32:16] [02puppet] 07Universal-Omega closed pull request 03#3799: changeprop: add support for clustering - 13https://github.com/miraheze/puppet/pull/3799 [02:32:19] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/2f155997f47c...a1c68838b858 [02:32:22] [02miraheze/puppet] 07Universal-Omega 03a1c6883 - changeprop: add support for clustering (#3799) [02:32:24] [02miraheze/puppet] 07Universal-Omega deleted branch 03cp-cluster [02:32:25] [02puppet] 07Universal-Omega deleted branch 03cp-cluster - 13https://github.com/miraheze/puppet [02:36:19] PROBLEM - changeprop151 changeprop on changeprop151 is CRITICAL: connect to address 10.0.15.148 and port 7200: Connection refused [02:38:05] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a1c68838b858...0cf70f54bb87 [02:38:06] [02miraheze/puppet] 07Universal-Omega 030cf70f5 - changeprop: fix partition_default [02:40:19] RECOVERY - changeprop151 changeprop on changeprop151 is OK: TCP OK - 0.001 second response time on 10.0.15.148 port 7200 [02:43:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/acec0b9e2f0f...53986dfe7ebf [02:43:28] [02miraheze/mw-config] 07Universal-Omega 0353986df - Temp disable QuickInstantCommons [02:43:44] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [02:43:45] !log [universalomega@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [02:43:48] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [02:43:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:43:59] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [02:44:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:44:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:44:12] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.323 second response time [02:44:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:44:23] miraheze/mw-config - Universal-Omega the build passed. [02:46:54] PROBLEM - kinnikupedia.org - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [02:53:36] !log [@test151] starting deploy of {'config': True} to test151 [02:53:37] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [02:53:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:53:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:55:05] PROBLEM - wiki.cityrp.org - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [03:00:13] PROBLEM - replication-ops.com - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [03:02:35] PROBLEM - wiki.manos.pt - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [03:07:15] !log [@mwtask171] starting deploy of {'config': True} to all [03:07:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:07:23] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 7s [03:07:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:08:17] PROBLEM - wiki.aetherexplorers.science - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [03:54:30] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.15, 19.23, 14.07 [03:57:30] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.35, 19.78, 14.19 [03:58:30] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.86, 22.77, 16.86 [03:59:29] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.45, 6.41, 5.16 [03:59:30] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 12.25, 16.86, 13.81 [04:00:30] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.21, 19.08, 16.24 [04:01:29] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.34, 6.06, 5.19 [04:18:24] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/7fab44aec9a2...17ac4f5cbd4a [04:18:25] [02miraheze/ssl] 07WikiTideSSLBot 0317ac4f5 - Bot: Add SSL cert for exordium.wiki [04:18:58] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/17ac4f5cbd4a...d566c9d971b3 [04:18:59] [02miraheze/ssl] 07WikiTideSSLBot 03d566c9d - Bot: Add SSL cert for wiltshire.xyz [04:19:25] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/d566c9d971b3...17044bfb7483 [04:19:28] [02miraheze/ssl] 07WikiTideSSLBot 0317044bf - Bot: Add SSL cert for rainverse.wiki [04:34:50] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [04:36:22] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 174.12 ms [04:38:25] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 175.65 ms [04:42:20] [02mw-config] 07mic5555 commented on commit 0353986dfe7ebfa0b7c96d7556bfb83196aec240ff - 13https://github.com/miraheze/mw-config/commit/53986dfe7ebfa0b7c96d7556bfb83196aec240ff#commitcomment-140363800 [04:48:33] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 174.20 ms [04:50:33] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 178.63 ms [04:56:16] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.03, 23.56, 17.60 [04:57:09] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.42, 19.03, 15.31 [04:58:30] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.28, 20.36, 15.90 [04:59:08] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.60, 18.85, 15.69 [05:00:15] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.12, 21.51, 17.43 [05:02:03] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 15.25, 22.30, 19.25 [05:02:15] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.36, 20.75, 17.69 [05:02:26] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:03:38] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.59, 6.87, 6.04 [05:03:59] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.63, 24.24, 20.31 [05:04:54] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.64, 20.56, 15.82 [05:05:02] PROBLEM - ping on changeprop151 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 575.17 ms [05:05:05] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.97, 22.25, 18.19 [05:05:37] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.30, 20.73, 16.18 [05:06:15] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.41, 23.72, 19.45 [05:06:54] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.29, 18.86, 15.80 [05:07:03] RECOVERY - ping on changeprop151 is OK: PING OK - Packet loss = 0%, RTA = 0.30 ms [05:09:03] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.16, 23.06, 19.47 [05:09:27] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.24, 7.56, 6.52 [05:09:37] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.32, 22.95, 18.22 [05:10:23] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.25, 22.78, 20.36 [05:11:02] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.57, 24.11, 20.29 [05:11:24] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.41, 7.19, 6.52 [05:14:15] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 17.25, 21.71, 20.77 [05:15:00] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.30, 23.56, 20.99 [05:15:37] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.16, 18.43, 17.93 [05:16:20] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.61, 19.15, 19.57 [05:17:50] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/d79c847b93a3...df0067975140 [05:17:52] [02miraheze/mediawiki-repos] 07Universal-Omega 03df00679 - Use branch for QuickInstantCommons [05:18:15] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 13.57, 19.06, 20.06 [05:18:58] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.70, 18.07, 19.31 [05:19:32] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.42, 23.78, 23.69 [05:24:59] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.32, 7.09, 6.77 [05:26:55] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.38, 6.40, 6.56 [05:27:32] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.21, 16.90, 20.39 [05:33:32] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.94, 21.73, 21.27 [05:34:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.28, 7.30, 6.90 [05:37:07] [02mw-config] 07Universal-Omega commented on commit 0353986dfe7ebfa0b7c96d7556bfb83196aec240ff - 13https://github.com/miraheze/mw-config/commit/53986dfe7ebfa0b7c96d7556bfb83196aec240ff#commitcomment-140364671 [05:38:11] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.70, 22.30, 19.04 [05:38:15] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.81, 23.34, 20.02 [05:38:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.72, 21.83, 19.07 [05:40:03] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.51, 18.18, 15.56 [05:40:11] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.52, 22.67, 19.61 [05:40:47] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.00, 22.23, 19.57 [05:41:37] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.66, 20.47, 18.13 [05:41:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/53986dfe7ebf...29358cc083dc [05:41:54] [02miraheze/mw-config] 07Universal-Omega 0329358cc - Re-enable QuickInstantCommons [05:42:03] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.88, 17.45, 15.63 [05:42:11] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.20, 23.20, 20.15 [05:42:28] !log [universalomega@mwtask181] starting deploy of {'world': True, 'versions': '1.41'} to all [05:42:30] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.91, 7.75, 7.44 [05:42:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:42:42] miraheze/mw-config - Universal-Omega the build passed. [05:42:43] !log [universalomega@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [05:42:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:42:54] !log [universalomega@mwtask181] starting deploy of {'world': True, 'versions': '1.41'} to mwtask181 [05:43:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:43:12] !log [universalomega@mwtask181] finished deploy of {'world': True, 'versions': '1.41'} to mwtask181 - SUCCESS in 18s [05:43:21] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [05:43:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:43:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:43:41] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 19s [05:43:43] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [05:44:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:44:06] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [05:44:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:44:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:44:46] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.06, 23.35, 20.64 [05:48:11] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 10.96, 20.09, 20.14 [05:48:15] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 14.45, 23.94, 22.48 [05:48:44] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 8.75, 18.87, 19.71 [05:49:26] PROBLEM - cp41 Current Load on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:49:32] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 6.22, 19.86, 23.05 [05:49:38] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 4.32, 14.90, 17.23 [05:51:22] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.12, 6.11, 6.76 [05:52:15] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.53, 23.58, 22.31 [05:53:19] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.21, 6.55, 6.83 [05:53:32] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.21, 23.76, 23.81 [05:53:50] !log [@test151] starting deploy of {'config': True} to test151 [05:53:51] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [05:54:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:54:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:54:40] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.17, 20.70, 20.05 [05:56:11] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.55, 20.25, 19.65 [05:57:11] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.33, 7.09, 7.02 [05:58:11] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.29, 22.39, 20.50 [05:58:15] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.58, 21.77, 22.14 [06:02:11] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.28, 22.86, 21.10 [06:02:15] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.09, 22.87, 22.47 [06:03:01] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.57, 6.38, 6.75 [06:04:11] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.34, 22.82, 21.26 [06:04:20] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.61, 20.56, 18.98 [06:04:36] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.20, 23.52, 21.78 [06:04:58] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 173.38 ms [06:06:54] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.09, 20.62, 18.22 [06:06:56] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.77, 7.08, 6.97 [06:07:28] !log [@mwtask171] starting deploy of {'config': True} to all [06:07:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:07:45] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 17s [06:08:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:08:10] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.81, 22.17, 20.09 [06:09:02] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.77 ms [06:10:49] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.34, 8.35, 7.46 [06:10:54] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.44, 19.67, 18.39 [06:12:01] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.70, 22.09, 20.43 [06:13:56] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.00, 22.45, 20.77 [06:14:42] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.11, 7.41, 7.34 [06:14:54] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.80, 20.86, 19.24 [06:16:54] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.96, 19.24, 18.85 [06:18:11] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.47, 22.73, 22.98 [06:18:35] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.16, 7.68, 7.45 [06:20:11] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.36, 24.59, 23.63 [06:21:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:21:37] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.51, 23.85, 21.65 [06:22:29] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.20, 7.29, 7.33 [06:22:44] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.27, 21.18, 17.81 [06:22:53] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 28.63, 23.62, 19.91 [06:22:54] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 29.11, 24.24, 20.99 [06:24:27] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.31, 7.68, 7.46 [06:24:44] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.01, 19.66, 17.65 [06:24:50] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.30, 23.13, 20.21 [06:24:54] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.36, 23.53, 21.12 [06:25:37] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.07, 23.06, 21.94 [06:27:41] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/17044bfb7483...c9531e39cc43 [06:27:43] [02miraheze/ssl] 07WikiTideSSLBot 03c9531e3 - Bot: Add SSL cert for spiceandwolf.wiki [06:28:27] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.65, 7.76, 7.55 [06:28:43] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.52, 20.28, 19.74 [06:28:54] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.40, 20.06, 20.25 [06:32:27] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.31, 8.59, 7.84 [06:33:37] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.18, 22.91, 22.11 [06:37:37] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.80, 23.56, 22.60 [06:38:54] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.24, 20.53, 20.40 [06:40:54] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.50, 19.76, 20.12 [06:41:32] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.50, 21.53, 23.65 [06:51:36] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.17, 22.51, 22.75 [06:52:46] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.24, 21.20, 20.72 [06:53:33] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.35, 22.01, 22.54 [06:57:26] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.33, 23.07, 22.82 [06:57:30] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.22, 17.73, 19.86 [06:59:06] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.92, 20.78, 19.55 [06:59:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.63, 21.74, 22.37 [07:01:06] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 14.53, 18.42, 18.83 [07:02:14] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.56, 7.28, 7.88 [07:02:22] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.57, 21.88, 21.26 [07:04:14] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.03, 7.52, 7.89 [07:04:17] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 18.06, 20.76, 20.95 [07:05:20] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.79, 21.79, 20.72 [07:06:12] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 13.56, 19.04, 20.34 [07:06:14] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.80, 7.27, 7.78 [07:07:09] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.21, 17.96, 20.30 [07:07:20] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.99, 18.90, 19.75 [07:07:58] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.64, 21.63, 23.73 [07:09:38] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 12.36, 20.85, 24.00 [07:10:01] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 15.05, 20.59, 23.72 [07:11:34] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.53, 22.67, 24.27 [07:11:55] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.15, 23.18, 23.74 [07:12:00] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.04, 23.60, 24.45 [07:12:14] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.40, 5.49, 6.78 [07:13:01] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.59, 20.13, 20.37 [07:14:53] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.10, 21.58, 20.54 [07:15:20] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.28, 22.41, 20.67 [07:15:35] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.06, 20.70, 18.32 [07:15:53] PROBLEM - ping on changeprop151 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 1146.84 ms [07:15:58] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.92, 22.73, 23.95 [07:16:35] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/851754071b42...2f9fd625542c [07:16:38] [02miraheze/dns] 07AgentIsai 032f9fd62 - Mark Japan as always up [07:17:20] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 15.40, 20.41, 20.20 [07:17:30] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 12.46, 18.42, 17.83 [07:17:50] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 13.59, 22.68, 23.80 [07:17:51] RECOVERY - ping on changeprop151 is OK: PING OK - Packet loss = 0%, RTA = 0.59 ms [07:18:43] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 7.73, 17.20, 19.27 [07:18:51] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 7.21, 17.61, 19.81 [07:19:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 5.60, 17.72, 22.35 [07:19:20] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 4.47, 14.54, 18.07 [07:19:42] PROBLEM - cp41 Varnish Backends on cp41 is WARNING: No backends detected. If this is an error, see readme.txt [07:19:55] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:19:56] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 4.65, 13.95, 20.25 [07:20:40] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is CRITICAL: CRITICAL - NGINX Error Rate is 99% [07:21:17] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 3.03, 12.75, 19.97 [07:21:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:21:46] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 3.70, 12.20, 19.24 [07:22:51] PROBLEM - cp41 Nginx Backend for mw172 on cp41 is CRITICAL: connect to address localhost and port 8118: Connection refused [07:22:58] PROBLEM - cp41 Nginx Backend for mw152 on cp41 is CRITICAL: connect to address localhost and port 8114: Connection refused [07:22:59] PROBLEM - cp41 Nginx Backend for matomo151 on cp41 is CRITICAL: connect to address localhost and port 8203: Connection refused [07:23:02] PROBLEM - cp41 Nginx Backend for mw182 on cp41 is CRITICAL: connect to address localhost and port 8120: Connection refused [07:23:08] PROBLEM - cp41 Nginx Backend for reports171 on cp41 is CRITICAL: connect to address localhost and port 8205: Connection refused [07:23:11] PROBLEM - cp41 Nginx Backend for swiftproxy171 on cp41 is CRITICAL: connect to address localhost and port 8207: Connection refused [07:23:13] PROBLEM - cp41 Nginx Backend for mwtask181 on cp41 is CRITICAL: connect to address localhost and port 8160: Connection refused [07:23:13] PROBLEM - cp41 Nginx Backend for test151 on cp41 is CRITICAL: connect to address localhost and port 8181: Connection refused [07:23:51] PROBLEM - cp41 Nginx Backend for mw151 on cp41 is CRITICAL: connect to address localhost and port 8113: Connection refused [07:23:58] PROBLEM - cp41 Nginx Backend for mwtask171 on cp41 is CRITICAL: connect to address localhost and port 8161: Connection refused [07:24:01] PROBLEM - cp41 Nginx Backend for mw181 on cp41 is CRITICAL: connect to address localhost and port 8119: Connection refused [07:24:06] PROBLEM - cp41 Nginx Backend for mw171 on cp41 is CRITICAL: connect to address localhost and port 8117: Connection refused [07:24:23] PROBLEM - cp41 Nginx Backend for mw162 on cp41 is CRITICAL: connect to address localhost and port 8116: Connection refused [07:24:32] PROBLEM - cp41 Nginx Backend for mon181 on cp41 is CRITICAL: connect to address localhost and port 8201: Connection refused [07:24:33] PROBLEM - cp41 Nginx Backend for swiftproxy161 on cp41 is CRITICAL: connect to address localhost and port 8206: Connection refused [07:24:36] PROBLEM - cp41 Nginx Backend for puppet181 on cp41 is CRITICAL: connect to address localhost and port 8204: Connection refused [07:24:37] PROBLEM - cp41 Nginx Backend for phorge171 on cp41 is CRITICAL: connect to address localhost and port 8202: Connection refused [07:24:40] PROBLEM - cp41 Nginx Backend for mw161 on cp41 is CRITICAL: connect to address localhost and port 8115: Connection refused [07:25:47] RECOVERY - cp41 Nginx Backend for mw151 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8113 [07:25:55] RECOVERY - cp41 Nginx Backend for mwtask171 on cp41 is OK: TCP OK - 0.001 second response time on localhost port 8161 [07:26:02] RECOVERY - cp41 Nginx Backend for mw181 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8119 [07:26:06] RECOVERY - cp41 Nginx Backend for mw171 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8117 [07:26:20] RECOVERY - cp41 Nginx Backend for mw162 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8116 [07:26:28] RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 16% [07:26:30] RECOVERY - cp41 Nginx Backend for mon181 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8201 [07:26:31] RECOVERY - cp41 Nginx Backend for swiftproxy161 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8206 [07:26:35] RECOVERY - cp41 Nginx Backend for puppet181 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8204 [07:26:38] RECOVERY - cp41 Nginx Backend for phorge171 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8202 [07:26:39] RECOVERY - cp41 Nginx Backend for mw161 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8115 [07:26:51] RECOVERY - cp41 Nginx Backend for mw172 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8118 [07:26:53] RECOVERY - cp41 Nginx Backend for mw152 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8114 [07:26:56] RECOVERY - cp41 Nginx Backend for matomo151 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8203 [07:27:00] RECOVERY - cp41 Nginx Backend for mw182 on cp41 is OK: TCP OK - 0.001 second response time on localhost port 8120 [07:27:05] RECOVERY - cp41 Nginx Backend for swiftproxy171 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8207 [07:27:07] RECOVERY - cp41 Nginx Backend for reports171 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8205 [07:27:08] RECOVERY - cp41 Nginx Backend for test151 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8181 [07:27:13] RECOVERY - cp41 Nginx Backend for mwtask181 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8160 [07:27:28] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [07:27:35] RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3636 bytes in 0.849 second response time [07:29:05] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.99, 16.18, 17.65 [07:29:37] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.27, 17.95, 18.05 [07:29:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.23, 17.73, 17.50 [07:30:51] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.81, 16.84, 16.27 [07:31:02] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.82, 18.62, 18.40 [07:31:20] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.61, 16.83, 15.71 [07:32:58] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.16, 20.89, 19.23 [07:34:51] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.76, 18.84, 17.31 [07:35:32] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.15, 23.02, 20.42 [07:36:57] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.63, 23.12, 20.61 [07:37:20] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 19.93, 20.00, 17.68 [07:37:30] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.87, 6.35, 5.52 [07:37:47] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.68, 22.93, 20.50 [07:39:26] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.27, 6.42, 5.65 [07:39:29] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.46, 23.05, 20.96 [07:40:57] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.75, 24.70, 21.76 [07:41:45] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.21, 24.28, 21.61 [07:45:24] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.88, 23.79, 22.23 [07:45:43] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 12.95, 21.59, 21.33 [07:46:57] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 4.64, 17.50, 20.24 [07:46:57] PROBLEM - cp41 Nginx Backend for swiftproxy171 on cp41 is CRITICAL: connect to address localhost and port 8207: Connection refused [07:47:07] PROBLEM - cp41 Nginx Backend for reports171 on cp41 is CRITICAL: connect to address localhost and port 8205: Connection refused [07:47:07] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp41.wikitide.net port 443 after 134 ms: Couldn't connect to server [07:47:13] PROBLEM - cp41 Nginx Backend for mwtask181 on cp41 is CRITICAL: connect to address localhost and port 8160: Connection refused [07:47:14] PROBLEM - cp41 Nginx Backend for mw151 on cp41 is CRITICAL: connect to address localhost and port 8113: Connection refused [07:47:22] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 3.88, 16.49, 19.74 [07:47:26] PROBLEM - cp41 Nginx Backend for mwtask171 on cp41 is CRITICAL: connect to address localhost and port 8161: Connection refused [07:47:42] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 2.91, 14.97, 18.94 [07:47:47] PROBLEM - cp41 Nginx Backend for mw162 on cp41 is CRITICAL: connect to address localhost and port 8116: Connection refused [07:48:01] PROBLEM - cp41 Nginx Backend for mw181 on cp41 is CRITICAL: connect to address localhost and port 8119: Connection refused [07:48:04] PROBLEM - cp41 Nginx Backend for mon181 on cp41 is CRITICAL: connect to address localhost and port 8201: Connection refused [07:48:05] PROBLEM - cp41 Nginx Backend for mw171 on cp41 is CRITICAL: connect to address localhost and port 8117: Connection refused [07:48:08] PROBLEM - cp41 Nginx Backend for swiftproxy161 on cp41 is CRITICAL: connect to address localhost and port 8206: Connection refused [07:48:16] PROBLEM - cp41 Nginx Backend for puppet181 on cp41 is CRITICAL: connect to address localhost and port 8204: Connection refused [07:48:20] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 40% [07:48:29] PROBLEM - cp41 Nginx Backend for mw161 on cp41 is CRITICAL: connect to address localhost and port 8115: Connection refused [07:48:37] PROBLEM - cp41 Nginx Backend for phorge171 on cp41 is CRITICAL: connect to address localhost and port 8202: Connection refused [07:48:38] PROBLEM - cp41 Nginx Backend for matomo151 on cp41 is CRITICAL: connect to address localhost and port 8203: Connection refused [07:48:39] PROBLEM - cp41 Nginx Backend for test151 on cp41 is CRITICAL: connect to address localhost and port 8181: Connection refused [07:48:44] PROBLEM - cp41 Nginx Backend for mw182 on cp41 is CRITICAL: connect to address localhost and port 8120: Connection refused [07:48:51] PROBLEM - cp41 Nginx Backend for mw172 on cp41 is CRITICAL: connect to address localhost and port 8118: Connection refused [07:48:52] PROBLEM - cp41 Nginx Backend for mw152 on cp41 is CRITICAL: connect to address localhost and port 8114: Connection refused [07:48:57] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [07:49:55] PROBLEM - cp41 Puppet on cp41 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 14 minutes ago with 0 failures [08:01:18] PROBLEM - ping on changeprop151 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 464.02 ms [08:05:13] RECOVERY - ping on changeprop151 is OK: PING OK - Packet loss = 0%, RTA = 0.27 ms [08:39:55] PROBLEM - ping on changeprop151 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 228.45 ms [08:41:53] RECOVERY - ping on changeprop151 is OK: PING OK - Packet loss = 0%, RTA = 0.24 ms [09:37:49] PROBLEM - db181 Backups SQL on db181 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864134 seconds old and 115136 bytes [09:39:31] PROBLEM - ping on changeprop151 is CRITICAL: PING CRITICAL - Packet loss = 25%, RTA = 1150.18 ms [09:41:29] RECOVERY - ping on changeprop151 is OK: PING OK - Packet loss = 0%, RTA = 0.23 ms [10:29:42] [1/5] farthestfrontierwiki: ``` [10:29:42] [2/5] Found: 1214 [10:29:43] [3/5] Added: 498 [10:29:43] [4/5] Skipped: 510 [10:29:43]