[00:03:01] RECOVERY - www.lgbtqia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lgbtqia.wiki' will expire on Thu 08 Aug 2024 10:24:14 PM GMT +0000. [00:39:28] How on Earth does it take 10 hours to transfer less than 5% of 20GB from one server to another... [00:47:15] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 14 minutes ago with 0 failures [00:47:35] I’m getting flashbacks to MHL servers [01:02:37] PROBLEM - mwtask181 ferm_active on mwtask181 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly [01:04:11] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [01:04:37] RECOVERY - mwtask181 ferm_active on mwtask181 is OK: OK ferm input default policy is set [01:04:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 46.250.240.167/cpweb, 2407:3641:2161:9774::1/cpweb [01:04:57] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 100% [01:05:08] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 46.250.240.167/cpweb, 2407:3641:2161:9774::1/cpweb [01:05:59] PROBLEM - cp51 Nginx Backend for swiftproxy161 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:15] PROBLEM - cp51 Nginx Backend for puppet181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:27] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:35] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:39] I guess it takes so long because I was not doing server to server but server to my phone and then my phone ran out of storage lmao... big whoops [01:06:42] PROBLEM - cp51 ferm_active on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:44] PROBLEM - cp51 Nginx Backend for test151 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:50] PROBLEM - cp51 Nginx Backend for mw182 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [01:07:03] PROBLEM - cp51 Nginx Backend for mw161 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:07:05] PROBLEM - cp51 Nginx Backend for phorge171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:07:09] PROBLEM - cp51 Nginx Backend for mw152 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:07:19] PROBLEM - cp51 Nginx Backend for mw151 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:07:21] PROBLEM - cp51 conntrack_table_size on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:07:35] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [01:07:41] PROBLEM - cp51 Nginx Backend for mwtask171 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [01:07:46] PROBLEM - cp51 Nginx Backend for matomo151 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [01:07:59] PROBLEM - cp51 Nginx Backend for mw162 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:08:06] PROBLEM - cp51 Nginx Backend for mon181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:08:07] PROBLEM - cp51 NTP time on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:08:07] PROBLEM - cp51 Nginx Backend for mw171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:08:08] RECOVERY - cp51 Nginx Backend for swiftproxy161 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8206 [01:08:11] PROBLEM - cp51 PowerDNS Recursor on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:08:21] PROBLEM - cp51 Disk Space on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:09:01] It took 4 minutes to do all of server -> intermediate server -> final destination server lol [01:09:05] PROBLEM - cp51 Current Load on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:09:30] RECOVERY - cp51 Nginx Backend for test151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8181 [01:09:36] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 4% [01:09:36] RECOVERY - cp51 Nginx Backend for phorge171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8202 [01:12:12] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=dcmultiversewiki --search-recursively --skip-dupes /home/universalomega/dcmultiversewiki (START) [01:12:14] RECOVERY - cp51 ferm_active on cp51 is OK: OK ferm input default policy is set [01:12:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:18] RECOVERY - cp51 Nginx Backend for mw182 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8120 [01:12:27] RECOVERY - cp51 Nginx Backend for mw152 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8114 [01:12:34] RECOVERY - cp51 Nginx Backend for mon181 on cp51 is OK: TCP OK - 0.001 second response time on localhost port 8201 [01:12:44] RECOVERY - cp51 Nginx Backend for mw151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8113 [01:13:15] PROBLEM - cp51 SSH on cp51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:13:52] PROBLEM - cp51 Nginx Backend for mw172 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:13:52] PROBLEM - cp51 Nginx Backend for reports171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:16:03] PROBLEM - cp51 Nginx Backend for test151 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [01:16:30] PROBLEM - cp51 Nginx Backend for phorge171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:16:38] RECOVERY - cp51 Nginx Backend for reports171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8205 [01:16:58] RECOVERY - cp51 Nginx Backend for mwtask171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8161 [01:17:34] RECOVERY - cp51 Nginx Backend for matomo151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8203 [01:17:46] RECOVERY - cp51 Nginx Backend for puppet181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8204 [01:18:01] RECOVERY - cp51 Current Load on cp51 is OK: LOAD OK - total load average: 0.22, 0.17, 0.11 [01:18:05] PROBLEM - cp51 Nginx Backend for mw181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:18:16] PROBLEM - cp51 Nginx Backend for mwtask181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:18:38] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:18:46] RECOVERY - cp51 conntrack_table_size on cp51 is OK: OK: nf_conntrack is 0 % full [01:18:53] RECOVERY - cp51 Nginx Backend for mw162 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8116 [01:19:41] PROBLEM - cp51 Nginx Backend for mw151 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:20:01] RECOVERY - cp51 Nginx Backend for phorge171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8202 [01:20:18] RECOVERY - cp51 Nginx Backend for mw181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8119 [01:20:19] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 19 minutes ago with 0 failures [01:20:54] RECOVERY - cp51 Nginx Backend for mwtask181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8160 [01:21:26] RECOVERY - cp51 Nginx Backend for mw172 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8118 [01:21:39] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 13% [01:22:29] RECOVERY - cp51 Disk Space on cp51 is OK: DISK OK - free space: / 18313MiB (19% inode=98%); [01:22:33] PROBLEM - cp51 Nginx Backend for mw152 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:24:13] PROBLEM - cp51 Nginx Backend for puppet181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:24:27] PROBLEM - cp51 Nginx Backend for matomo151 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:24:40] RECOVERY - cp51 Nginx Backend for mw151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8113 [01:24:54] PROBLEM - cp51 Current Load on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:25:06] PROBLEM - cp51 NTP time on cp51 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [01:25:38] PROBLEM - cp51 conntrack_table_size on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:26:11] PROBLEM - cp51 ferm_active on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:26:24] PROBLEM - cp51 Nginx Backend for swiftproxy161 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:26:31] PROBLEM - cp51 Nginx Backend for mwtask181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:26:48] RECOVERY - cp51 Nginx Backend for puppet181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8204 [01:26:53] RECOVERY - cp51 Nginx Backend for matomo151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8203 [01:27:15] RECOVERY - cp51 Nginx Backend for test151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8181 [01:27:22] RECOVERY - cp51 Nginx Backend for mw161 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8115 [01:27:48] RECOVERY - cp51 Nginx Backend for mw171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8117 [01:28:01] PROBLEM - cp51 NTP time on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:28:18] PROBLEM - cp51 Nginx Backend for swiftproxy171 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [01:28:18] PROBLEM - cp51 Nginx Backend for mw172 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:28:26] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:29:08] RECOVERY - cp51 Nginx Backend for mwtask181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8160 [01:30:09] PROBLEM - cp51 Nginx Backend for mon181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:30:18] PROBLEM - cp51 Nginx Backend for phorge171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:30:30] RECOVERY - cp51 Nginx Backend for swiftproxy171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8207 [01:30:39] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:31:03] PROBLEM - cp51 NTP time on cp51 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [01:31:06] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 26% [01:31:20] RECOVERY - cp51 Nginx Backend for swiftproxy161 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8206 [01:31:25] RECOVERY - cp51 ferm_active on cp51 is OK: OK ferm input default policy is set [01:32:16] RECOVERY - cp51 Current Load on cp51 is OK: LOAD OK - total load average: 0.24, 0.18, 0.12 [01:32:35] RECOVERY - cp51 Nginx Backend for mon181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8201 [01:33:18] RECOVERY - cp51 conntrack_table_size on cp51 is OK: OK: nf_conntrack is 0 % full [01:33:22] RECOVERY - cp51 Nginx Backend for mw172 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8118 [01:33:39] RECOVERY - cp51 Nginx Backend for mw152 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8114 [01:33:43] PROBLEM - cp51 NTP time on cp51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:58] RECOVERY - cp51 SSH on cp51 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [01:35:11] PROBLEM - cp51 Nginx Backend for mw181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:35:25] RECOVERY - cp51 Nginx Backend for phorge171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8202 [01:38:43] PROBLEM - cp51 Nginx Backend for swiftproxy171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:38:44] PROBLEM - cp51 Nginx Backend for test151 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [01:38:47] PROBLEM - cp51 Nginx Backend for mw162 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:39:31] PROBLEM - cp51 Nginx Backend for mw152 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [01:40:49] RECOVERY - cp51 Nginx Backend for test151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8181 [01:41:48] RECOVERY - cp51 Nginx Backend for mw162 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8116 [01:41:51] RECOVERY - cp51 Nginx Backend for swiftproxy171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8207 [01:42:41] RECOVERY - cp51 Nginx Backend for mw181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8119 [01:44:24] PROBLEM - cp51 conntrack_table_size on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:45:37] PROBLEM - cp51 Nginx Backend for puppet181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:46:37] RECOVERY - cp51 conntrack_table_size on cp51 is OK: OK: nf_conntrack is 0 % full [01:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:47:39] PROBLEM - cp51 SSH on cp51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:17] PROBLEM - cp51 Nginx Backend for mw182 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:48:21] RECOVERY - cp51 Nginx Backend for mw152 on cp51 is OK: TCP OK - 0.001 second response time on localhost port 8114 [01:49:24] PROBLEM - cp51 NTP time on cp51 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [01:49:37] RECOVERY - cp51 SSH on cp51 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [01:50:21] RECOVERY - cp51 Nginx Backend for puppet181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8204 [01:50:59] RECOVERY - cp51 Nginx Backend for mw182 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8120 [01:53:31] PROBLEM - cp51 Nginx Backend for mwtask171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:53:38] PROBLEM - cp51 Nginx Backend for mwtask181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:54:10] PROBLEM - cp51 Nginx Backend for mw172 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:54:51] PROBLEM - cp51 Nginx Backend for mw161 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:55:08] PROBLEM - cp51 NTP time on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:55:13] PROBLEM - cp51 Nginx Backend for mw181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:56:16] PROBLEM - cp51 Nginx Backend for test151 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:57:05] RECOVERY - cp51 Nginx Backend for mw161 on cp51 is OK: TCP OK - 0.001 second response time on localhost port 8115 [01:58:00] PROBLEM - cp51 SSH on cp51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:28] RECOVERY - cp51 Nginx Backend for mwtask181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8160 [01:59:53] PROBLEM - cp51 Nginx Backend for mon181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:00:17] PROBLEM - cp51 ferm_active on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:00:43] PROBLEM - cp51 NTP time on cp51 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [02:00:51] PROBLEM - cp51 conntrack_table_size on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:01:12] RECOVERY - cp51 Nginx Backend for mw181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8119 [02:01:18] RECOVERY - cp51 Nginx Backend for mwtask171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8161 [02:02:17] PROBLEM - cp51 Nginx Backend for mw162 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:02:27] PROBLEM - cp51 Nginx Backend for mw171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:02:51] PROBLEM - cp51 Current Load on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:03:09] PROBLEM - cp51 NTP time on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [02:03:48] PROBLEM - cp51 Disk Space on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [02:04:32] RECOVERY - cp51 Nginx Backend for mw162 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8116 [02:04:35] RECOVERY - cp51 Nginx Backend for test151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8181 [02:04:42] RECOVERY - cp51 Nginx Backend for mw171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8117 [02:04:56] RECOVERY - cp51 Current Load on cp51 is OK: LOAD OK - total load average: 0.05, 0.07, 0.08 [02:04:58] RECOVERY - cp51 Nginx Backend for mon181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8201 [02:05:15] PROBLEM - cp51 Nginx Backend for puppet181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:05:41] RECOVERY - cp51 ferm_active on cp51 is OK: OK ferm input default policy is set [02:05:54] PROBLEM - cp51 Nginx Backend for mwtask181 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [02:06:03] RECOVERY - cp51 Disk Space on cp51 is OK: DISK OK - free space: / 18312MiB (19% inode=98%); [02:06:08] RECOVERY - cp51 conntrack_table_size on cp51 is OK: OK: nf_conntrack is 0 % full [02:07:33] RECOVERY - cp51 Nginx Backend for puppet181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8204 [02:08:38] RECOVERY - cp51 Nginx Backend for mw172 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8118 [02:10:22] RECOVERY - cp51 SSH on cp51 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [02:11:02] RECOVERY - cp51 Nginx Backend for mwtask181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8160 [02:11:18] PROBLEM - www.rosettacode.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rosettacode.org' expires in 15 day(s) (Mon 27 May 2024 02:08:24 AM GMT +0000). [02:12:21] PROBLEM - cp51 Nginx Backend for mw161 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:13:09] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=dcmultiversewiki --no-updates /home/universalomega/dcmultiversewiki_xml_5dc56b3d424835c081e4.xml (END - exit=0) [02:13:10] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=dcmultiversewiki (START) [02:13:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:13:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:18] PROBLEM - cp51 Nginx Backend for mon181 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [02:16:41] PROBLEM - cp51 SSH on cp51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:17:29] RECOVERY - cp51 Nginx Backend for mon181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8201 [02:18:24] RECOVERY - cp51 NTP time on cp51 is OK: NTP OK: Offset 0.07318198681 secs [02:18:40] RECOVERY - cp51 SSH on cp51 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [02:19:39] PROBLEM - cp51 Nginx Backend for mw172 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:19:42] PROBLEM - cp51 Nginx Backend for mw162 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:19:45] PROBLEM - cp51 Nginx Backend for swiftproxy171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:19:51] PROBLEM - cp51 Nginx Backend for mw171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:21:45] RECOVERY - cp51 Nginx Backend for mw172 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8118 [02:21:57] RECOVERY - cp51 Nginx Backend for mw162 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8116 [02:22:07] PROBLEM - cp51 Nginx Backend for mw151 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:22:26] RECOVERY - cp51 Nginx Backend for mw171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8117 [02:22:29] RECOVERY - cp51 Nginx Backend for swiftproxy171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8207 [02:23:03] RECOVERY - cp51 Nginx Backend for mw161 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8115 [02:23:54] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=dcmultiversewiki --search-recursively --skip-dupes /home/universalomega/dcmultiversewiki (END - exit=0) [02:23:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:24:10] PROBLEM - cp51 NTP time on cp51 is WARNING: NTP WARNING: Offset -0.1422498226 secs [02:24:12] RECOVERY - cp51 Nginx Backend for mw151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8113 [02:25:25] PROBLEM - cp51 conntrack_table_size on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:26:30] PROBLEM - cp51 NTP time on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [02:27:38] RECOVERY - cp51 conntrack_table_size on cp51 is OK: OK: nf_conntrack is 0 % full [02:28:28] PROBLEM - cp51 Nginx Backend for mw182 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [02:28:47] PROBLEM - cp51 NTP time on cp51 is WARNING: NTP WARNING: Offset -0.2348392308 secs [02:29:13] PROBLEM - cp51 SSH on cp51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:31:42] PROBLEM - cp51 NTP time on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:32:04] PROBLEM - rosettacode.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rosettacode.org' expires in 15 day(s) (Mon 27 May 2024 02:08:24 AM GMT +0000). [02:33:17] RECOVERY - cp51 SSH on cp51 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [02:34:20] RECOVERY - cp51 NTP time on cp51 is OK: NTP OK: Offset -0.005248069763 secs [02:36:30] RECOVERY - cp51 Nginx Backend for mw182 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8120 [02:39:54] PROBLEM - cp51 conntrack_table_size on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:40:34] PROBLEM - cp51 Disk Space on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:42:44] RECOVERY - cp51 Disk Space on cp51 is OK: DISK OK - free space: / 18311MiB (19% inode=98%); [02:42:56] PROBLEM - cp51 ferm_active on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:45:21] PROBLEM - cp51 Nginx Backend for mw152 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:45:24] RECOVERY - cp51 ferm_active on cp51 is OK: OK ferm input default policy is set [02:47:37] RECOVERY - cp51 Nginx Backend for mw152 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8114 [02:47:38] RECOVERY - cp51 conntrack_table_size on cp51 is OK: OK: nf_conntrack is 0 % full [02:49:03] PROBLEM - cp51 Nginx Backend for mwtask181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:50:08] PROBLEM - cp51 NTP time on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:50:30] PROBLEM - cp51 Nginx Backend for mw151 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:52:38] RECOVERY - cp51 NTP time on cp51 is OK: NTP OK: Offset -0.06067486604 secs [02:54:21] RECOVERY - cp51 Nginx Backend for mwtask181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8160 [02:55:27] PROBLEM - cp51 Nginx Backend for puppet181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:56:13] PROBLEM - cp51 Nginx Backend for mw171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:56:15] PROBLEM - cp51 SSH on cp51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:56:41] PROBLEM - cp51 Nginx Backend for test151 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [02:57:00] PROBLEM - cp51 Nginx Backend for mw161 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:59:05] PROBLEM - cp51 Nginx Backend for phorge171 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [02:59:07] RECOVERY - cp51 Nginx Backend for mw151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8113 [02:59:17] PROBLEM - cp51 Nginx Backend for mw181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:59:33] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:59:40] PROBLEM - cp51 Disk Space on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:59:40] PROBLEM - cp51 Current Load on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:00:15] RECOVERY - mon181 Backups Grafana on mon181 is OK: FILE_AGE OK: /var/log/grafana-backup.log is 14 seconds old and 0 bytes [03:00:34] PROBLEM - cp51 Nginx Backend for mw182 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:00:57] PROBLEM - cp51 conntrack_table_size on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:01:03] PROBLEM - cp51 ferm_active on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:01:06] PROBLEM - cp51 Nginx Backend for mon181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:02:14] RECOVERY - cp51 Current Load on cp51 is OK: LOAD OK - total load average: 0.19, 0.17, 0.12 [03:02:47] PROBLEM - cp51 Nginx Backend for mw172 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:03:02] RECOVERY - cp51 Disk Space on cp51 is OK: DISK OK - free space: / 18311MiB (19% inode=98%); [03:03:14] RECOVERY - cp51 Nginx Backend for mw171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8117 [03:03:42] RECOVERY - cp51 Nginx Backend for mon181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8201 [03:04:20] PROBLEM - cp51 Nginx Backend for swiftproxy161 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:04:26] RECOVERY - cp51 SSH on cp51 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [03:05:03] PROBLEM - cp51 Nginx Backend for mwtask181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:05:45] PROBLEM - cp51 NTP time on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:05:49] PROBLEM - cp51 Nginx Backend for mw151 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:06:30] PROBLEM - cp51 Nginx Backend for matomo151 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:06:44] PROBLEM - cp51 Nginx Backend for mw152 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:06:46] PROBLEM - cp51 Nginx Backend for mw162 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:08:36] PROBLEM - cp51 Nginx Backend for mwtask171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:09:04] PROBLEM - cp51 Nginx Backend for reports171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:09:06] PROBLEM - cp51 Current Load on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:09:53] PROBLEM - cp51 Disk Space on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:10:07] PROBLEM - cp51 Nginx Backend for mw171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:10:12] PROBLEM - cp51 Nginx Backend for swiftproxy171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:10:31] PROBLEM - cp51 Nginx Backend for mon181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:11:58] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 296.72 ms [03:13:42] RECOVERY - cp51 Nginx Backend for mwtask181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8160 [03:14:43] RECOVERY - cp51 Nginx Backend for matomo151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8203 [03:14:44] RECOVERY - cp51 Nginx Backend for mw161 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8115 [03:15:08] RECOVERY - cp51 Nginx Backend for mw182 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8120 [03:15:34] RECOVERY - cp51 Nginx Backend for swiftproxy171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8207 [03:16:03] RECOVERY - cp51 Nginx Backend for test151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8181 [03:16:03] RECOVERY - cp51 Nginx Backend for mw152 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8114 [03:16:14] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 1384.81 ms [03:16:26] RECOVERY - cp51 Nginx Backend for phorge171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8202 [03:17:18] RECOVERY - cp51 Nginx Backend for mw172 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8118 [03:18:22] RECOVERY - cp51 Nginx Backend for swiftproxy161 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8206 [03:18:34] RECOVERY - cp51 Nginx Backend for mw162 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8116 [03:19:01] RECOVERY - cp51 Nginx Backend for mw171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8117 [03:19:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:19:40] RECOVERY - cp51 Nginx Backend for reports171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8205 [03:19:56] RECOVERY - cp51 conntrack_table_size on cp51 is OK: OK: nf_conntrack is 0 % full [03:20:33] RECOVERY - cp51 Nginx Backend for mw181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8119 [03:21:54] RECOVERY - cp51 Nginx Backend for mon181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8201 [03:22:23] RECOVERY - cp51 Nginx Backend for mwtask171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8161 [03:22:25] PROBLEM - cp51 Nginx Backend for mw182 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:22:44] PROBLEM - cp51 Nginx Backend for swiftproxy171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:23:08] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 46.250.240.167/cpweb, 2407:3641:2161:9774::1/cpweb [03:23:59] PROBLEM - cp51 Nginx Backend for phorge171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:24:21] PROBLEM - cp51 Nginx Backend for mw172 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:24:26] RECOVERY - cp51 Disk Space on cp51 is OK: DISK OK - free space: / 18311MiB (19% inode=98%); [03:24:35] PROBLEM - cp51 Nginx Backend for swiftproxy161 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:24:52] PROBLEM - cp51 SSH on cp51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:24:55] RECOVERY - cp51 Nginx Backend for swiftproxy171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8207 [03:25:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:25:08] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 45% [03:26:00] PROBLEM - cp51 NTP time on cp51 is WARNING: NTP WARNING: Offset 0.2341057062 secs [03:26:48] RECOVERY - cp51 Nginx Backend for puppet181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8204 [03:26:51] RECOVERY - cp51 Nginx Backend for phorge171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8202 [03:27:13] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:27:20] RECOVERY - cp51 Nginx Backend for mw172 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8118 [03:27:31] RECOVERY - cp51 Nginx Backend for mw151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8113 [03:27:45] PROBLEM - cp51 Nginx Backend for test151 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:28:10] PROBLEM - cp51 Nginx Backend for mw152 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:28:17] PROBLEM - cp51 NTP time on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:28:19] PROBLEM - cp51 Nginx Backend for mon181 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:29:08] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2407:3641:2161:9774::1/cpweb [03:29:29] RECOVERY - cp51 ferm_active on cp51 is OK: OK ferm input default policy is set [03:29:34] RECOVERY - cp51 Nginx Backend for swiftproxy161 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8206 [03:30:35] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=dcmultiversewiki (END - exit=0) [03:30:36] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=dcmultiversewiki --active --update (END - exit=0) [03:30:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:30:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:31:15] PROBLEM - cp51 Nginx Backend for matomo151 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:31:27] PROBLEM - cp51 Nginx Backend for mwtask181 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:33:05] RECOVERY - cp51 SSH on cp51 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [03:33:32] RECOVERY - cp51 Nginx Backend for mw182 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8120 [03:33:42] PROBLEM - cp51 Nginx Backend for puppet181 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:33:43] PROBLEM - cp51 Nginx Backend for reports171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:33:46] RECOVERY - cp51 Nginx Backend for mw152 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8114 [03:33:48] PROBLEM - cp51 conntrack_table_size on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:33:49] RECOVERY - cp51 Nginx Backend for mwtask181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8160 [03:33:59] PROBLEM - cp51 Nginx Backend for mw161 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:34:16] RECOVERY - cp51 Nginx Backend for matomo151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8203 [03:34:27] PROBLEM - cp51 NTP time on cp51 is WARNING: NTP WARNING: Offset 0.1118182838 secs [03:34:30] RECOVERY - cp51 Nginx Backend for mon181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8201 [03:34:31] PROBLEM - cp51 Nginx Backend for mw171 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:35:39] PROBLEM - cp51 Nginx Backend for swiftproxy171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:35:40] RECOVERY - cp51 Current Load on cp51 is OK: LOAD OK - total load average: 0.08, 0.47, 4.49 [03:35:52] PROBLEM - cp51 Nginx Backend for mw162 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:36:02] PROBLEM - cp51 Nginx Backend for swiftproxy161 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:36:03] RECOVERY - cp51 Nginx Backend for test151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8181 [03:36:09] RECOVERY - cp51 Nginx Backend for reports171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8205 [03:36:21] RECOVERY - cp51 conntrack_table_size on cp51 is OK: OK: nf_conntrack is 0 % full [03:37:22] PROBLEM - cp51 NTP time on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:38:07] PROBLEM - cp51 Nginx Backend for mw181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:38:43] RECOVERY - cp51 Nginx Backend for swiftproxy161 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8206 [03:39:33] RECOVERY - cp51 Nginx Backend for mw171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8117 [03:40:00] PROBLEM - cp51 Nginx Backend for mw152 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:40:25] PROBLEM - cp51 Nginx Backend for mwtask181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:42:25] PROBLEM - cp51 ferm_active on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:42:46] PROBLEM - cp51 Nginx Backend for reports171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:42:50] RECOVERY - cp51 Nginx Backend for mwtask181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8160 [03:43:42] RECOVERY - cp51 Nginx Backend for mw162 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8116 [03:44:22] PROBLEM - cp51 Nginx Backend for mw182 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:44:48] RECOVERY - cp51 Nginx Backend for puppet181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8204 [03:44:50] PROBLEM - cp51 conntrack_table_size on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:45:58] PROBLEM - cp51 Nginx Backend for mw172 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:46:01] PROBLEM - cp51 Nginx Backend for mwtask171 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:46:07] PROBLEM - cp51 Nginx Backend for mw171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:47:34] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=dcmultiversewiki --procs=40 (START) [03:47:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:47:49] RECOVERY - cp51 conntrack_table_size on cp51 is OK: OK: nf_conntrack is 0 % full [03:48:57] PROBLEM - cp51 Nginx Backend for swiftproxy161 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:49:03] RECOVERY - cp51 Nginx Backend for mw172 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8118 [03:49:55] PROBLEM - cp51 SSH on cp51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:50:13] PROBLEM - cp51 Nginx Backend for matomo151 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:50:50] PROBLEM - cp51 Nginx Backend for mwtask181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:50:54] PROBLEM - cp51 Nginx Backend for puppet181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:50:56] PROBLEM - cp51 Nginx Backend for mw162 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:51:04] PROBLEM - cp51 Nginx Backend for mw151 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:51:49] PROBLEM - cp51 NTP time on cp51 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [03:51:59] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.74, 19.48, 14.64 [03:52:11] RECOVERY - cp51 Nginx Backend for mw152 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8114 [03:52:47] RECOVERY - cp51 Nginx Backend for matomo151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8203 [03:53:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:53:14] RECOVERY - cp51 Nginx Backend for mwtask181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8160 [03:53:32] RECOVERY - cp51 Nginx Backend for mw151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8113 [03:53:56] PROBLEM - cp51 NTP time on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:53:59] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.69, 20.30, 15.55 [03:54:10] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 21% [03:54:42] PROBLEM - cp51 Nginx Backend for test151 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:54:46] PROBLEM - cp51 Nginx Backend for mw172 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:55:26] PROBLEM - cp51 Disk Space on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:55:59] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 30.51, 24.09, 17.51 [03:56:01] PROBLEM - cp51 Nginx Backend for phorge171 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:56:04] RECOVERY - cp51 Nginx Backend for swiftproxy171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8207 [03:56:09] PROBLEM - cp51 Current Load on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:56:39] RECOVERY - cp51 Nginx Backend for reports171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8205 [03:56:50] RECOVERY - cp51 Nginx Backend for test151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8181 [03:57:08] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2407:3641:2161:9774::1/cpweb [03:58:52] PROBLEM - cp51 Nginx Backend for mw151 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [03:59:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:59:17] RECOVERY - cp51 Nginx Backend for puppet181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8204 [03:59:19] RECOVERY - cp51 Nginx Backend for phorge171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8202 [03:59:20] RECOVERY - cp51 Current Load on cp51 is OK: LOAD OK - total load average: 0.05, 0.05, 1.04 [03:59:25] PROBLEM - cp51 conntrack_table_size on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:59:42] RECOVERY - cp51 Nginx Backend for mw171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8117 [03:59:56] PROBLEM - cp51 NTP time on cp51 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [04:01:12] RECOVERY - cp51 Nginx Backend for mw181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8119 [04:01:59] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 12.13, 20.01, 18.42 [04:02:10] PROBLEM - cp51 NTP time on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [04:02:35] PROBLEM - cp51 Nginx Backend for mon181 on cp51 is CRITICAL: CHECK_NRPE: Error - Could not connect to 2407:3641:2161:9774::1: Connection reset by peer [04:02:39]