[00:01:17] RECOVERY - mw134 Disk Space on mw134 is OK: DISK OK - free space: / 3500MiB (14% inode=70%); [00:02:05] RECOVERY - mw132 Disk Space on mw132 is OK: DISK OK - free space: / 2788MiB (11% inode=69%); [00:03:02] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/af72014c14d9 [00:03:04] [02miraheze/dns] 07paladox 03af72014 - Switch ns1 ips [00:03:09] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.96, 7.16, 6.84 [00:03:13] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [00:03:15] [02dns] 07paladox opened pull request 03#478: Switch ns1 ips - 13https://github.com/miraheze/dns/pull/478 [00:03:26] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/compare/af72014c14d9...4507fa5316d9 [00:03:29] [02miraheze/dns] 07paladox 034507fa5 - Update miraheze.org [00:03:30] [02dns] 07paladox synchronize pull request 03#478: Switch ns1 ips - 13https://github.com/miraheze/dns/pull/478 [00:04:25] PROBLEM - cp26 Disk Space on cp26 is WARNING: DISK WARNING - free space: / 8480MiB (9% inode=98%); [00:06:22] RECOVERY - cp26 Disk Space on cp26 is OK: DISK OK - free space: / 11116MiB (11% inode=98%); [00:07:08] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.03, 7.58, 7.08 [00:09:08] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.22, 7.45, 7.09 [00:10:21] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/a548bcf7fd15...50b51884c8dc [00:10:23] [02miraheze/dns] 07paladox 0350b5188 - Add cp36/37 to dns [00:10:45] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/7c006a51ada8 [00:10:48] [02miraheze/dns] 07paladox 037c006a5 - Add bast161 ipv4 address [00:10:49] [02dns] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/dns [00:10:52] [02dns] 07paladox opened pull request 03#479: Add bast161 ipv4 address - 13https://github.com/miraheze/dns/pull/479 [00:13:32] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/50b51884c8dc...10efaa105c85 [00:13:34] [02miraheze/dns] 07paladox 0310efaa1 - Fix cp37 ip [00:15:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.14, 6.81, 6.58 [00:17:53] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.23, 6.85, 6.50 [00:19:08] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.16, 7.14, 7.02 [00:19:48] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.22, 6.65, 6.47 [00:21:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.64, 7.47, 7.17 [00:21:38] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.46, 6.79, 6.66 [00:25:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.40, 7.40, 7.16 [00:28:44] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [00:33:06] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.84, 7.21, 7.36 [00:35:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/8629a190bfa9 [00:35:01] [02miraheze/puppet] 07paladox 038629a19 - Install ns1/cp36/cp37 [00:35:02] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [00:35:07] [02puppet] 07paladox opened pull request 03#3712: Install ns1/cp36/cp37 - 13https://github.com/miraheze/puppet/pull/3712 [00:35:08] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [00:35:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/8629a190bfa9...8f05e61afd82 [00:35:25] [02miraheze/puppet] 07paladox 038f05e61 - Create cp37.yaml [00:35:27] [02puppet] 07paladox synchronize pull request 03#3712: Install ns1/cp36/cp37 - 13https://github.com/miraheze/puppet/pull/3712 [00:36:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8f05e61afd82...3ba8637461a0 [00:36:38] [02miraheze/puppet] 07paladox 033ba8637 - Update site.pp [00:36:39] [02puppet] 07paladox synchronize pull request 03#3712: Install ns1/cp36/cp37 - 13https://github.com/miraheze/puppet/pull/3712 [00:36:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3ba8637461a0...2ce8f4dbf5c3 [00:37:00] [02miraheze/puppet] 07paladox 032ce8f4d - Update site.pp [00:37:01] [02puppet] 07paladox synchronize pull request 03#3712: Install ns1/cp36/cp37 - 13https://github.com/miraheze/puppet/pull/3712 [00:37:24] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [00:37:26] [02puppet] 07paladox closed pull request 03#3712: Install ns1/cp36/cp37 - 13https://github.com/miraheze/puppet/pull/3712 [00:37:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/puppet/compare/b0a6bd953e95...0e9accf1597a [00:37:30] [02miraheze/puppet] 07paladox 030e9accf - Install ns1/cp36/cp37 (#3712) [00:37:31] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [00:39:06] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.82, 5.90, 6.74 [00:42:34] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:42:48] [02dns] 07paladox closed pull request 03#478: Switch ns1 ips - 13https://github.com/miraheze/dns/pull/478 [00:42:49] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/10efaa105c85...b4b6b0d654b7 [00:42:51] [02miraheze/dns] 07paladox 03b4b6b0d - Switch ns1 ips (#478) [00:42:54] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [00:42:56] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [00:44:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0e9accf1597a...d4e0859ce976 [00:45:00] [02miraheze/puppet] 07paladox 03d4e0859 - dns::recursor: switch ns1 ip [00:45:49] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.60, 6.80, 6.49 [00:46:43] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:46:44] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [00:46:59] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [00:47:16] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -0.02287778258 secs [00:47:30] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:47:50] PROBLEM - cp37 HTTPS on cp37 is WARNING: HTTP WARNING: HTTP/2 404 - 3454 bytes in 0.543 second response time [00:49:10] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.49, 7.06, 6.61 [00:51:36] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.19, 6.74, 6.60 [00:55:01] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.85, 6.57, 6.56 [00:56:47] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.64, 7.44, 6.97 [01:00:46] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.10, 7.80, 7.28 [01:00:54] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.39, 6.99, 6.74 [01:04:36] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp34.miraheze.org - CNAME OK [01:04:48] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.00, 6.54, 6.62 [01:06:20] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.86, 6.83, 6.54 [01:08:20] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.62, 6.76, 6.55 [01:10:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f6a38c698574...b2353722e951 [01:10:23] [02miraheze/mw-config] 07paladox 03b235372 - Add cp36/37 to wgCdnServers [01:11:21] miraheze/mw-config - paladox the build passed. [01:20:41] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.54, 5.94, 6.63 [01:23:16] !log [@test151] starting deploy of {'config': True} to test151 [01:23:17] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [01:23:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:32:07] !log [@mwtask181] starting deploy of {'config': True} to all [01:32:11] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.14, 6.74, 6.63 [01:32:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:32:18] !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw132.miraheze.org [01:32:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:35:41] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [01:38:03] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.52, 6.80, 6.71 [01:41:08] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.06, 6.74, 6.28 [01:43:08] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.47, 7.28, 6.53 [01:45:08] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.23, 7.31, 6.64 [01:45:10] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03no-unset [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/92bca21cad02 [01:45:13] [02miraheze/puppet] 07Universal-Omega 0392bca21 - Don't unset X-Miraheze-Debug [01:45:15] [02puppet] 07Universal-Omega created branch 03no-unset - 13https://github.com/miraheze/puppet [01:45:18] [02puppet] 07Universal-Omega opened pull request 03#3713: Don't unset X-Miraheze-Debug - 13https://github.com/miraheze/puppet/pull/3713 [01:47:08] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.55, 6.76, 6.53 [01:48:21] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.02, 7.04, 6.85 [01:48:36] [02puppet] 07paladox closed pull request 03#3713: Don't unset X-Miraheze-Debug - 13https://github.com/miraheze/puppet/pull/3713 [01:48:38] [02miraheze/puppet] 07paladox deleted branch 03no-unset [01:48:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d4e0859ce976...ee08c63c953e [01:48:42] [02miraheze/puppet] 07Universal-Omega 03ee08c63 - Don't unset X-Miraheze-Debug (#3713) [01:48:43] [02puppet] 07paladox deleted branch 03no-unset - 13https://github.com/miraheze/puppet [01:48:45] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.01, 6.46, 5.63 [01:49:43] PROBLEM - ns1 Auth DNS on ns1 is UNKNOWN: check_dns: Invalid hostname/address [01:50:18] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.43, 6.45, 6.66 [01:52:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.10, 7.24, 6.18 [01:53:10] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw162.wikitide.net port 443 after 3059 ms: Couldn't connect to server [01:53:38] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw161.wikitide.net port 443 after 390 ms: Couldn't connect to server [01:54:10] PROBLEM - Host bast161 is DOWN: CRITICAL - Host Unreachable (10.0.16.127) [01:54:16] PROBLEM - ping on mem161 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.131) [01:54:25] PROBLEM - mw162 ferm_active on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host [01:54:28] PROBLEM - Host mem161 is DOWN: CRITICAL - Host Unreachable (10.0.16.131) [01:54:28] PROBLEM - mw162 Disk Space on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host [01:54:29] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw161 mw162 [01:54:29] PROBLEM - swiftobject161 Current Load on swiftobject161 is CRITICAL: connect to address 10.0.16.134 port 5666: No route to hostconnect to host 10.0.16.134 port 5666: No route to host [01:54:29] PROBLEM - swiftobject161 Swift Object Service on swiftobject161 is CRITICAL: connect to address 10.0.16.134 and port 6000: No route to host [01:54:29] PROBLEM - db161 Puppet on db161 is CRITICAL: connect to address 10.0.16.128 port 5666: No route to hostconnect to host 10.0.16.128 port 5666: No route to host [01:54:29] PROBLEM - Host db161 is DOWN: CRITICAL - Host Unreachable (10.0.16.128) [01:54:30] PROBLEM - swiftproxy161 SSH on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 and port 22: No route to host [01:54:32] PROBLEM - mw161 Puppet on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host PROBLEM - mw161 Disk Space on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host [01:54:32] PROBLEM - os161 SSH on os161 is CRITICAL: connect to address 10.0.16.129 and port 22: No route to host [01:54:33] PROBLEM - swiftproxy161 Puppet on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 port 5666: No route to hostconnect to host 10.0.16.135 port 5666: No route to host [01:54:34] PROBLEM - mw162 php-fpm on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host [01:54:35] PROBLEM - mw162 Puppet on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host [01:54:35] PROBLEM - os161 PowerDNS Recursor on os161 is CRITICAL: connect to address 10.0.16.129 port 5666: No route to hostconnect to host 10.0.16.129 port 5666: No route to host [01:54:37] PROBLEM - swiftproxy161 NTP time on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 port 5666: No route to hostconnect to host 10.0.16.135 port 5666: No route to host [01:54:37] PROBLEM - graylog161 ferm_active on graylog161 is CRITICAL: connect to address 10.0.16.130 port 5666: No route to hostconnect to host 10.0.16.130 port 5666: No route to host [01:54:38] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw161 mw162 [01:54:38] PROBLEM - os161 Current Load on os161 is CRITICAL: connect to address 10.0.16.129 port 5666: No route to hostconnect to host 10.0.16.129 port 5666: No route to host [01:54:38] PROBLEM - os161 APT on os161 is CRITICAL: connect to address 10.0.16.129 port 5666: No route to hostconnect to host 10.0.16.129 port 5666: No route to host [01:54:40] PROBLEM - swiftproxy161 HTTP on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 and port 80: No route to hostHTTP CRITICAL - Unable to open TCP socket [01:54:40] PROBLEM - swiftproxy161 PowerDNS Recursor on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 port 5666: No route to hostconnect to host 10.0.16.135 port 5666: No route to host [01:54:40] PROBLEM - graylog161 Puppet on graylog161 is CRITICAL: connect to address 10.0.16.130 port 5666: No route to hostconnect to host 10.0.16.130 port 5666: No route to host [01:54:42] PROBLEM - mw162 Current Load on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host [01:54:43] PROBLEM - swiftobject161 ferm_active on swiftobject161 is CRITICAL: connect to address 10.0.16.134 port 5666: No route to hostconnect to host 10.0.16.134 port 5666: No route to host [01:54:57] PROBLEM - mw162 conntrack_table_size on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host [01:54:57] PROBLEM - swiftproxy161 HTTPS on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [01:54:57] PROBLEM - swiftproxy161 ferm_active on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 port 5666: No route to hostconnect to host 10.0.16.135 port 5666: No route to host [01:54:58] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 2 backends are down. mw161 mw162 [01:55:01] PROBLEM - swiftobject161 NTP time on swiftobject161 is CRITICAL: connect to address 10.0.16.134 port 5666: No route to hostconnect to host 10.0.16.134 port 5666: No route to host [01:55:01] PROBLEM - Host mw161 is DOWN: CRITICAL - Host Unreachable (10.0.16.132) [01:55:04] PROBLEM - os161 ferm_active on os161 is CRITICAL: connect to address 10.0.16.129 port 5666: No route to hostconnect to host 10.0.16.129 port 5666: No route to host [01:55:08] PROBLEM - mw162 NTP time on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host [01:55:12] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 2 backends are down. mw161 mw162 [01:55:15] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: connect to address 10.0.16.134 port 5666: No route to hostconnect to host 10.0.16.134 port 5666: No route to host [01:55:16] PROBLEM - swiftproxy161 Current Load on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 port 5666: No route to hostconnect to host 10.0.16.135 port 5666: No route to host [01:55:18] PROBLEM - swiftobject161 conntrack_table_size on swiftobject161 is CRITICAL: connect to address 10.0.16.134 port 5666: No route to hostconnect to host 10.0.16.134 port 5666: No route to host [01:55:21] PROBLEM - Host swiftobject161 is DOWN: CRITICAL - Host Unreachable (10.0.16.134) [01:55:23] PROBLEM - Host os161 is DOWN: CRITICAL - Host Unreachable (10.0.16.129) [01:55:25] PROBLEM - ping on mw162 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.133) [01:55:27] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw161 mw162 [01:55:29] PROBLEM - ping on swiftproxy161 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.135) [01:55:30] PROBLEM - mw162 APT on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host [01:55:30] PROBLEM - mw162 PowerDNS Recursor on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host [01:55:37] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw161 mw162 [01:55:38] PROBLEM - Host swiftproxy161 is DOWN: CRITICAL - Host Unreachable (10.0.16.135) [01:55:42] PROBLEM - mw162 SSH on mw162 is CRITICAL: connect to address 10.0.16.133 and port 22: No route to host [01:55:58] PROBLEM - Host mw162 is DOWN: CRITICAL - Host Unreachable (10.0.16.133) [01:56:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ee08c63c953e...18354f295f4c [01:56:27] [02miraheze/puppet] 07paladox 0318354f2 - dns: fix monitoring with a private ip [01:56:37] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.09, 6.62, 6.17 [01:58:46] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 25 backends are healthy [02:02:28] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.43, 6.97, 6.44 [02:02:35] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 2 backends are down. mw161 mw162 [02:03:30] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:06:24] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.33, 7.49, 6.75 [02:08:52] !log [agent@db121] Finished dumping db121 to db161 [02:09:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:12:04] PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 0.15 ms [02:12:17] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.41, 7.52, 7.09 [02:14:06] RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.20 ms [02:14:07] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.22, 6.83, 6.47 [02:14:52] RECOVERY - Host mw161 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [02:15:01] RECOVERY - Host swiftobject161 is UP: PING OK - Packet loss = 0%, RTA = 0.18 ms [02:15:04] RECOVERY - ping on swiftobject161 is OK: PING OK - Packet loss = 0%, RTA = 0.19 ms [02:15:09] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [02:15:12] RECOVERY - swiftobject161 conntrack_table_size on swiftobject161 is OK: OK: nf_conntrack is 2 % full [02:15:22] RECOVERY - Host swiftproxy161 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [02:15:29] PROBLEM - mw161 NTP time on mw161 is CRITICAL: NTP CRITICAL: Offset 0.9890497923 secs [02:15:30] PROBLEM - mw161 HTTPS on mw161 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.227 second response time [02:15:32] RECOVERY - Host os161 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms [02:15:36] RECOVERY - Host bast161 is UP: PING OK - Packet loss = 0%, RTA = 0.45 ms [02:15:37] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.636 second response time [02:15:44] PROBLEM - bast161 Puppet on bast161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:15:49] PROBLEM - bast161 NTP time on bast161 is CRITICAL: NTP CRITICAL: Offset 0.5599853098 secs [02:15:53] RECOVERY - os161 Current Load on os161 is OK: LOAD OK - total load average: 0.68, 0.29, 0.10 [02:15:54] RECOVERY - os161 SSH on os161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [02:15:57] RECOVERY - os161 APT on os161 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [02:15:59] RECOVERY - mw161 Disk Space on mw161 is OK: DISK OK - free space: / 39397MiB (73% inode=89%); [02:16:02] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.03, 6.96, 6.57 [02:16:04] RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:16:04] RECOVERY - swiftproxy161 SSH on swiftproxy161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [02:16:11] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 25 backends are healthy [02:16:11] RECOVERY - Host mem161 is UP: PING OK - Packet loss = 0%, RTA = 0.20 ms [02:16:12] RECOVERY - os161 PowerDNS Recursor on os161 is OK: DNS OK: 0.289 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [02:16:12] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 25 backends are healthy [02:16:13] RECOVERY - Host mw162 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [02:16:14] RECOVERY - swiftobject161 ferm_active on swiftobject161 is OK: OK ferm input default policy is set [02:16:14] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [02:16:14] RECOVERY - swiftobject161 Swift Object Service on swiftobject161 is OK: TCP OK - 0.000 second response time on 10.0.16.134 port 6000 [02:16:14] RECOVERY - swiftproxy161 PowerDNS Recursor on swiftproxy161 is OK: DNS OK: 0.039 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [02:16:21] RECOVERY - Host db161 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [02:16:22] RECOVERY - mw162 php-fpm on mw162 is OK: PROCS OK: 31 processes with command name 'php-fpm8.2' [02:16:24] RECOVERY - swiftproxy161 HTTPS on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.266 second response time [02:16:25] RECOVERY - mw162 Disk Space on mw162 is OK: DISK OK - free space: / 39396MiB (73% inode=89%); [02:16:25] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 25 backends are healthy [02:16:25] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [02:16:28] RECOVERY - mw161 APT on mw161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [02:16:29] RECOVERY - swiftproxy161 HTTP on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.484 second response time [02:16:29] RECOVERY - swiftproxy161 NTP time on swiftproxy161 is OK: NTP OK: Offset 0.0004851818085 secs [02:16:32] RECOVERY - os161 ferm_active on os161 is OK: OK ferm input default policy is set [02:16:36] RECOVERY - mw161 conntrack_table_size on mw161 is OK: OK: nf_conntrack is 0 % full [02:16:38] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 25 backends are healthy [02:16:43] RECOVERY - Host graylog161 is UP: PING OK - Packet loss = 0%, RTA = 1.27 ms [02:16:48] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 25 backends are healthy [02:16:49] RECOVERY - swiftobject161 Current Load on swiftobject161 is OK: LOAD OK - total load average: 0.37, 0.24, 0.10 [02:16:49] RECOVERY - ping on swiftproxy161 is OK: PING OK - Packet loss = 0%, RTA = 0.20 ms [02:16:49] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 0.16, 0.13, 0.05 [02:16:49] RECOVERY - ping on mw161 is OK: PING OK - Packet loss = 0%, RTA = 0.21 ms [02:16:49] RECOVERY - mw162 PowerDNS Recursor on mw162 is OK: DNS OK: 0.033 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [02:16:54] RECOVERY - mw162 conntrack_table_size on mw162 is OK: OK: nf_conntrack is 0 % full [02:16:54] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:16:54] RECOVERY - ping on mw162 is OK: PING OK - Packet loss = 0%, RTA = 0.21 ms [02:16:59] RECOVERY - swiftproxy161 Disk Space on swiftproxy161 is OK: DISK OK - free space: / 22773MiB (85% inode=94%); [02:16:59] RECOVERY - mw162 SSH on mw162 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [02:17:04] RECOVERY - mw162 APT on mw162 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [02:17:12] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:17:14] RECOVERY - swiftproxy161 ferm_active on swiftproxy161 is OK: OK ferm input default policy is set [02:17:15] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 25 backends are healthy [02:17:15] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:17:19] RECOVERY - swiftproxy161 Current Load on swiftproxy161 is OK: LOAD OK - total load average: 2.45, 1.09, 0.43 [02:17:20] RECOVERY - swiftobject161 NTP time on swiftobject161 is OK: NTP OK: Offset 0.003213405609 secs [02:17:24] PROBLEM - mw162 HTTPS on mw162 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.070 second response time [02:17:27] RECOVERY - mw161 NTP time on mw161 is OK: NTP OK: Offset 0.001777023077 secs [02:17:37] RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:17:39] RECOVERY - mw162 NTP time on mw162 is OK: NTP OK: Offset -0.006971776485 secs [02:17:47] RECOVERY - bast161 NTP time on bast161 is OK: NTP OK: Offset -0.002159416676 secs [02:17:49] RECOVERY - mw162 ferm_active on mw162 is OK: OK ferm input default policy is set [02:17:54] RECOVERY - ping on mem161 is OK: PING OK - Packet loss = 0%, RTA = 0.23 ms [02:17:54] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:17:58] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.33, 6.26, 5.04 [02:18:09] RECOVERY - graylog161 ferm_active on graylog161 is OK: OK ferm input default policy is set [02:18:24] RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:18:54] PROBLEM - graylog161 Current Load on graylog161 is WARNING: LOAD WARNING - total load average: 7.16, 3.90, 1.57 [02:19:12] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.108 second response time [02:19:14] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.568 second response time [02:19:58] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.36, 6.42, 5.27 [02:20:46] RECOVERY - graylog161 Current Load on graylog161 is OK: LOAD OK - total load average: 1.80, 2.96, 1.50 [02:20:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.87, 7.14, 6.36 [02:24:47] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.30, 6.74, 6.37 [02:25:33] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.027 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [02:25:37] PROBLEM - cp36 Nginx Backend for mw132 on cp36 is WARNING: Could not resolve hostname : Name or service not known [02:25:40] PROBLEM - cp36 Nginx Backend for mw131 on cp36 is WARNING: Could not resolve hostname : Name or service not known [02:25:44] PROBLEM - cp36 Nginx Backend for mon181 on cp36 is WARNING: Could not resolve hostname : Name or service not known [02:25:45] PROBLEM - cp36 ferm_active on cp36 is WARNING: Could not resolve hostname : Name or service not known [02:25:47] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Unable to lookup IP address for '': getaddrinfo returned -2 - Name or service not known [02:25:51] PROBLEM - cp36 Disk Space on cp36 is WARNING: Could not resolve hostname : Name or service not known [02:25:53] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/f3946727cd8d [02:25:53] PROBLEM - cp36 Puppet on cp36 is WARNING: Could not resolve hostname : Name or service not known [02:25:55] PROBLEM - cp36 Nginx Backend for swiftproxy171 on cp36 is WARNING: Could not resolve hostname : Name or service not known [02:25:55] [02miraheze/dns] 07paladox 03f394672 - Switch cp34/35 to cp36/37 [02:25:55] PROBLEM - cp36 PowerDNS Recursor on cp36 is WARNING: Could not resolve hostname : Name or service not known [02:25:56] PROBLEM - Host cp36 is DOWN: check_ping: Invalid hostname/address - Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [02:25:57] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [02:25:59] [02dns] 07paladox opened pull request 03#480: Switch cp34/35 to cp36/37 - 13https://github.com/miraheze/dns/pull/480 [02:26:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.43, 7.56, 7.15 [02:26:22] [02dns] 07paladox closed pull request 03#480: Switch cp34/35 to cp36/37 - 13https://github.com/miraheze/dns/pull/480 [02:26:25] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/b4b6b0d654b7...7a4a8305c187 [02:26:26] [02miraheze/dns] 07paladox 037a4a830 - Switch cp34/35 to cp36/37 (#480) [02:26:29] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [02:26:30] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [02:26:45] [02miraheze/ManageWiki] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/3cd78164a5fc...0465226d6525 [02:26:48] [02miraheze/ManageWiki] 07OAuthority 030465226 - update defaults with new split perms [02:27:04] [02miraheze/ManageWiki] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/0465226d6525...f93116072883 [02:27:06] [02miraheze/ManageWiki] 07OAuthority 03f931160 - no spaces pls [02:28:06] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.08, 6.80, 6.43 [02:28:51] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 10% [02:29:00] PROBLEM - www.thesimswiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.thesimswiki.com could not be found [02:29:05] PROBLEM - talumn.neptune.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for talumn.neptune.wiki could not be found [02:29:05] PROBLEM - wiki.lewisgill.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.lewisgill.com could not be found [02:29:10] PROBLEM - www.themagical.world - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.themagical.world could not be found [02:29:19] PROBLEM - lebork.info.pl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lebork.info.pl could not be found [02:29:19] PROBLEM - www.cgradegames.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.cgradegames.net could not be found [02:29:21] PROBLEM - encyclopediarobotica.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for encyclopediarobotica.org could not be found [02:29:23] PROBLEM - emmytherobot.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for emmytherobot.wiki could not be found [02:29:32] PROBLEM - wiki.digitalcandela.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.digitalcandela.com could not be found [02:29:33] PROBLEM - www.istpcomputing.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.istpcomputing.com could not be found [02:29:34] PROBLEM - wiki.seamly.io - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.seamly.io could not be found [02:29:34] PROBLEM - wiki.anempireofdreams.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.anempireofdreams.com could not be found [02:29:36] PROBLEM - wiki.ciptamedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ciptamedia.org could not be found [02:29:36] PROBLEM - wiki.aetherexplorers.science - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.aetherexplorers.science could not be found [02:29:37] PROBLEM - fanonpedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fanonpedia.com could not be found [02:29:38] PROBLEM - project-patterns.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for project-patterns.com could not be found [02:29:38] PROBLEM - wiki.showtapeselection.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.showtapeselection.com could not be found [02:29:40] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.67, 6.45, 6.64 [02:29:40] PROBLEM - iceria.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for iceria.org could not be found [02:29:40] PROBLEM - wiki.cubestudios.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cubestudios.xyz could not be found [02:29:46] PROBLEM - wiki.minkyu.kim - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.minkyu.kim could not be found [02:29:49] PROBLEM - croc.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for croc.miraheze.org could not be found [02:29:51] PROBLEM - wiki.virtualjet.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.virtualjet.net could not be found [02:29:54] PROBLEM - wikislamica.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikislamica.org could not be found [02:29:55] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [02:30:10] PROBLEM - lostmediawiki.ru - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lostmediawiki.ru could not be found [02:30:11] PROBLEM - worldsanskrit.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for worldsanskrit.net could not be found [02:30:12] PROBLEM - mario.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mario.miraheze.org could not be found [02:30:24] PROBLEM - psycho.engineering - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for psycho.engineering could not be found [02:30:24] PROBLEM - encyclopedia.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for encyclopedia.eu.org could not be found [02:30:24] PROBLEM - taswin.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for taswin.miraheze.org could not be found [02:30:26] PROBLEM - donkeykong.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for donkeykong.miraheze.org could not be found [02:30:32] PROBLEM - wiki.kirbygang.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kirbygang.com could not be found [02:30:32] PROBLEM - portalsofphereon.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for portalsofphereon.com could not be found [02:30:37] PROBLEM - wiki.bluevertigo.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.bluevertigo.org could not be found [02:30:38] PROBLEM - hardcore.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hardcore.wiki could not be found [02:30:39] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fc.songcontests.eu could not be found [02:30:42] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for private.yahyabd.xyz could not be found [02:30:53] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.66, 6.84, 6.07 [02:30:54] PROBLEM - wiki.esufranchise.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.esufranchise.com could not be found [02:31:00] PROBLEM - www.widedata.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.widedata.org could not be found [02:31:01] PROBLEM - marinebiodiversitymatrix.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for marinebiodiversitymatrix.org could not be found [02:31:04] PROBLEM - wiki.albrusmusic.ru - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.albrusmusic.ru could not be found [02:31:08] PROBLEM - orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for orain.org could not be found [02:31:21] PROBLEM - wiki.auraxis.co - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.auraxis.co could not be found [02:31:22] PROBLEM - itora.bunnies1230.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for itora.bunnies1230.com could not be found [02:31:24] PROBLEM - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gesamtschule-nordkirchen.de could not be found [02:31:26] PROBLEM - crocwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for crocwiki.com could not be found [02:31:28] miraheze/ManageWiki - OAuthority the build passed. [02:31:37] PROBLEM - opendatascot.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for opendatascot.org could not be found [02:31:38] PROBLEM - rodzinka.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rodzinka.wiki could not be found [02:31:39] PROBLEM - yoshi.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for yoshi.miraheze.org could not be found [02:31:39] PROBLEM - pokemon.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pokemon.miraheze.org could not be found [02:31:45] PROBLEM - wiki.make717.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.make717.org could not be found [02:31:47] PROBLEM - en.wikiyri.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.wikiyri.org could not be found [02:31:47] PROBLEM - wizardia.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wizardia.wiki could not be found [02:31:52] PROBLEM - guia.cineastas.pt - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.cineastas.pt could not be found [02:31:53] PROBLEM - grayravens.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for grayravens.com could not be found [02:31:53] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found [02:31:55] PROBLEM - theenglishlakes.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for theenglishlakes.uk could not be found [02:31:55] PROBLEM - runzeppelin.ru - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for runzeppelin.ru could not be found [02:31:59] PROBLEM - echoes-wiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for echoes-wiki.com could not be found [02:32:00] PROBLEM - beamngmodding.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for beamngmodding.com could not be found [02:32:04] PROBLEM - hololive.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hololive.wiki could not be found [02:32:05] PROBLEM - wiki.counterculturelabs.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.counterculturelabs.org could not be found [02:32:11] PROBLEM - bebaskanpengetahuan.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bebaskanpengetahuan.id could not be found [02:32:15] PROBLEM - uncyclopediakr.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for uncyclopediakr.miraheze.org could not be found [02:32:17] PROBLEM - wiki.mcsoft.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mcsoft.org could not be found [02:32:22] PROBLEM - cyberlaw.ccdcoe.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cyberlaw.ccdcoe.org could not be found [02:32:26] PROBLEM - wiki.pixlies.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.pixlies.net could not be found [02:32:29] PROBLEM - wiki.danvs.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.danvs.net could not be found [02:32:31] PROBLEM - pokemonwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pokemonwiki.info could not be found [02:32:35] PROBLEM - www.ethene.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.ethene.wiki could not be found [02:32:35] PROBLEM - wiki.arsrobotics.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.arsrobotics.org could not be found [02:32:43] PROBLEM - wiki.meeusen.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.meeusen.net could not be found [02:32:46] PROBLEM - wiki.seredrau.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.seredrau.xyz could not be found [02:32:46] PROBLEM - wiki.col6.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.col6.de could not be found [02:32:49] PROBLEM - wiki.zergmafia.top - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zergmafia.top could not be found [02:32:51] PROBLEM - zomvilles.pixiemeta.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zomvilles.pixiemeta.com could not be found [02:32:53] PROBLEM - wiki.starship.digital - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.starship.digital could not be found [02:32:54] PROBLEM - trollpasta.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for trollpasta.com could not be found [02:32:54] PROBLEM - tno.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tno.wiki could not be found [02:32:56] PROBLEM - landofliberos.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for landofliberos.com could not be found [02:32:59] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/60d03d059996 [02:33:00] PROBLEM - wiki.autocountsoft.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.autocountsoft.com could not be found [02:33:00] [02miraheze/dns] 07paladox 0360d03d0 - Add cp25 to de map [02:33:01] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [02:33:02] [02dns] 07paladox opened pull request 03#481: Add cp25 to de map - 13https://github.com/miraheze/dns/pull/481 [02:33:02] PROBLEM - abeyancewiki.bumblebee-studios.se - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for abeyancewiki.bumblebee-studios.se could not be found [02:33:05] PROBLEM - wiki.macc.nyc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.macc.nyc could not be found [02:33:09] PROBLEM - guia.esporo.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.esporo.net could not be found [02:33:10] PROBLEM - o.kaoyu666.cn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for o.kaoyu666.cn could not be found [02:33:13] PROBLEM - hcw.tomat.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hcw.tomat.dev could not be found [02:33:22] PROBLEM - japanesefeaturephones.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for japanesefeaturephones.miraheze.org could not be found [02:33:33] PROBLEM - thestories.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for thestories.kr could not be found [02:33:40] PROBLEM - sonicpedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sonicpedia.org could not be found [02:33:44] PROBLEM - archives.nsgov.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for archives.nsgov.org could not be found [02:33:44] PROBLEM - looneypyramids.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for looneypyramids.wiki could not be found [02:33:45] PROBLEM - wiki.worlduniversityandschool.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worlduniversityandschool.org could not be found [02:33:46] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.08, 7.80, 7.51 [02:33:49] PROBLEM - www.glitchcity.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.glitchcity.wiki could not be found [02:33:52] PROBLEM - programming.red - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for programming.red could not be found [02:33:52] PROBLEM - www.trollpasta.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.trollpasta.com could not be found [02:33:52] PROBLEM - wiki.triplescripts.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.triplescripts.org could not be found [02:33:54] PROBLEM - ff8.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ff8.wiki could not be found [02:33:54] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.04, 7.37, 6.76 [02:33:55] PROBLEM - creativecommons.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for creativecommons.id could not be found [02:33:56] PROBLEM - squareenixwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for squareenixwiki.com could not be found [02:33:56] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mahdiruiz.line.pm could not be found [02:33:58] PROBLEM - pt.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pt.graalmilitary.com could not be found [02:34:04] PROBLEM - apocrypha.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for apocrypha.wiki could not be found [02:34:06] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.religiononfire.mar.in.ua could not be found [02:34:06] [02dns] 07paladox closed pull request 03#481: Add cp25 to de map - 13https://github.com/miraheze/dns/pull/481 [02:34:08] PROBLEM - steem.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for steem.wiki could not be found [02:34:08] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [02:34:10] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [02:34:13] PROBLEM - luxuryelevator.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for luxuryelevator.miraheze.org could not be found [02:34:13] PROBLEM - www.pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.pornwiki.org could not be found [02:34:15] PROBLEM - wiki.walkscape.app - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.walkscape.app could not be found [02:34:19] PROBLEM - dutki.kro.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dutki.kro.kr could not be found [02:34:38] PROBLEM - threedomwiki.pcast.site - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for threedomwiki.pcast.site could not be found [02:34:39] PROBLEM - nonciclopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nonciclopedia.org could not be found [02:34:39] PROBLEM - ethene.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ethene.wiki could not be found [02:34:41] PROBLEM - wiki.consid.vn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.consid.vn could not be found [02:34:41] PROBLEM - www.tintinpedia.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.tintinpedia.fr could not be found [02:34:41] PROBLEM - xomnipedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for xomnipedia.org could not be found [02:34:43] PROBLEM - wiki.sukotuze.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sukotuze.com could not be found [02:34:45] PROBLEM - wiki.vinesh.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.vinesh.eu.org could not be found [02:34:47] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.75, 6.79, 6.22 [02:34:48] PROBLEM - tssm.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tssm.wiki could not be found [02:34:57] PROBLEM - wiki.gtnewhorizons.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gtnewhorizons.com could not be found [02:35:01] PROBLEM - wiki.shaazzz.ir - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.shaazzz.ir could not be found [02:35:09] PROBLEM - wiki.cityrp.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cityrp.org could not be found [02:35:12] PROBLEM - sims.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sims.miraheze.org could not be found [02:35:12] PROBLEM - aman.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for aman.awiki.org could not be found [02:35:13] PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vedopedia.witches-empire.com could not be found [02:35:15] PROBLEM - roll-of-arms.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for roll-of-arms.com could not be found [02:35:17] PROBLEM - kunwok.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kunwok.org could not be found [02:35:21] PROBLEM - wiki.harlonserver.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.harlonserver.net could not be found [02:35:22] PROBLEM - wiki.gestaltonline.ca - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gestaltonline.ca could not be found [02:35:22] PROBLEM - wikiru.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikiru.wildterra2.com could not be found [02:35:23] PROBLEM - wiki.knowledgerevolution.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.knowledgerevolution.eu could not be found [02:35:26] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:35:28] RECOVERY - Host cp36 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms [02:35:29] PROBLEM - www.programming.red - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.programming.red could not be found [02:35:31] PROBLEM - segawiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for segawiki.com could not be found [02:35:32] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.96, 7.62, 6.93 [02:35:34] RECOVERY - cp36 Nginx Backend for mw132 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8107 [02:35:35] PROBLEM - speleo.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for speleo.wiki could not be found [02:35:40] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:35:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.45, 6.84, 6.61 [02:35:43] PROBLEM - wiki.geoparkcorumbatai.com.br - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.geoparkcorumbatai.com.br could not be found [02:35:46] PROBLEM - bushcraftwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bushcraftwiki.com could not be found [02:35:46] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 3 backends are down. mw141 mw142 mw143 [02:35:47] PROBLEM - chobots.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for chobots.wiki could not be found [02:35:48] PROBLEM - wiki.sofurry.games - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sofurry.games could not be found [02:35:48] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.96, 6.88, 6.77 [02:35:49] PROBLEM - cp36 HTTPS on cp36 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 6.037 second response time [02:35:50] PROBLEM - kombinat.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kombinat.eu.org could not be found [02:35:55] PROBLEM - wiki.tulpa.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.tulpa.info could not be found [02:36:00] PROBLEM - wiki.coolstation.space - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.coolstation.space could not be found [02:36:03] PROBLEM - wiki.kingmingshrimp.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kingmingshrimp.com could not be found [02:36:05] PROBLEM - themagical.world - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for themagical.world could not be found [02:36:06] PROBLEM - legacygt.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for legacygt.wiki could not be found [02:36:06] RECOVERY - cp36 PowerDNS Recursor on cp36 is OK: DNS OK: 0.048 seconds response time. miraheze.org returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [02:36:11] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.qadrishattari.xyz could not be found [02:36:16] PROBLEM - partopedia.ru - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for partopedia.ru could not be found [02:36:19] PROBLEM - inext.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for inext.miraheze.org could not be found [02:36:23] PROBLEM - www.allthetropes.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.allthetropes.org could not be found [02:36:24] PROBLEM - repository.archiopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for repository.archiopedia.org could not be found [02:36:28] PROBLEM - wiki.susqu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.susqu.org could not be found [02:36:30] PROBLEM - wiki.songngu.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.songngu.xyz could not be found [02:36:31] PROBLEM - removededmsongs.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for removededmsongs.miraheze.org could not be found [02:36:33] PROBLEM - ndgkb.nena.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ndgkb.nena.org could not be found [02:36:38] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 14.36, 12.68, 9.35 [02:36:40] PROBLEM - wiki.umbranet.mandelbrotgames.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.umbranet.mandelbrotgames.com could not be found [02:36:46] PROBLEM - glitchcity.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for glitchcity.wiki could not be found [02:36:47] PROBLEM - burnout.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for burnout.wiki could not be found [02:36:48] PROBLEM - wiki.wikimedia.cat - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wikimedia.cat could not be found [02:36:49] PROBLEM - kingdomdeath.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kingdomdeath.wiki could not be found [02:36:52] PROBLEM - www.bluepageswiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.bluepageswiki.org could not be found [02:37:05] PROBLEM - wiki.mikrodev.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mikrodev.com could not be found [02:37:05] PROBLEM - wiki.thunis.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thunis.eu could not be found [02:37:06] RECOVERY - cp36 Nginx Backend for swiftproxy171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8207 [02:37:14] PROBLEM - wiki.corgicam.tv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.corgicam.tv could not be found [02:37:16] PROBLEM - smashbroswiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for smashbroswiki.com could not be found [02:37:16] RECOVERY - cp36 ferm_active on cp36 is OK: OK ferm input default policy is set [02:37:16] RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:37:16] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.05, 7.62, 7.47 [02:37:19] PROBLEM - 321nails.crpteam.club - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for 321nails.crpteam.club could not be found [02:37:20] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.39, 9.80, 7.49 [02:37:21] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 23966MiB (27% inode=98%); [02:37:21] RECOVERY - cp36 Nginx Backend for mon181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8201 [02:37:21] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.825 second response time [02:37:23] PROBLEM - template.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for template.miraheze.org could not be found [02:37:26] RECOVERY - cp36 Nginx Backend for mw131 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8106 [02:37:28] PROBLEM - gil.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for gil.wiki could not be found [02:37:29] PROBLEM - icclopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for icclopedia.org could not be found [02:37:36] PROBLEM - ajhalili2006.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ajhalili2006.miraheze.org could not be found [02:37:36] PROBLEM - wiki.cdntennis.ca - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cdntennis.ca could not be found [02:37:39] PROBLEM - n64brew.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for n64brew.dev could not be found [02:37:40] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 25 backends are healthy [02:37:43] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.58, 6.73, 6.59 [02:37:47] PROBLEM - wikicompliance.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikicompliance.com could not be found [02:37:48] PROBLEM - www.mcpk.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mcpk.wiki could not be found [02:37:49] PROBLEM - wiki.site97.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.site97.fr could not be found [02:37:49] PROBLEM - wiki.evscope.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.evscope.org could not be found [02:37:49] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:37:51] PROBLEM - www.iceria.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.iceria.org could not be found [02:37:51] PROBLEM - wiki.hacklabsr.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.hacklabsr.org could not be found [02:37:54] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:38:05] PROBLEM - petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for petrawiki.org could not be found [02:38:10] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 3 backends are down. mw141 mw142 mw143 [02:38:12] PROBLEM - avas.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for avas.wiki could not be found [02:38:14] PROBLEM - smashbros.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for smashbros.miraheze.org could not be found [02:38:17] PROBLEM - wiki.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.miraheze.org could not be found [02:38:17] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:38:22] PROBLEM - wiki.artmechanicum.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.artmechanicum.com could not be found [02:38:23] PROBLEM - www.zenbuddhism.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.zenbuddhism.info could not be found [02:38:24] PROBLEM - hoseki.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hoseki.wiki could not be found [02:38:25] PROBLEM - yokaiwatch.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for yokaiwatch.miraheze.org could not be found [02:38:30] PROBLEM - gogigantic.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for gogigantic.wiki could not be found [02:38:32] PROBLEM - journeytheword.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for journeytheword.wiki could not be found [02:38:35] PROBLEM - worldtobuild.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for worldtobuild.wiki could not be found [02:38:35] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 26.29, 16.14, 10.41 [02:38:37] PROBLEM - otherkin.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for otherkin.wiki could not be found [02:38:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/4496b6da55ea [02:38:40] [02miraheze/puppet] 07paladox 034496b6d - opensearch: switch to os151-private [02:38:41] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [02:38:42] [02puppet] 07paladox opened pull request 03#3714: opensearch: switch to os151-private - 13https://github.com/miraheze/puppet/pull/3714 [02:38:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.94, 6.94, 6.42 [02:38:44] PROBLEM - enc.for.uz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for enc.for.uz could not be found [02:38:49] PROBLEM - wiki.tdrweb.top - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.tdrweb.top could not be found [02:38:51] PROBLEM - pwsc.polandballwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pwsc.polandballwiki.com could not be found [02:38:58] PROBLEM - permanentfuturelab.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for permanentfuturelab.wiki could not be found [02:39:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4496b6da55ea...f1b679237414 [02:39:06] PROBLEM - wiki.fatcatfablab.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fatcatfablab.org could not be found [02:39:08] [02miraheze/puppet] 07paladox 03f1b6792 - Update opensearch_rest.rb [02:39:11] [02puppet] 07paladox synchronize pull request 03#3714: opensearch: switch to os151-private - 13https://github.com/miraheze/puppet/pull/3714 [02:39:13] [02puppet] 07paladox closed pull request 03#3714: opensearch: switch to os151-private - 13https://github.com/miraheze/puppet/pull/3714 [02:39:13] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.77, 7.58, 7.47 [02:39:15] PROBLEM - metroidpedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for metroidpedia.com could not be found [02:39:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/18354f295f4c...e4535c2f0ae5 [02:39:17] [02miraheze/puppet] 07paladox 03e4535c2 - opensearch: switch to os151-private (#3714) [02:39:18] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [02:39:18] PROBLEM - wiki.theluxuryelevator.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.theluxuryelevator.com could not be found [02:39:19] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [02:39:19] PROBLEM - miraheze.gq - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.gq could not be found [02:39:20] PROBLEM - comprehensibleinputwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for comprehensibleinputwiki.org could not be found [02:39:22] PROBLEM - theagonian.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for theagonian.com could not be found [02:39:23] PROBLEM - wiki.grab-tools.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.grab-tools.live could not be found [02:39:24] PROBLEM - wiki.nowchess.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nowchess.org could not be found [02:39:31] PROBLEM - crustypedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for crustypedia.org could not be found [02:39:32] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.56, 6.61, 6.68 [02:39:36] PROBLEM - dragonquestwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dragonquestwiki.com could not be found [02:39:40] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:39:44] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3478 bytes in 1.204 second response time [02:39:48] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.495 second response time [02:39:49] PROBLEM - cp36 HTTPS on cp36 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 0.407 second response time [02:39:52] PROBLEM - wiki.newnormal.fans - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.newnormal.fans could not be found [02:39:52] PROBLEM - wikien.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikien.wildterra2.com could not be found [02:39:58] PROBLEM - solaswiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for solaswiki.org could not be found [02:40:04] PROBLEM - files.pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for files.pornwiki.org could not be found [02:40:13] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.543 second response time [02:40:16] PROBLEM - mabi.myomi.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mabi.myomi.net could not be found [02:40:17] PROBLEM - wiki.misslessfighters.club - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.misslessfighters.club could not be found [02:40:20] PROBLEM - www.clinitheque.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.clinitheque.fr could not be found [02:40:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e4535c2f0ae5...78d7e3027cbe [02:40:21] PROBLEM - yoshipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for yoshipedia.com could not be found [02:40:21] PROBLEM - hr.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hr.petrawiki.org could not be found [02:40:22] [02miraheze/puppet] 07paladox 0378d7e30 - Update common.yaml [02:40:22] PROBLEM - www.puzzles.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.puzzles.wiki could not be found [02:40:23] PROBLEM - mystiverse.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mystiverse.wiki could not be found [02:40:35] PROBLEM - isv.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for isv.orain.org could not be found [02:40:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/78d7e3027cbe...0c0cc3e1b59e [02:40:40] [02miraheze/puppet] 07paladox 030c0cc3e - Update opensearch_rest.rb [02:40:44] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zymonic.com could not be found [02:40:50] PROBLEM - eng.archiopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for eng.archiopedia.org could not be found [02:40:50] PROBLEM - tintinpedia.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tintinpedia.fr could not be found [02:40:52] PROBLEM - null-cpu.emudev.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for null-cpu.emudev.org could not be found [02:40:58] PROBLEM - alternatewiki.tombricks.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for alternatewiki.tombricks.com could not be found [02:41:00] PROBLEM - bluearchive.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bluearchive.wiki could not be found [02:41:03] PROBLEM - wiki.jsal.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jsal.org could not be found [02:41:04] PROBLEM - www.satepedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.satepedia.com could not be found [02:41:09] PROBLEM - wiki.sequoiafabrica.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sequoiafabrica.org could not be found [02:41:10] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.55, 8.01, 7.65 [02:41:13] RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [02:41:13] PROBLEM - wiki.potabi.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.potabi.com could not be found [02:41:18] PROBLEM - corru.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for corru.wiki could not be found [02:41:18] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 8.66, 11.62, 8.97 [02:41:19] PROBLEM - wiki.themarkside.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.themarkside.com could not be found [02:41:19] PROBLEM - wiki.yumeka.icu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yumeka.icu could not be found [02:41:20] PROBLEM - squareenix.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for squareenix.miraheze.org could not be found [02:41:22] PROBLEM - www.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.miraheze.org could not be found [02:41:22] PROBLEM - wiki.manos.pt - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.manos.pt could not be found [02:41:27] PROBLEM - metroid.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for metroid.miraheze.org could not be found [02:41:28] PROBLEM - ncuwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ncuwiki.com could not be found [02:41:33] PROBLEM - dlfm-wiki.top - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dlfm-wiki.top could not be found [02:41:36] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.51, 6.78, 6.79 [02:41:37] PROBLEM - wiki.gamergeeked.us - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gamergeeked.us could not be found [02:41:38] PROBLEM - coffeewiki.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for coffeewiki.net could not be found [02:41:38] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for uk.religiononfire.mar.in.ua could not be found [02:41:38] PROBLEM - istpcomputing.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for istpcomputing.com could not be found [02:41:39] PROBLEM - patternarchive.online - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for patternarchive.online could not be found [02:41:40] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.598 second response time [02:41:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 7.13, 6.79 [02:41:45] PROBLEM - archive.a2b2.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for archive.a2b2.org could not be found [02:41:48] PROBLEM - tech.exputra.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tech.exputra.com could not be found [02:41:49] PROBLEM - wiki.metaiitgn.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.metaiitgn.org could not be found [02:41:49] PROBLEM - wiki.hspdev.0t0.jp - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.hspdev.0t0.jp could not be found [02:41:51] PROBLEM - wiki.oasismp.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.oasismp.com could not be found [02:41:54] PROBLEM - www.johanloopmans.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.johanloopmans.nl could not be found [02:41:56] PROBLEM - wiki.yuanpi.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yuanpi.eu.org could not be found [02:41:56] PROBLEM - wiki.iurevar.world - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.iurevar.world could not be found [02:41:58] PROBLEM - crashspyrowiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for crashspyrowiki.com could not be found [02:42:03] PROBLEM - staratlaswiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for staratlaswiki.org could not be found [02:42:03] PROBLEM - www.rothwell-leeds.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.rothwell-leeds.co.uk could not be found [02:42:04] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 25 backends are healthy [02:42:05] PROBLEM - rockrevolution.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rockrevolution.wiki could not be found [02:42:06] PROBLEM - grc.repository.archiopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for grc.repository.archiopedia.org could not be found [02:42:11] PROBLEM - sona.pona.la - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sona.pona.la could not be found [02:42:11] PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.alathramc.com could not be found [02:42:11] PROBLEM - www.pyramidgames.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.pyramidgames.wiki could not be found [02:42:17] PROBLEM - acgn.sfdev.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for acgn.sfdev.eu.org could not be found [02:42:25] PROBLEM - www.burnout.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.burnout.wiki could not be found [02:42:27] PROBLEM - fashiondreamer.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fashiondreamer.wiki could not be found [02:42:30] PROBLEM - johanloopmans.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for johanloopmans.nl could not be found [02:42:32] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:42:33] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 5.99, 11.11, 10.07 [02:42:33] PROBLEM - wiki.simorgh.me - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.simorgh.me could not be found [02:42:35] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 6.58, 10.39, 9.34 [02:42:42] PROBLEM - kalons-reverie.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kalons-reverie.com could not be found [02:42:46] PROBLEM - wiki.wildport.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wildport.com could not be found [02:42:48] PROBLEM - wiki.omegabuild.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.omegabuild.uk could not be found [02:42:49] PROBLEM - wiki.pixandbit.gay - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.pixandbit.gay could not be found [02:42:50] PROBLEM - ecole.science - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ecole.science could not be found [02:42:51] PROBLEM - tep.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tep.wiki could not be found [02:42:56] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [02:43:07] PROBLEM - wiki.recaptime.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.recaptime.eu.org could not be found [02:43:08] PROBLEM - tintinpedia.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tintinpedia.miraheze.org could not be found [02:43:16] PROBLEM - miraheze.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.com could not be found [02:43:32] PROBLEM - wmworld.sktz.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wmworld.sktz.live could not be found [02:43:38] PROBLEM - closinglogosgroup.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for closinglogosgroup.miraheze.org could not be found [02:43:50] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 4.45, 8.67, 8.26 [02:43:55] PROBLEM - smafallfest.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for smafallfest.org could not be found [02:44:00] PROBLEM - wiki.pulsus.cc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.pulsus.cc could not be found [02:44:03] PROBLEM - evilgeniuswiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for evilgeniuswiki.com could not be found [02:44:04] PROBLEM - wiki.valkyrienskies.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.valkyrienskies.org could not be found [02:44:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.21, 6.85 [02:44:11] PROBLEM - wiki.teessidehackspace.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.teessidehackspace.org.uk could not be found [02:44:11] PROBLEM - wiki.overwood.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.overwood.xyz could not be found [02:44:13] PROBLEM - ru.countryhumans.polandball.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ru.countryhumans.polandball.wiki could not be found [02:44:14] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds [02:44:14] PROBLEM - de.berlinwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for de.berlinwiki.org could not be found [02:44:16] PROBLEM - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.bikinibottomdiscord.org could not be found [02:44:21] PROBLEM - www.gengbaike.top - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.gengbaike.top could not be found [02:44:22] PROBLEM - lostidols.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lostidols.wiki could not be found [02:44:27] PROBLEM - wiki.ricochetuniverse.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ricochetuniverse.com could not be found [02:44:30] PROBLEM - puritwiki.p-e.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for puritwiki.p-e.kr could not be found [02:44:31] PROBLEM - wiki.digitaldesignhq.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.digitaldesignhq.com could not be found [02:44:36] PROBLEM - forecasting.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for forecasting.wiki could not be found [02:44:48] PROBLEM - commonswiki.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for commonswiki.miraheze.org could not be found [02:44:54] PROBLEM - privacywiki.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for privacywiki.xyz could not be found [02:44:58] PROBLEM - wiki.rosestulipsandliberty.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.rosestulipsandliberty.com could not be found [02:45:05] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 4.47, 8.70, 9.33 [02:45:06] PROBLEM - www.hebammenwiki.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.hebammenwiki.de could not be found [02:45:07] PROBLEM - wiki.funkey-project.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.funkey-project.com could not be found [02:45:08] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 5.81, 8.48, 8.76 [02:45:16] PROBLEM - wiki.luemir.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.luemir.xyz could not be found [02:45:18] PROBLEM - adadevelopersacademy.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for adadevelopersacademy.wiki could not be found [02:45:19] PROBLEM - www.project-patterns.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.project-patterns.com could not be found [02:45:21] PROBLEM - wiki.lefrenchmelee.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.lefrenchmelee.fr could not be found [02:45:21] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.74, 7.70, 6.91 [02:45:25] PROBLEM - beidipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for beidipedia.com could not be found [02:45:33] PROBLEM - data.nonbinary.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for data.nonbinary.wiki could not be found [02:45:35] PROBLEM - witchingworld.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for witchingworld.org could not be found [02:45:36] PROBLEM - thelangyalist.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for thelangyalist.miraheze.org could not be found [02:45:43] PROBLEM - equestripedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for equestripedia.org could not be found [02:45:46] PROBLEM - cloud.neptune.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cloud.neptune.wiki could not be found [02:45:52] PROBLEM - dkpedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dkpedia.com could not be found [02:45:57] PROBLEM - wiki.joust.ro - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.joust.ro could not be found [02:46:08] PROBLEM - wiki.grottocenter.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.grottocenter.org could not be found [02:46:09] PROBLEM - www.baharna.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.baharna.org could not be found [02:46:09] PROBLEM - dariawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dariawiki.org could not be found [02:46:15] PROBLEM - cities.simulz.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cities.simulz.kr could not be found [02:46:18] PROBLEM - www.turtletown.ca - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.turtletown.ca could not be found [02:46:23] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/7a4a8305c187...ee025beb7d7e [02:46:24] PROBLEM - sarovia.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sarovia.graalmilitary.com could not be found [02:46:25] [02miraheze/dns] 07paladox 03ee025be - Add acme challenge to train.org [02:46:25] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.79, 7.37, 7.54 [02:46:28] PROBLEM - ping6 on cp34 is CRITICAL: PING CRITICAL - Packet loss = 100% [02:46:32] PROBLEM - podpedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for podpedia.org could not be found [02:46:33] PROBLEM - wiki.candelabrem.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.candelabrem.com could not be found [02:46:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.97, 7.11, 6.86 [02:46:37] [02dns] 07paladox commented on commit 03ee025beb7d7ecbea333c08127e5b64671ed6cd86 - 13https://github.com/miraheze/dns/commit/ee025beb7d7ecbea333c08127e5b64671ed6cd86#commitcomment-137893945 [02:46:40] PROBLEM - cp34 SSH on cp34 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:46:41] PROBLEM - fanpedia.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fanpedia.miraheze.org could not be found [02:46:44] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.57, 7.00, 6.91 [02:46:47] PROBLEM - cp35 Nginx Backend for mwtask181 on cp35 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:46:47] PROBLEM - 9b9t.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for 9b9t.wiki could not be found [02:46:50] PROBLEM - wiki.drycleaninglobby.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.drycleaninglobby.net could not be found [02:46:51] PROBLEM - cp35 PowerDNS Recursor on cp35 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:46:56] PROBLEM - wikimoma.art - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikimoma.art could not be found [02:46:58] PROBLEM - encyclopediaofastrobiology.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for encyclopediaofastrobiology.org could not be found [02:47:02] PROBLEM - wiki.moores.tech - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.moores.tech could not be found [02:47:06] PROBLEM - wiki.3point0.science - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.3point0.science could not be found [02:47:09] PROBLEM - wiki.aoemods.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.aoemods.com could not be found [02:47:10] PROBLEM - wiki.buryland.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.buryland.net could not be found [02:47:11] PROBLEM - Host cp35 is DOWN: PING CRITICAL - Packet loss = 100% [02:47:12] PROBLEM - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.tmyt105.leyhp.com could not be found [02:47:16] PROBLEM - es.countryhumans.polandball.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for es.countryhumans.polandball.wiki could not be found [02:47:18] PROBLEM - www.mh142.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mh142.com could not be found [02:47:19] PROBLEM - wiki.wilderyogi.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wilderyogi.eu could not be found [02:47:21] PROBLEM - Host cp34 is DOWN: PING CRITICAL - Packet loss = 100% [02:47:24] PROBLEM - nijigen.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nijigen.wiki could not be found [02:47:24] PROBLEM - wiki.projectdiablo2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.projectdiablo2.com could not be found [02:47:29] PROBLEM - lab.wikimimo.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lab.wikimimo.com could not be found [02:47:30] PROBLEM - poserdazfreebies.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for poserdazfreebies.orain.org could not be found [02:47:42] PROBLEM - www.lab612.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.lab612.at could not be found [02:47:46] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.41, 7.51, 6.95 [02:47:49] PROBLEM - antiguabarbudacalypso.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for antiguabarbudacalypso.com could not be found [02:47:56] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.altilunium.my.id could not be found [02:47:58] PROBLEM - psychevos.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for psychevos.org could not be found [02:47:58] PROBLEM - wiki.openhatch.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.openhatch.org could not be found [02:47:59] PROBLEM - wiki.warframestat.us - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.warframestat.us could not be found [02:48:00] PROBLEM - test2.specialchat.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for test2.specialchat.org could not be found [02:48:04] PROBLEM - miraheze.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.tk could not be found [02:48:06] PROBLEM - ru-teirailway.f5.si - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ru-teirailway.f5.si could not be found [02:48:08] PROBLEM - publictestwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for publictestwiki.com could not be found [02:48:09] PROBLEM - you.r-fit.cc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for you.r-fit.cc could not be found [02:48:10] PROBLEM - bn.wikiyri.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bn.wikiyri.org could not be found [02:48:19] PROBLEM - xn--80ahcnmgnyj.xn--p1ai - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for xn--80ahcnmgnyj.xn--p1ai could not be found [02:48:19] PROBLEM - vise.dayid.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vise.dayid.org could not be found [02:48:21] PROBLEM - wikipwaadia.works - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikipwaadia.works could not be found [02:48:29] PROBLEM - airlineinsider.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for airlineinsider.org could not be found [02:48:34] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.41, 7.27, 6.96 [02:48:37] PROBLEM - wiki.villagecollaborative.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.villagecollaborative.net could not be found [02:48:41] PROBLEM - persist.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for persist.wiki could not be found [02:48:42] PROBLEM - mirabeta.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mirabeta.org could not be found [02:48:46] PROBLEM - en.clockup.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.clockup.wiki could not be found [02:48:50] PROBLEM - www.permanentfuturelab.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.permanentfuturelab.wiki could not be found [02:48:52] PROBLEM - wiki.beergeeks.co.il - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.beergeeks.co.il could not be found [02:48:54] PROBLEM - dragonquest2.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dragonquest2.miraheze.org could not be found [02:48:55] PROBLEM - dragdown.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dragdown.wiki could not be found [02:48:56] PROBLEM - wiki.k2cho.me - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.k2cho.me could not be found [02:48:57] PROBLEM - wiki.leapit.no - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.leapit.no could not be found [02:48:59] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yahyabd.xyz could not be found [02:49:01] PROBLEM - wiki.aridia.space - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.aridia.space could not be found [02:49:02] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.10, 6.29, 6.66 [02:49:06] PROBLEM - wiki.whentheycry.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.whentheycry.org could not be found [02:49:07] PROBLEM - rarewarewiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rarewarewiki.com could not be found [02:49:07] PROBLEM - www.hoolehistoryheritagesociety.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.hoolehistoryheritagesociety.org.uk could not be found [02:49:08] PROBLEM - www.sekaipedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.sekaipedia.org could not be found [02:49:08] PROBLEM - wiki.mastodon.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mastodon.kr could not be found [02:49:08] PROBLEM - civwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for civwiki.org could not be found [02:49:12] PROBLEM - wiki.cyberfurs.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cyberfurs.org could not be found [02:49:12] PROBLEM - heavyironmodding.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for heavyironmodding.org could not be found [02:49:13] PROBLEM - swests.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for swests.org could not be found [02:49:16] PROBLEM - www.publictestwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.publictestwiki.com could not be found [02:49:17] PROBLEM - reviews.thejoshmeister.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for reviews.thejoshmeister.com could not be found [02:49:20] PROBLEM - wiki.elysiamc.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.elysiamc.xyz could not be found [02:49:27] PROBLEM - wiki.ssij.se - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ssij.se could not be found [02:49:29] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/ee025beb7d7e...760951085db4 [02:49:31] [02miraheze/dns] 07paladox 037609510 - Fix [02:49:47] PROBLEM - electowiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for electowiki.org could not be found [02:49:49] PROBLEM - grk.archiopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for grk.archiopedia.org could not be found [02:49:50] PROBLEM - wikimas.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikimas.kr could not be found [02:49:55] PROBLEM - small.polandballwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for small.polandballwiki.com could not be found [02:50:00] PROBLEM - wiki.strangereons.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.strangereons.com could not be found [02:50:01] PROBLEM - christipedia.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for christipedia.nl could not be found [02:50:02] PROBLEM - history.estill.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for history.estill.org could not be found [02:50:05] PROBLEM - wiki.rsf.world - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.rsf.world could not be found [02:50:05] PROBLEM - correypedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for correypedia.org could not be found [02:50:06] PROBLEM - slymods.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for slymods.info could not be found [02:50:07] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.02, 7.53, 7.02 [02:50:09] PROBLEM - pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pornwiki.org could not be found [02:50:10] PROBLEM - wiki.mostly.vet - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mostly.vet could not be found [02:50:11] PROBLEM - wiki.starshipsailing.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.starshipsailing.net could not be found [02:50:14] PROBLEM - projectsekai.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for projectsekai.miraheze.org could not be found [02:50:14] PROBLEM - www.christipedia.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.christipedia.nl could not be found [02:50:15] PROBLEM - www.fossilfighterswiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.fossilfighterswiki.com could not be found [02:50:15] PROBLEM - wiki.ff6worldscollide.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ff6worldscollide.com could not be found [02:50:15] PROBLEM - wiki.lesunk.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.lesunk.com could not be found [02:50:17] PROBLEM - wiki.barengreza.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.barengreza.my.id could not be found [02:50:20] PROBLEM - pastport.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pastport.org could not be found [02:50:20] PROBLEM - pwiki.arkcls.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pwiki.arkcls.com could not be found [02:50:21] PROBLEM - mariopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mariopedia.org could not be found [02:50:22] PROBLEM - wiki.nevillepedia.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nevillepedia.eu could not be found [02:50:23] PROBLEM - recaptime.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for recaptime.miraheze.org could not be found [02:50:25] PROBLEM - rare.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rare.miraheze.org could not be found [02:50:30] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c42b812d5809...0c669351dd90 [02:50:31] [02miraheze/ssl] 07MirahezeSSLBot 030c66935 - Bot: Update SSL cert for orain.org [02:50:31] PROBLEM - wiki.zamnhacking.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zamnhacking.net could not be found [02:50:37] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.andreijiroh.uk.eu.org could not be found [02:50:40] PROBLEM - miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.wiki could not be found [02:50:40] PROBLEM - ao90.pinho.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ao90.pinho.org could not be found [02:50:41] PROBLEM - fallofsanctuary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fallofsanctuary.com could not be found [02:50:45] PROBLEM - mockgovernments.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mockgovernments.com could not be found [02:50:45] PROBLEM - wiki.jagtek.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jagtek.org could not be found [02:50:46] PROBLEM - evil.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for evil.wiki could not be found [02:50:46] PROBLEM - wiki.creaturathegame.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.creaturathegame.com could not be found [02:50:47] PROBLEM - pwiki.drydraytonvillagehall.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pwiki.drydraytonvillagehall.org.uk could not be found [02:50:49] PROBLEM - cheeseepedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cheeseepedia.org could not be found [02:51:02] PROBLEM - elephants.ggreig.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for elephants.ggreig.com could not be found [02:51:03] PROBLEM - wiki.apico.buzz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.apico.buzz could not be found [02:51:03] PROBLEM - wiki.milkywayidle.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.milkywayidle.com could not be found [02:51:04] PROBLEM - internalwiki.coolstation.space - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for internalwiki.coolstation.space could not be found [02:51:06] PROBLEM - www.lostidols.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.lostidols.wiki could not be found [02:51:07] PROBLEM - pyramidgames.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pyramidgames.wiki could not be found [02:51:07] PROBLEM - allthetropes.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for allthetropes.org could not be found [02:51:07] PROBLEM - allthetropes.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for allthetropes.miraheze.org could not be found [02:51:08] PROBLEM - wiki.otir.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.otir.nl could not be found [02:51:09] PROBLEM - translate.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for translate.petrawiki.org could not be found [02:51:10] PROBLEM - wiki.thesimswiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thesimswiki.com could not be found [02:51:12] PROBLEM - vitriol.spoon.army - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vitriol.spoon.army could not be found [02:51:15] PROBLEM - wiki.aclevo.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.aclevo.com could not be found [02:51:15] PROBLEM - dodgecircle.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dodgecircle.com could not be found [02:51:17] PROBLEM - tl.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tl.awiki.org could not be found [02:51:17] PROBLEM - steamdecklinux.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for steamdecklinux.wiki could not be found [02:51:18] PROBLEM - 321nailswiki.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for 321nailswiki.miraheze.org could not be found [02:51:22] PROBLEM - vmklegacy.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vmklegacy.wiki could not be found [02:51:26] PROBLEM - starveil.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for starveil.wiki could not be found [02:51:31] PROBLEM - wiki.jeremyboyd.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jeremyboyd.org could not be found [02:51:37] PROBLEM - schizoidnightmares.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for schizoidnightmares.wiki could not be found [02:51:42] PROBLEM - pandorastale.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pandorastale.miraheze.org could not be found [02:51:47] PROBLEM - archive.stellurgists.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for archive.stellurgists.wiki could not be found [02:51:51] PROBLEM - bristolstudenthousingcoop.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bristolstudenthousingcoop.org could not be found [02:51:54] PROBLEM - wiki.mobilityengineer.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mobilityengineer.com could not be found [02:52:00] PROBLEM - rct.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rct.wiki could not be found [02:52:02] PROBLEM - kb.nena.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kb.nena.org could not be found [02:52:03] PROBLEM - arrieretoile.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for arrieretoile.fr could not be found [02:52:06] PROBLEM - thestarsareright.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for thestarsareright.org could not be found [02:52:08] PROBLEM - puzzles.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for puzzles.wiki could not be found [02:52:09] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.73, 7.23, 6.97 [02:52:11] PROBLEM - kaylaandrachel.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kaylaandrachel.miraheze.org could not be found [02:52:13] PROBLEM - wiki.controlistes.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.controlistes.fr could not be found [02:52:15] PROBLEM - freedomplanetwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for freedomplanetwiki.com could not be found [02:52:16] PROBLEM - kirisame-kissaten.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kirisame-kissaten.cf could not be found [02:52:17] PROBLEM - wiki.climatechange.ai - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.climatechange.ai could not be found [02:52:23] PROBLEM - www.portalsofphereon.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.portalsofphereon.com could not be found [02:52:24] PROBLEM - wikappedia.cc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikappedia.cc could not be found [02:52:25] PROBLEM - wiki.kachow.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kachow.org could not be found [02:52:28] PROBLEM - worldtriggerwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for worldtriggerwiki.com could not be found [02:52:29] PROBLEM - touhouplace.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for touhouplace.wiki could not be found [02:52:29] PROBLEM - pandorastale.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pandorastale.wiki could not be found [02:52:30] PROBLEM - pubwiki.lab.co.pl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pubwiki.lab.co.pl could not be found [02:52:31] PROBLEM - progressbar95.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for progressbar95.wiki could not be found [02:52:32] PROBLEM - wiki.theatlas.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.theatlas.live could not be found [02:52:36] PROBLEM - www.winenjoy.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.winenjoy.net could not be found [02:52:40] PROBLEM - infectowiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for infectowiki.com could not be found [02:52:48] PROBLEM - wiki.infomedia.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.infomedia.co.uk could not be found [02:52:51] PROBLEM - baharna.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for baharna.org could not be found [02:52:52] PROBLEM - resources.africanvision.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for resources.africanvision.org could not be found [02:52:52] PROBLEM - knifepointhorror.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for knifepointhorror.wiki could not be found [02:52:52] PROBLEM - celeste.ink - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for celeste.ink could not be found [02:52:53] PROBLEM - www.moiswiki.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.moiswiki.net could not be found [02:52:53] PROBLEM - apeirology.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for apeirology.com could not be found [02:52:54] PROBLEM - elsterclopedie.marithaime.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for elsterclopedie.marithaime.nl could not be found [02:52:55] PROBLEM - grk.repository.archiopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for grk.repository.archiopedia.org could not be found [02:52:57] PROBLEM - wiki.mxlinuxusers.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mxlinuxusers.de could not be found [02:52:57] PROBLEM - yokaiwatchwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for yokaiwatchwiki.com could not be found [02:52:57] PROBLEM - wiki.meregos.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.meregos.com could not be found [02:52:58] PROBLEM - monarchists.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for monarchists.wiki could not be found [02:52:59] PROBLEM - miraheze.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.cf could not be found [02:53:02] PROBLEM - inourownwords.online - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for inourownwords.online could not be found [02:53:05] PROBLEM - en.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.petrawiki.org could not be found [02:53:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.21, 6.89, 6.82 [02:53:08] PROBLEM - wiki.yunachannel.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yunachannel.com could not be found [02:53:10] PROBLEM - wiki.orvyn.ca - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.orvyn.ca could not be found [02:53:13] PROBLEM - wiki.fbpml.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fbpml.org could not be found [02:53:20] PROBLEM - sega.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sega.miraheze.org could not be found [02:53:23] PROBLEM - osfirsttimer.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for osfirsttimer.wiki could not be found [02:53:24] PROBLEM - data.wikiyri.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for data.wikiyri.org could not be found [02:53:39] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mineland.eu could not be found [02:53:51] PROBLEM - buildabearwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for buildabearwiki.info could not be found [02:53:51] PROBLEM - wiki.msnld.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.msnld.com could not be found [02:53:52] PROBLEM - fid.koymi.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fid.koymi.net could not be found [02:53:53] PROBLEM - kinnikupedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kinnikupedia.org could not be found [02:53:54] PROBLEM - stablestate.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for stablestate.org could not be found [02:53:57] PROBLEM - hastursnotebook.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hastursnotebook.org could not be found [02:54:00] PROBLEM - www.pastport.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.pastport.org could not be found [02:54:07] PROBLEM - wiki.mc.marochuru.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mc.marochuru.net could not be found [02:54:09] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for with.cpt-ra.bid could not be found [02:54:09] PROBLEM - wiki.ivy.cm - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ivy.cm could not be found [02:54:28] PROBLEM - index.pdcommunity.ir - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for index.pdcommunity.ir could not be found [02:54:32] PROBLEM - wiki.veloren.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.veloren.net could not be found [02:54:38] PROBLEM - wiki.astralprojections.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.astralprojections.org could not be found [02:54:38] PROBLEM - 100acg.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for 100acg.wiki could not be found [02:54:44] PROBLEM - wiki.denby.tech - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.denby.tech could not be found [02:54:45] PROBLEM - wiki.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.graalmilitary.com could not be found [02:54:46] PROBLEM - wiki.limaru.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.limaru.net could not be found [02:54:46] PROBLEM - ie.unigon.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ie.unigon.net could not be found [02:54:47] PROBLEM - webkinzguide.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for webkinzguide.com could not be found [02:54:54] PROBLEM - wiki.ooer.ooo - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ooer.ooo could not be found [02:54:56] PROBLEM - wiki.cutefame.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cutefame.net could not be found [02:54:59] PROBLEM - royal-wiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for royal-wiki.org could not be found [02:55:00] PROBLEM - www.journeytheword.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.journeytheword.wiki could not be found [02:55:00] PROBLEM - piwik.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for piwik.miraheze.org could not be found [02:55:04] PROBLEM - wiki.mhe.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mhe.my.id could not be found [02:55:07] PROBLEM - allthetropes.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for allthetropes.orain.org could not be found [02:55:08] PROBLEM - minescape.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for minescape.wiki could not be found [02:55:09] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.73, 7.41, 7.01 [02:55:10] PROBLEM - wiki.globasa.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.globasa.net could not be found [02:55:13] PROBLEM - www.dovearchives.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.dovearchives.wiki could not be found [02:55:14] PROBLEM - nl.xliving.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nl.xliving.org could not be found [02:55:19] PROBLEM - www.samafia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.samafia.org could not be found [02:55:21] PROBLEM - kodiak.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kodiak.wiki could not be found [02:55:24] PROBLEM - dft.wikiyri.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dft.wikiyri.org could not be found [02:55:33] PROBLEM - wiki.closai.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.closai.com could not be found [02:55:34] PROBLEM - wiki.solarconquest.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.solarconquest.net could not be found [02:55:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0c0cc3e1b59e...1897fe578b36 [02:55:34] PROBLEM - wonderfuleveryday.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wonderfuleveryday.org could not be found [02:55:36] PROBLEM - wiki.taes.com.br - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.taes.com.br could not be found [02:55:37] [02miraheze/puppet] 07paladox 031897fe5 - Fix regex for wikitide [02:55:53] PROBLEM - storytime.jdstroy.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for storytime.jdstroy.cf could not be found [02:55:58] PROBLEM - wiki.s23.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.s23.org could not be found [02:56:14] PROBLEM - wiki.eversd.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.eversd.com could not be found [02:56:20] PROBLEM - www.documentopublico.com.br - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.documentopublico.com.br could not be found [02:56:20] PROBLEM - wiki.blockate.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.blockate.com could not be found [02:56:21] PROBLEM - wiki.queenscourt.games - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.queenscourt.games could not be found [02:56:21] PROBLEM - mhdh.pj568.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mhdh.pj568.eu.org could not be found [02:56:22] PROBLEM - thelonsdalebattalion.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for thelonsdalebattalion.co.uk could not be found [02:56:22] PROBLEM - wiki.elgeis.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.elgeis.com could not be found [02:56:27] PROBLEM - wiki.mcjones.gay - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mcjones.gay could not be found [02:56:28] PROBLEM - removededm.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for removededm.com could not be found [02:56:28] PROBLEM - fanon.polandballwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fanon.polandballwiki.com could not be found [02:56:29] PROBLEM - wiki.insideearth.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.insideearth.info could not be found [02:56:32] PROBLEM - wiki.widwwa.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.widwwa.co.uk could not be found [02:56:32] PROBLEM - familiacorsi.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for familiacorsi.com could not be found [02:56:34] PROBLEM - wiki.dwarvesrp.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.dwarvesrp.com could not be found [02:56:36] PROBLEM - wikiyri.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikiyri.org could not be found [02:56:37] PROBLEM - wikigenius.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikigenius.org could not be found [02:56:48] PROBLEM - wiki.fediverse.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fediverse.kr could not be found [02:56:59] PROBLEM - rothwell-leeds.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rothwell-leeds.co.uk could not be found [02:56:59] PROBLEM - zhacg.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zhacg.wiki could not be found [02:57:01] PROBLEM - feathercare.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for feathercare.net could not be found [02:57:02] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.14, 7.71, 7.43 [02:57:05] PROBLEM - en.pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.pornwiki.org could not be found [02:57:08] PROBLEM - www.dariawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.dariawiki.org could not be found [02:57:10] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.12, 7.42, 7.14 [02:57:11] PROBLEM - iol.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for iol.wiki could not be found [02:57:11] PROBLEM - richterian.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for richterian.com could not be found [02:57:30] PROBLEM - wiki.sheepservermc.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sheepservermc.net could not be found [02:57:38] PROBLEM - shiroko.us - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for shiroko.us could not be found [02:57:43] PROBLEM - schizoidnightmares.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for schizoidnightmares.miraheze.org could not be found [02:57:43] PROBLEM - quadball.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for quadball.wiki could not be found [02:57:48] PROBLEM - www.pomologia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.pomologia.org could not be found [02:57:49] PROBLEM - polcompball.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for polcompball.miraheze.org could not be found [02:57:50] PROBLEM - wiki.zahira.me - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zahira.me could not be found [02:57:54] PROBLEM - dkwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dkwiki.org could not be found [02:57:54] PROBLEM - en.omniversalis.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.omniversalis.org could not be found [02:57:56] PROBLEM - wiki.puucraft.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.puucraft.net could not be found [02:58:06] PROBLEM - wiki.worldbuilding.network - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worldbuilding.network could not be found [02:58:17] PROBLEM - wiki.scvo.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.scvo.org could not be found [02:58:18] PROBLEM - thepolicyhub.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for thepolicyhub.org.uk could not be found [02:58:23] PROBLEM - cnt.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cnt.miraheze.org could not be found [02:58:24] PROBLEM - crashspyro.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for crashspyro.miraheze.org could not be found [02:58:49] PROBLEM - wiki.esnmilanostatale.it - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.esnmilanostatale.it could not be found [02:58:49] PROBLEM - files.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for files.petrawiki.org could not be found [02:59:07] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.89, 7.45, 7.14 [02:59:12] PROBLEM - largedu.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for largedu.eu.org could not be found [02:59:12] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.59, 7.05, 7.23 [02:59:13] PROBLEM - reviwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for reviwiki.info could not be found [02:59:15] PROBLEM - mh142.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mh142.com could not be found [02:59:20] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.56, 7.36, 7.15 [02:59:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1897fe578b36...bc04b1b91bca [02:59:36] [02miraheze/puppet] 07paladox 03bc04b1b - Update commands.conf [02:59:39] PROBLEM - history.sdtef.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for history.sdtef.org could not be found [03:00:11] PROBLEM - robloxapi.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for robloxapi.wiki could not be found [03:02:05] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.23, 6.09, 6.61 [03:03:41] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.12, 6.15, 6.76 [03:06:14] PROBLEM - revi.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for revi.wiki could not be found [03:07:25] PROBLEM - replication-ops.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for replication-ops.com could not be found [03:07:41] RECOVERY - orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 26 Apr 2024 01:50:24 AM GMT +0000. [03:08:51] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b2353722e951...236103371611 [03:08:54] [02miraheze/mw-config] 07paladox 032361033 - Remove cp34/35 [03:08:56] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:09:07] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/236103371611...76f12b5081d4 [03:09:08] [02miraheze/mw-config] 07paladox 0376f12b5 - Remove cp24/25 [03:10:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.90, 6.64, 6.76 [03:10:08] miraheze/mw-config - paladox the build passed. [03:11:32] RECOVERY - isv.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 26 Apr 2024 01:50:24 AM GMT +0000. [03:11:34] PROBLEM - polcompball.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for polcompball.wiki could not be found [03:12:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.12, 7.65, 7.12 [03:12:40] PROBLEM - housing.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for housing.wiki could not be found [03:13:46] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.69, 7.55, 7.22 [03:14:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 6.98, 6.74 [03:15:10] RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 26 Apr 2024 01:50:24 AM GMT +0000. [03:15:46] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.57, 7.63, 7.29 [03:15:48] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.42, 7.38, 7.08 [03:16:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.38, 7.45, 6.94 [03:17:44] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.18, 7.72, 7.36 [03:17:47] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.64, 7.43, 7.13 [03:18:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.09, 7.37, 6.98 [03:19:30] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 26 Apr 2024 01:50:24 AM GMT +0000. [03:19:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.36, 7.55, 7.34 [03:19:56] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [03:20:11] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:20:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.28, 7.72, 7.15 [03:20:33] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [03:20:52] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [03:21:04] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:21:06] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [03:21:22] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3368 bytes in 0.373 second response time [03:21:27] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3392 bytes in 0.559 second response time [03:21:35] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3362 bytes in 0.006 second response time [03:21:52] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3478 bytes in 1.342 second response time [03:22:01] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.55, 7.75, 7.64 [03:22:06] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.626 second response time [03:22:10] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:22:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.68, 7.27, 7.06 [03:22:47] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 3 backends are down. mw141 mw134 mw143 [03:23:12] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 6 backends are down. mw132 mw141 mw142 mw133 mw134 mw143 [03:23:22] !log [@test151] starting deploy of {'config': True} to test151 [03:23:23] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [03:23:26] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.88, 6.35, 6.70 [03:23:33] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [03:23:48] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3361 bytes in 0.524 second response time [03:24:02] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:24:16] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 2 backends are down. mw142 mw134 [03:24:17] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.37, 6.23, 6.70 [03:24:28] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 2 backends are down. mw142 mw134 [03:24:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:24:45] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 6.062 second response time [03:24:58] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 6.615 second response time [03:25:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:25:16] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 4.799 second response time [03:25:18] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.005 second response time [03:25:25] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.618 second response time [03:25:28] PROBLEM - cp37 HTTPS on cp37 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 0.453 second response time [03:25:37] PROBLEM - cp36 HTTPS on cp36 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 0.473 second response time [03:25:48] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3477 bytes in 1.295 second response time [03:25:58] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.482 second response time [03:26:01] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 3.68, 5.23, 6.61 [03:26:05] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.572 second response time [03:26:14] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 25 backends are healthy [03:26:28] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 25 backends are healthy [03:26:40] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 25 backends are healthy [03:27:13] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 25 backends are healthy [03:27:32] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 25 backends are healthy [03:29:26] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.03, 6.66, 6.70 [03:31:25] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.31, 6.51, 6.63 [03:31:35] !log [@mwtask181] starting deploy of {'config': True} to all [03:31:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:32:24] !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw151.wikitide.net [03:32:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:35:18] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [03:39:42] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.67, 6.66, 6.80 [03:43:30] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.68, 7.67, 7.30 [03:43:42] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.85, 6.88, 6.87 [03:45:28] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.03, 7.44, 7.26 [03:45:57] !log [paladox@mwtask181] starting deploy of {'config': True, 'force': True} to all [03:46:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:46:31] !log [paladox@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 33s [03:46:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:03:13] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [04:15:16] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.08, 7.51, 7.21 [04:17:16] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.33, 7.40, 7.20 [04:17:29] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.31, 6.63, 6.79 [04:21:27] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.69, 6.79, 6.83 [04:29:27] PROBLEM - cp26 Disk Space on cp26 is WARNING: DISK WARNING - free space: / 10242MiB (10% inode=98%); [04:31:26] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.59, 6.70, 6.79 [04:35:24] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.75, 6.94, 6.89 [04:43:23] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.88, 6.50, 6.73 [04:46:03] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [04:47:23] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.08, 6.80, 6.79 [04:53:22] PROBLEM - mw132 Disk Space on mw132 is WARNING: DISK WARNING - free space: / 2520MiB (10% inode=69%); [04:54:04] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.79, 6.81, 6.63 [04:55:22] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.79, 6.62, 6.77 [04:56:05] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.69, 6.76, 6.64 [05:02:04] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.95, 6.83, 6.69 [05:05:18] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.21, 6.39, 6.77 [05:10:02] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.34, 6.68, 6.71 [05:14:37] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.andreijiroh.uk.eu.org could not be found [05:16:08] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.91, 7.02, 6.92 [05:21:01] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.39, 6.86, 6.72 [05:23:01] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.52, 6.74, 6.70 [05:24:12] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:24:41] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:25:09] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [05:25:36] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:25:55] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 4 backends are down. mw132 mw141 mw142 mw143 [05:26:12] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.967 second response time [05:26:16] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 3 backends are down. mw132 mw141 mw142 [05:26:40] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:26:40] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:26:50] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 4 backends are down. mw132 mw142 mw133 mw134 [05:26:56] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3392 bytes in 1.186 second response time [05:27:01] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.56, 6.84, 6.76 [05:27:10] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 5 backends are down. mw131 mw132 mw133 mw134 mw143 [05:27:22] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 3 backends are down. mw131 mw133 mw143 [05:27:32] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.935 second response time [05:28:00] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.16, 6.56, 6.75 [05:28:42] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:28:57] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.360 second response time [05:29:15] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3392 bytes in 0.477 second response time [05:29:46] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.373 second response time [05:30:20] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:31:02] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.33, 6.67, 6.72 [05:31:06] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:31:22] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 7.128 second response time [05:31:38] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:32:44] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 9.988 second response time [05:33:06] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.551 second response time [05:33:12] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [05:33:20] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.581 second response time [05:34:50] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:35:10] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [05:35:40] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.484 second response time [05:35:59] PROBLEM - mw133 MediaWiki Rendering on mw133 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.501 second response time [05:36:19] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:37:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.98, 7.18, 6.94 [05:37:15] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 9.354 second response time [05:37:21] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 9.662 second response time [05:37:25] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:37:59] RECOVERY - mw133 MediaWiki Rendering on mw133 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.034 second response time [05:38:12] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.594 second response time [05:38:40] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:39:20] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:39:44] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:40:21] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [05:41:06] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.21, 6.64, 6.77 [05:41:16] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.150 second response time [05:41:17] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.224 second response time [05:41:31] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [05:42:10] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:42:19] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.657 second response time [05:43:07] PROBLEM - cp37 HTTPS on cp37 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 9.656 second response time [05:43:15] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 7.039 second response time [05:43:24] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [05:44:25] !log [agent@db131] Finished dumping db131 to db171 [05:45:08] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.308 second response time [05:45:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:45:13] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.488 second response time [05:45:19] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.631 second response time [05:45:20] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.235 second response time [05:45:32] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.700 second response time [05:45:34] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.506 second response time [05:45:41] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.490 second response time [05:45:44] PROBLEM - cp36 HTTPS on cp36 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 0.428 second response time [05:45:48] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.37, 7.07, 6.83 [05:45:52] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 25 backends are healthy [05:46:12] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.930 second response time [05:46:25] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 25 backends are healthy [05:46:52] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.487 second response time [05:46:53] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 25 backends are healthy [05:46:57] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.934 second response time [05:47:10] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 25 backends are healthy [05:47:21] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 25 backends are healthy [05:51:47] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.76, 6.71, 6.78 [05:53:04] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.00, 6.80, 6.69 [05:57:02] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.98, 6.56, 6.64 [06:02:38] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.66, 7.04, 6.82 [06:05:05] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.39, 6.91, 6.58 [06:07:00] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.79, 6.87, 6.73 [06:09:05] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.10, 6.63, 6.56 [06:10:49] PROBLEM - db131 APT on db131 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [06:11:05] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.54, 6.78, 6.73 [06:12:21] PROBLEM - mw134 MediaWiki Rendering on mw134 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.654 second response time [06:12:23] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.708 second response time [06:12:30] PROBLEM - mw143 MediaWiki Rendering on mw143 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.661 second response time [06:12:43] PROBLEM - mw133 MediaWiki Rendering on mw133 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.707 second response time [06:12:46] PROBLEM - db131 APT on db131 is CRITICAL: APT CRITICAL: 55 packages available for upgrade (25 critical updates). [06:12:59] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.259 second response time [06:13:07] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 1.041 second response time [06:13:15] PROBLEM - db131 MariaDB Connections on db131 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db131.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections. [06:13:15] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db131.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [06:13:24] PROBLEM - db131 MariaDB on db131 is CRITICAL: Can't connect to server on 'db131.miraheze.org' (115) [06:13:34] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.668 second response time [06:13:44] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.872 second response time [06:17:06] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.02, 6.80, 6.61 [06:19:07] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.21, 6.56, 6.54 [06:19:16] RECOVERY - db131 APT on db131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:24:22] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.02, 6.60, 6.78 [06:24:46] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [06:26:42] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 37% [06:30:35] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 45% [06:34:26] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 25% [06:42:06] PROBLEM - db142 MariaDB on db142 is CRITICAL: Can't connect to server on 'db142.miraheze.org' (115) [06:43:21] PROBLEM - db142 MariaDB Connections on db142 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db142.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections. [06:43:21] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db142.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [06:46:09] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.85, 6.83, 6.68 [06:46:50] PROBLEM - db101 MariaDB on db101 is CRITICAL: Can't connect to server on 'db101.miraheze.org' (115) [06:48:04] PROBLEM - db101 MariaDB Connections on db101 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db101.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections. [06:48:04] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db101.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [06:48:07] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.30, 6.60, 6.61 [06:48:16] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to server on 'db121.miraheze.org' (115) [06:48:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/76f12b5081d4...b928bb0cd26e [06:48:34] [02miraheze/mw-config] 07Universal-Omega 03b928bb0 - Switch mwtask181 to new infra [06:48:43] PROBLEM - db121 MariaDB Connections on db121 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db121.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections. [06:48:44] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db121.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [06:49:25] miraheze/mw-config - Universal-Omega the build passed. [06:53:41] !log [@test151] starting deploy of {'config': True} to test151 [06:53:42] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [07:02:07] !log [@mwtask181] starting deploy of {'config': True} to all [07:02:08] !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [07:05:22] RECOVERY - mw132 Disk Space on mw132 is OK: DISK OK - free space: / 2657MiB (11% inode=69%); [07:05:23] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [07:07:23] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 41% [07:09:18] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 31% [07:10:54] [02mw-config] 07Universal-Omega closed pull request 03#5447: Use new hostnames in DB - 13https://github.com/miraheze/mw-config/pull/5447 [07:10:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/b928bb0cd26e...6d2e8ad6a44d [07:10:58] [02miraheze/mw-config] 07Universal-Omega 036d2e8ad - Use new hostnames in DB (#5447) [07:11:01] [02miraheze/mw-config] 07Universal-Omega deleted branch 03db-hostnames [07:11:02] [02mw-config] 07Universal-Omega deleted branch 03db-hostnames - 13https://github.com/miraheze/mw-config [07:11:12] PROBLEM - db142 Puppet on db142 is WARNING: WARNING: Puppet is currently disabled, message: Database issues, last run 30 minutes ago with 0 failures [07:11:46] miraheze/mw-config - Universal-Omega the build passed. [07:12:12] PROBLEM - db101 Puppet on db101 is WARNING: WARNING: Puppet is currently disabled, message: Database issues, last run 21 minutes ago with 0 failures [07:12:27] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03backends-infra [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/7166ee6c9875 [07:12:29] [02miraheze/puppet] 07Universal-Omega 037166ee6 - Switch MediaWiki backends to new infra [07:12:31] [02puppet] 07Universal-Omega created branch 03backends-infra - 13https://github.com/miraheze/puppet [07:12:34] [02puppet] 07Universal-Omega opened pull request 03#3715: Switch MediaWiki backends to new infra - 13https://github.com/miraheze/puppet/pull/3715 [07:13:13] [02puppet] 07Universal-Omega synchronize pull request 03#3715: Switch MediaWiki backends to new infra - 13https://github.com/miraheze/puppet/pull/3715 [07:13:14] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03backends-infra [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7166ee6c9875...7cf833bb86ab [07:13:15] [02miraheze/puppet] 07Universal-Omega 037cf833b - Fix [07:13:21] PROBLEM - db131 Puppet on db131 is WARNING: WARNING: Puppet is currently disabled, message: Database issues, last run 12 minutes ago with 0 failures [07:13:27] PROBLEM - db121 Puppet on db121 is WARNING: WARNING: Puppet is currently disabled, message: Database issues, last run 24 minutes ago with 0 failures [07:23:34] !log [@test151] starting deploy of {'config': True} to test151 [07:23:35] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [07:32:12] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [07:32:31] !log [@mwtask181] starting deploy of {'config': True} to all [07:32:32] !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [08:02:09] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [08:03:23] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:06:14] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [08:32:03] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [08:34:49] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.andreijiroh.uk.eu.org could not be found [09:01:56] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mahdiruiz.line.pm could not be found [09:05:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.87, 6.59, 6.31 [09:07:59] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.11, 6.38, 6.27 [09:15:36] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.08, 6.71, 6.40 [09:17:34] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.49, 6.65, 6.42 [09:36:52] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.03, 6.72, 6.41 [09:38:51] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.41, 6.57, 6.39 [10:19:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6d2e8ad6a44d...897f8bd537f4 [10:20:02] [02miraheze/mw-config] 07Universal-Omega 03897f8bd - Add all clusters to $wgDatabaseClustersMaintenance [10:20:22] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [10:20:56] miraheze/mw-config - Universal-Omega the build passed. [10:21:03] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 40s [10:21:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/897f8bd537f4...a399b7794310 [10:21:27] [02miraheze/mw-config] 07Universal-Omega 03a399b77 - Remove [10:22:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a399b7794310...7f0b5a19585f [10:22:12] [02miraheze/mw-config] 07Universal-Omega 037f0b5a1 - Add all clusters to $wgDatabaseClustersMaintenance [10:22:18] miraheze/mw-config - Universal-Omega the build passed. [10:22:19] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [10:22:43] PROBLEM - mw133 MediaWiki Rendering on mw133 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:22:50] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 31s [10:22:55] PROBLEM - mw162 MediaWiki Rendering on mw162 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:22:57] PROBLEM - mw132 MediaWiki Rendering on mw132 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:22:57] PROBLEM - mw172 MediaWiki Rendering on mw172 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:23:05] miraheze/mw-config - Universal-Omega the build passed. [10:23:08] PROBLEM - mw141 MediaWiki Rendering on mw141 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:23:10] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:23:20] PROBLEM - mw181 MediaWiki Rendering on mw181 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:23:21] !log [@test151] starting deploy of {'config': True} to test151 [10:23:21] PROBLEM - mw152 MediaWiki Rendering on mw152 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:23:22] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [10:23:44] PROBLEM - mw142 MediaWiki Rendering on mw142 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:24:10] PROBLEM - mw134 MediaWiki Rendering on mw134 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:24:15] PROBLEM - mw171 MediaWiki Rendering on mw171 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:24:15] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/5234d10df9d3...579559bf2c4f [10:24:16] [02miraheze/ErrorPages] 07Universal-Omega 03579559b - - [10:24:19] PROBLEM - mw143 MediaWiki Rendering on mw143 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:24:21] PROBLEM - mw131 MediaWiki Rendering on mw131 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:24:22] PROBLEM - mw151 MediaWiki Rendering on mw151 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:24:40] PROBLEM - mw182 MediaWiki Rendering on mw182 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:24:43] PROBLEM - test151 MediaWiki Rendering on test151 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:24:46] !log [universalomega@mwtask181] starting deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True} to all [10:24:46] PROBLEM - mw161 MediaWiki Rendering on mw161 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [10:25:01] miraheze/ErrorPages - Universal-Omega the build passed. [10:25:17] !log [universalomega@mwtask181] finished deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True} to all - SUCCESS in 30s [10:32:59] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 56% [10:34:12] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/579559bf2c4f...5faa9e95b6b0 [10:34:13] [02miraheze/ErrorPages] 07Universal-Omega 035faa9e9 - Capital headers [10:34:19] !log [universalomega@mwtask181] starting deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True} to all [10:34:50] !log [universalomega@mwtask181] finished deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True} to all - SUCCESS in 30s [10:35:09] miraheze/ErrorPages - Universal-Omega the build passed. [10:36:59] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 31% [10:37:49] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/5faa9e95b6b0...38993c1a6f59 [10:37:52] [02miraheze/ErrorPages] 07Universal-Omega 0338993c1 - Remove invalid SRI [10:37:57] !log [universalomega@mwtask181] starting deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True} to all [10:38:28] !log [universalomega@mwtask181] finished deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True} to all - SUCCESS in 31s [10:38:45] miraheze/ErrorPages - Universal-Omega the build passed. [10:39:23] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/38993c1a6f59...1919badde50f [10:39:24] [02miraheze/ErrorPages] 07Universal-Omega 031919bad - -Meta link (it is down also) [10:40:09] miraheze/ErrorPages - Universal-Omega the build passed. [10:40:11] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/1919badde50f...edc111df7275 [10:40:13] [02miraheze/ErrorPages] 07Universal-Omega 03edc111d - Unscheduled maintenance [10:40:14] !log [universalomega@mwtask181] starting deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True} to all [10:40:24] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 52% [10:40:45] !log [universalomega@mwtask181] finished deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True} to all - SUCCESS in 30s [10:40:57] miraheze/ErrorPages - Universal-Omega the build passed. [10:44:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.83, 6.77, 6.62 [10:45:23] PROBLEM - mw132 Disk Space on mw132 is WARNING: DISK WARNING - free space: / 2608MiB (10% inode=69%); [10:46:51] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.86, 6.49, 6.54 [10:46:58] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 42% [10:47:13] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.91, 6.69, 6.47 [10:48:20] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [10:49:12] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.57, 6.64, 6.47 [10:50:18] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 53% [10:50:57] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 35% [10:52:16] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 31% [10:54:58] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 52% [10:56:11] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 52% [10:56:58] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 12% [10:57:42] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [11:00:07] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 23% [11:00:57] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 52% [11:01:17] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/edc111df7275...b52e74e01278 [11:01:20] [02miraheze/ErrorPages] 07Universal-Omega 03b52e74e - Use miraheze.org/discord [11:02:05] miraheze/ErrorPages - Universal-Omega the build passed. [11:02:36] !log [universalomega@mwtask181] starting deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True} to all [11:02:57] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [11:03:09] !log [universalomega@mwtask181] finished deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True} to all - SUCCESS in 32s [11:03:15] !log [agent@db131] Finished dumping db131 to db171 [11:04:01] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 51% [11:04:09] [02puppet] 07Universal-Omega closed pull request 03#3666: Add renewssl script - 13https://github.com/miraheze/puppet/pull/3666 [11:04:11] [02miraheze/puppet] 07Universal-Omega deleted branch 03renewssl [11:04:12] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+2/-0/±1] 13https://github.com/miraheze/puppet/compare/bc04b1b91bca...f29c0c3b821b [11:04:15] [02miraheze/puppet] 07Universal-Omega 03f29c0c3 - Add renewssl script (#3666) [11:04:17] [02puppet] 07Universal-Omega deleted branch 03renewssl - 13https://github.com/miraheze/puppet [11:04:57] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 38% [11:05:12] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ErrorPages Sync] [11:05:23] miraheze/puppet - Universal-Omega the build has errored. [11:05:53] [02puppet] 07Universal-Omega synchronize pull request 03#3630: Rewrite iaupload and add tests - 13https://github.com/miraheze/puppet/pull/3630 [11:05:54] [02miraheze/puppet] 07Universal-Omega pushed 03214 commits to 03rewrite-iaupload [+59/-105/±520] 13https://github.com/miraheze/puppet/compare/50c68f3aad66...e1f941518db1 [11:05:56] [02miraheze/puppet] 07Universal-Omega 03e1f9415 - Merge branch 'master' into rewrite-iaupload [11:05:59] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [11:06:25] [02puppet] 07Universal-Omega closed pull request 03#3630: Rewrite iaupload and add tests - 13https://github.com/miraheze/puppet/pull/3630 [11:06:28] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/f29c0c3b821b...84f1a3ec5230 [11:06:30] [02miraheze/puppet] 07Universal-Omega 0384f1a3e - Rewrite iaupload and add tests (#3630) [11:06:31] [02miraheze/puppet] 07Universal-Omega deleted branch 03rewrite-iaupload [11:06:33] [02puppet] 07Universal-Omega deleted branch 03rewrite-iaupload - 13https://github.com/miraheze/puppet [11:07:00] [02miraheze/puppet] 07Universal-Omega pushed 032 commits to 03backends-infra [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/7cf833bb86ab...2ab158ff17b3 [11:07:03] [02miraheze/puppet] 07Universal-Omega 03554b00a - Switch MediaWiki backends to new infra [11:07:06] [02miraheze/puppet] 07Universal-Omega 032ab158f - Fix [11:07:09] [02puppet] 07Universal-Omega synchronize pull request 03#3715: Switch MediaWiki backends to new infra - 13https://github.com/miraheze/puppet/pull/3715 [11:07:14] miraheze/puppet - Universal-Omega the build has errored. [11:07:47] miraheze/puppet - Universal-Omega the build has errored. [11:08:00] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 48% [11:10:51] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03backends-infra [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2ab158ff17b3...6ba133bcecf2 [11:10:54] [02miraheze/puppet] 07Universal-Omega 036ba133b - Remove [11:10:57] [02puppet] 07Universal-Omega synchronize pull request 03#3715: Switch MediaWiki backends to new infra - 13https://github.com/miraheze/puppet/pull/3715 [11:12:25] miraheze/puppet - Universal-Omega the build has errored. [11:13:51] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03backends-infra [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6ba133bcecf2...75f751e41cb9 [11:13:52] [02miraheze/puppet] 07Universal-Omega 0375f751e - Just use wikitide.net [11:13:53] [02puppet] 07Universal-Omega synchronize pull request 03#3715: Switch MediaWiki backends to new infra - 13https://github.com/miraheze/puppet/pull/3715 [11:13:54] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [11:15:07] miraheze/puppet - Universal-Omega the build has errored. [11:15:52] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 51% [11:16:26] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03backends-infra [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/75f751e41cb9...e3072a1eaae0 [11:16:29] [02miraheze/puppet] 07Universal-Omega 03e3072a1 - Use wikitide.net by default [11:16:30] [02puppet] 07Universal-Omega synchronize pull request 03#3715: Switch MediaWiki backends to new infra - 13https://github.com/miraheze/puppet/pull/3715 [11:17:39] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03backends-infra [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e3072a1eaae0...c183a5cfe552 [11:17:40] miraheze/puppet - Universal-Omega the build has errored. [11:17:41] [02miraheze/puppet] 07Universal-Omega 03c183a5c - - [11:17:42] [02puppet] 07Universal-Omega synchronize pull request 03#3715: Switch MediaWiki backends to new infra - 13https://github.com/miraheze/puppet/pull/3715 [11:17:48] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 56% [11:18:54] miraheze/puppet - Universal-Omega the build has errored. [11:19:46] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 33% [11:19:48] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 18% [11:20:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03use-private-ips [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/3de305b30795 [11:20:40] [02miraheze/mw-config] 07Universal-Omega 033de305b - Use private IPs when possible [11:20:41] [02mw-config] 07Universal-Omega created branch 03use-private-ips - 13https://github.com/miraheze/mw-config [11:20:44] [02mw-config] 07Universal-Omega opened pull request 03#5448: Use private IPs when possible - 13https://github.com/miraheze/mw-config/pull/5448 [11:21:39] miraheze/mw-config - Universal-Omega the build passed. [11:22:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03use-private-ips [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3de305b30795...a35bdb01c98e [11:22:26] [02miraheze/mw-config] 07Universal-Omega 03a35bdb0 - Update [11:22:28] [02mw-config] 07Universal-Omega synchronize pull request 03#5448: Use private IPs when possible - 13https://github.com/miraheze/mw-config/pull/5448 [11:23:20] miraheze/mw-config - Universal-Omega the build passed. [11:26:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03use-private-ips [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a35bdb01c98e...c23c005f894f [11:26:39] [02miraheze/mw-config] 07Universal-Omega 03c23c005 - Update [11:26:42] [02mw-config] 07Universal-Omega synchronize pull request 03#5448: Use private IPs when possible - 13https://github.com/miraheze/mw-config/pull/5448 [11:27:05] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.andreijiroh.uk.eu.org could not be found [11:27:31] miraheze/mw-config - Universal-Omega the build passed. [11:28:42] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 47% [11:30:22] !log [agent@db142] Finished dumping db142 to db181 [11:32:39] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 23% [11:33:14] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [11:33:34] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [11:35:34] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 30% [11:37:42] eagereren959@gmail.com [11:38:22] #sre [11:38:40] #join-log [11:39:53] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 134.65 ms [11:41:51] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 141.43 ms [11:44:35] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 54% [11:46:34] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 34% [11:47:25] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 52% [11:49:20] PROBLEM - db171 MariaDB Connections on db171 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections. [11:49:21] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [11:49:21] PROBLEM - db171 MariaDB on db171 is CRITICAL: Can't connect to server on 'db171.wikitide.net' (115) [11:49:23] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 13% [11:51:20] RECOVERY - db171 MariaDB Connections on db171 is OK: OK connection usage: 0.6%Current connections: 3 [11:51:21] RECOVERY - db171 MariaDB on db171 is OK: Uptime: 7 Threads: 3 Questions: 48 Slow queries: 0 Opens: 32 Open tables: 28 Queries per second avg: 6.857 [11:57:06] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [11:57:12] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 54% [11:57:22] [02mw-config] 07Universal-Omega closed pull request 03#5448: Use private IPs when possible - 13https://github.com/miraheze/mw-config/pull/5448 [11:57:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/7f0b5a19585f...84fd6986a1ce [11:57:26] [02miraheze/mw-config] 07Universal-Omega 0384fd698 - Use private IPs when possible (#5448) [11:57:27] [02miraheze/mw-config] 07Universal-Omega deleted branch 03use-private-ips [11:57:29] [02mw-config] 07Universal-Omega deleted branch 03use-private-ips - 13https://github.com/miraheze/mw-config [11:58:16] miraheze/mw-config - Universal-Omega the build passed. [11:59:10] [02puppet] 07Universal-Omega closed pull request 03#3715: Switch MediaWiki backends to new infra - 13https://github.com/miraheze/puppet/pull/3715 [11:59:13] [02miraheze/puppet] 07Universal-Omega deleted branch 03backends-infra [11:59:14] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/84f1a3ec5230...b401900227a0 [11:59:16] [02miraheze/puppet] 07Universal-Omega 03b401900 - Switch MediaWiki backends to new infra (#3715) [11:59:17] [02puppet] 07Universal-Omega deleted branch 03backends-infra - 13https://github.com/miraheze/puppet [12:00:21] miraheze/puppet - Universal-Omega the build has errored. [12:01:22] !log [@mwtask181] starting deploy of {'config': True} to all [12:01:23] !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [12:02:38] PROBLEM - cp51 Nginx Backend for mw141 on cp51 is CRITICAL: connect to address localhost and port 8108: Connection refused [12:03:09] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [12:03:16] PROBLEM - cp51 Nginx Backend for mw143 on cp51 is CRITICAL: connect to address localhost and port 8112: Connection refused [12:03:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/84fd6986a1ce...2294e88b86b6 [12:03:19] PROBLEM - cp51 Nginx Backend for mw132 on cp51 is CRITICAL: connect to address localhost and port 8107: Connection refused [12:03:20] [02miraheze/mw-config] 07Universal-Omega 032294e88 - Remove c1 and c5 from $wgDatabaseClustersMaintenance [12:03:26] PROBLEM - cp37 Nginx Backend for mw133 on cp37 is CRITICAL: connect to address localhost and port 8110: Connection refused [12:03:30] PROBLEM - cp37 Nginx Backend for mw132 on cp37 is CRITICAL: connect to address localhost and port 8107: Connection refused [12:03:31] PROBLEM - cp51 Nginx Backend for mw134 on cp51 is CRITICAL: connect to address localhost and port 8111: Connection refused [12:03:33] PROBLEM - cp51 Nginx Backend for mw142 on cp51 is CRITICAL: connect to address localhost and port 8109: Connection refused [12:03:35] PROBLEM - cp36 Nginx Backend for mw132 on cp36 is CRITICAL: connect to address localhost and port 8107: Connection refused [12:03:35] PROBLEM - cp36 Nginx Backend for mw141 on cp36 is CRITICAL: connect to address localhost and port 8108: Connection refused [12:03:41] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [12:03:42] PROBLEM - cp36 Nginx Backend for mw143 on cp36 is CRITICAL: connect to address localhost and port 8112: Connection refused [12:03:44] PROBLEM - cp51 Nginx Backend for mw133 on cp51 is CRITICAL: connect to address localhost and port 8110: Connection refused [12:03:47] PROBLEM - cp37 Nginx Backend for mw131 on cp37 is CRITICAL: connect to address localhost and port 8106: Connection refused [12:03:47] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 6s [12:04:00] PROBLEM - cp51 Nginx Backend for mw131 on cp51 is CRITICAL: connect to address localhost and port 8106: Connection refused [12:04:01] PROBLEM - cp26 Nginx Backend for mw133 on cp26 is CRITICAL: connect to address localhost and port 8110: Connection refused [12:04:01] PROBLEM - cp41 Nginx Backend for mw134 on cp41 is CRITICAL: connect to address localhost and port 8111: Connection refused [12:04:01] PROBLEM - cp26 Nginx Backend for mw134 on cp26 is CRITICAL: connect to address localhost and port 8111: Connection refused [12:04:07] miraheze/mw-config - Universal-Omega the build passed. [12:04:10] PROBLEM - cp36 Nginx Backend for mw131 on cp36 is CRITICAL: connect to address localhost and port 8106: Connection refused [12:04:15] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.280 second response time [12:04:16] PROBLEM - cp41 Nginx Backend for mw143 on cp41 is CRITICAL: connect to address localhost and port 8112: Connection refused [12:04:23] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.255 second response time [12:04:35] PROBLEM - cp26 Nginx Backend for mw143 on cp26 is CRITICAL: connect to address localhost and port 8112: Connection refused [12:04:37] PROBLEM - cp36 Nginx Backend for mw133 on cp36 is CRITICAL: connect to address localhost and port 8110: Connection refused [12:04:39] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.217 second response time [12:04:41] PROBLEM - cp36 Nginx Backend for mw142 on cp36 is CRITICAL: connect to address localhost and port 8109: Connection refused [12:04:43] PROBLEM - cp26 Nginx Backend for mw142 on cp26 is CRITICAL: connect to address localhost and port 8109: Connection refused [12:04:46] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.264 second response time [12:04:50] PROBLEM - cp41 Nginx Backend for mw132 on cp41 is CRITICAL: connect to address localhost and port 8107: Connection refused [12:04:52] PROBLEM - cp37 Nginx Backend for mw134 on cp37 is CRITICAL: connect to address localhost and port 8111: Connection refused [12:04:56] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.292 second response time [12:04:56] PROBLEM - cp26 Nginx Backend for mw131 on cp26 is CRITICAL: connect to address localhost and port 8106: Connection refused [12:04:57] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.263 second response time [12:04:58] PROBLEM - cp41 Nginx Backend for mw133 on cp41 is CRITICAL: connect to address localhost and port 8110: Connection refused [12:05:00] PROBLEM - cp26 Nginx Backend for mw141 on cp26 is CRITICAL: connect to address localhost and port 8108: Connection refused [12:05:04] PROBLEM - cp37 Nginx Backend for mw142 on cp37 is CRITICAL: connect to address localhost and port 8109: Connection refused [12:05:07] PROBLEM - cp26 Nginx Backend for mw132 on cp26 is CRITICAL: connect to address localhost and port 8107: Connection refused [12:05:07] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 15% [12:05:09] PROBLEM - cp41 Nginx Backend for mw142 on cp41 is CRITICAL: connect to address localhost and port 8109: Connection refused [12:05:09] PROBLEM - cp41 Nginx Backend for mw131 on cp41 is CRITICAL: connect to address localhost and port 8106: Connection refused [12:05:10] PROBLEM - cp37 Nginx Backend for mw141 on cp37 is CRITICAL: connect to address localhost and port 8108: Connection refused [12:05:11] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.227 second response time [12:05:20] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.210 second response time [12:05:20] PROBLEM - cp36 Nginx Backend for mw134 on cp36 is CRITICAL: connect to address localhost and port 8111: Connection refused [12:05:21] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.259 second response time [12:05:21] PROBLEM - cp37 Nginx Backend for mw143 on cp37 is CRITICAL: connect to address localhost and port 8112: Connection refused [12:05:32] PROBLEM - cp41 Nginx Backend for mw141 on cp41 is CRITICAL: connect to address localhost and port 8108: Connection refused [12:07:18] !log [universalomega@mwtask181] starting deploy of {'force': True, 'versions': ['1.40', '1.41'], 'upgrade_extensions': 'RequestSSL'} to all [12:07:29] !log [universalomega@mwtask181] finished deploy of {'force': True, 'versions': ['1.40', '1.41'], 'upgrade_extensions': 'RequestSSL'} to all - SUCCESS in 11s [12:11:03] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [12:13:00] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 41% [12:14:59] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 39% [12:23:49] !log [@test151] starting deploy of {'config': True} to test151 [12:23:50] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [12:27:07] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.andreijiroh.uk.eu.org could not be found [12:27:10] [02miraheze/MirahezeDebug] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeDebug/compare/ca07bd362448...98b027e4b8bc [12:27:11] [02miraheze/MirahezeDebug] 07Universal-Omega 0398b027e - Remove old servers [12:29:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2294e88b86b6...36880147b946 [12:29:49] [02miraheze/mw-config] 07Universal-Omega 033688014 - Remove c3 from $wgDatabaseClustersMaintenance [12:30:14] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [12:30:21] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 6s [12:30:41] miraheze/mw-config - Universal-Omega the build passed. [12:37:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.84, 6.85, 6.45 [12:37:57] !log [agent@db101] Finished dumping db101 to db121 [12:39:01] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [12:39:01] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.77, 6.48, 6.36 [12:47:16] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 43% [12:49:11] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 38% [12:53:26] !log [@test151] starting deploy of {'config': True} to test151 [12:53:27] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [12:56:41] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [13:07:01] PROBLEM - cp26 Disk Space on cp26 is CRITICAL: DISK CRITICAL - free space: / 5576MiB (5% inode=98%); [13:08:09] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found [13:15:33] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:24:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/36880147b946...4a688dcd15d3 [13:24:45] [02miraheze/mw-config] 07Universal-Omega 034a688dc - Remove c2 from $wgDatabaseClustersMaintenance [13:25:37] miraheze/mw-config - Universal-Omega the build passed. [13:26:17] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.andreijiroh.uk.eu.org could not be found [13:28:47] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [13:28:53] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 6s [13:29:01] RECOVERY - cp26 Disk Space on cp26 is OK: DISK OK - free space: / 15124MiB (16% inode=98%); [13:30:43] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [13:30:54] !log [paladox@mwtask181] starting deploy of {'config': True, 'force': True} to all [13:31:01] !log [paladox@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 6s [13:34:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.99, 6.69, 6.34 [13:36:04] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.16, 6.50, 6.32 [13:51:01] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.81, 6.74, 6.46 [13:53:02] !log [@test151] starting deploy of {'config': True} to test151 [13:53:03] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [13:55:01] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.52, 6.42, 6.42 [14:00:32] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mahdiruiz.line.pm could not be found [14:05:47] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.42, 6.79, 6.42 [14:09:44] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.46, 6.69, 6.46 [14:17:59] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.90, 6.86, 6.62 [14:19:58] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.34, 6.60, 6.55 [14:21:15] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.27, 6.93, 6.78 [14:23:13] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.58, 6.76, 6.73 [14:23:34] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.98, 6.55, 6.43 [14:25:33] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.77, 6.62, 6.47 [14:27:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 6.98, 6.47 [14:29:40] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.09, 6.66, 6.42 [14:30:34] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.88, 6.51, 5.75 [14:32:34] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.34, 6.13, 5.72 [14:32:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.84, 6.79, 6.65 [14:34:56] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.38, 6.61, 6.60 [14:44:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.61, 7.14, 6.77 [14:46:55] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.16, 6.98, 6.73 [14:50:55] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.61, 6.80, 6.71 [14:51:10] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.83, 6.53, 6.41 [14:54:36] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.46, 6.78, 6.76 [14:59:03] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.62, 6.41, 6.45 [15:02:48] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 140.27 ms [15:04:50] PROBLEM - ping6 on cp41 is WARNING: PING WARNING - Packet loss = 0%, RTA = 200.22 ms [15:04:56] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.62, 6.92, 6.61 [15:05:06] db161 still not working for me [15:05:25] Probably db171 too [15:06:34] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.22, 6.82, 6.67 [15:06:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.02, 6.54, 6.34 [15:06:51] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 168.71 ms [15:08:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.65, 6.78, 6.63 [15:10:46] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.83, 6.30, 6.31 [15:34:34] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.42, 6.72, 6.79 [15:36:32] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 174.83 ms [15:38:34] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 147.90 ms [15:42:34] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.82, 6.79, 6.79 [16:00:34] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.75, 6.72, 6.80 [16:04:34] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.58, 6.98, 6.87 [16:04:55] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10000 milliseconds [16:05:55] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.18, 6.41, 6.13 [16:06:54] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.755 second response time [16:09:48] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.22, 6.61, 5.81 [16:09:55] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.03, 6.91, 6.37 [16:10:34] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.63, 6.71, 6.79 [16:11:48] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.25, 7.21, 6.13 [16:13:47] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.13, 7.17, 6.73 [16:14:34] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.63, 8.26, 7.40 [16:15:47] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.20, 7.78, 7.00 [16:16:13] [02RequestSSL] 07redbluegreenhat reviewed pull request 03#24 commit - 13https://github.com/miraheze/RequestSSL/pull/24#discussion_r1468516805 [16:16:34] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.00, 7.80, 7.34 [16:19:47] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.19, 7.99, 7.01 [16:21:47] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.18, 7.83, 7.32 [16:23:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.51, 7.51, 7.34 [16:25:46] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.51, 6.74, 6.74 [16:33:46] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 4.99, 5.92, 6.60 [16:34:34] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.04, 7.86, 7.55 [16:38:00] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.73, 6.20, 6.68 [16:38:35] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.50, 7.77, 7.60 [16:42:03] PROBLEM - mw134 MediaWiki Rendering on mw134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:28] PROBLEM - mw133 MediaWiki Rendering on mw133 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:31] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [16:42:41] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [16:43:10] PROBLEM - mw132 Disk Space on mw132 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:43:16] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:01] PROBLEM - db131 MariaDB on db131 is UNKNOWN: [16:44:17] PROBLEM - db131 Puppet on db131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:44:23] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [16:44:26] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [16:44:31] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:46] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [16:45:16] PROBLEM - ping6 on mw132 is CRITICAL: PING CRITICAL - Packet loss = 100% [16:45:25] PROBLEM - mw132 SSH on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:27] PROBLEM - db142 MariaDB on db142 is UNKNOWN: [16:45:28] PROBLEM - mw131 SSH on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:33] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.12, 7.95, 7.71 [16:45:37] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:38] PROBLEM - mw133 SSH on mw133 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:38] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [16:45:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/commit/1967802ffa88 [16:45:46] [02miraheze/puppet] 07Universal-Omega 031967802 - Delete hieradata/hosts/cp34.yaml [16:45:48] [02puppet] 07Universal-Omega created branch 03decom - 13https://github.com/miraheze/puppet [16:45:56] [02puppet] 07Universal-Omega opened pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:46:08] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:46:16] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/1967802ffa88...18355518f1c8 [16:46:17] PROBLEM - mw143 MediaWiki Rendering on mw143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:46:18] [02miraheze/puppet] 07Universal-Omega 031835551 - Delete hieradata/hosts/cp35.yaml [16:46:21] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:46:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/18355518f1c8...149a46ab5c83 [16:46:43] [02miraheze/puppet] 07Universal-Omega 03149a46a - Delete hieradata/hosts/db101.yaml [16:46:46] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:46:47] PROBLEM - db142 Puppet on db142 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:46:51] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [16:46:57] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/149a46ab5c83...e2d8215f1f94 [16:46:58] [02miraheze/puppet] 07Universal-Omega 03e2d8215 - Delete hieradata/hosts/db131.yaml [16:47:01] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:47:11] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/e2d8215f1f94...fbe193852ad7 [16:47:12] [02miraheze/puppet] 07Universal-Omega 03fbe1938 - Delete hieradata/hosts/db142.yaml [16:47:13] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:47:16] PROBLEM - mem131 SSH on mem131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:47:17] PROBLEM - ping6 on mw133 is CRITICAL: PING CRITICAL - Packet loss = 100% [16:47:22] PROBLEM - Host mw134 is DOWN: PING CRITICAL - Packet loss = 100% [16:47:23] PROBLEM - ping6 on mw131 is CRITICAL: PING CRITICAL - Packet loss = 100% [16:47:31] PROBLEM - ping6 on db131 is CRITICAL: PING CRITICAL - Packet loss = 100% [16:47:32] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/fbe193852ad7...13131a277838 [16:47:35] [02miraheze/puppet] 07Universal-Omega 0313131a2 - Delete hieradata/hosts/jobchron121.yaml [16:47:38] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:47:44] PROBLEM - ping6 on mem131 is CRITICAL: PING CRITICAL - Packet loss = 100% [16:48:01] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/13131a277838...62486463c0c1 [16:48:02] PROBLEM - mw133 php-fpm on mw133 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:48:03] PROBLEM - mem131 memcached on mem131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:04] [02miraheze/puppet] 07Universal-Omega 036248646 - Delete hieradata/hosts/mem131.yaml [16:48:06] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:48:19] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/62486463c0c1...f58cd88eb6dd [16:48:20] [02miraheze/puppet] 07Universal-Omega 03f58cd88 - Delete hieradata/hosts/mem141.yaml [16:48:22] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:48:23] PROBLEM - Host db131 is DOWN: PING CRITICAL - Packet loss = 100% [16:48:31] PROBLEM - mw133 Puppet on mw133 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:48:35] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/f58cd88eb6dd...50859b4cb051 [16:48:38] [02miraheze/puppet] 07Universal-Omega 0350859b4 - Delete hieradata/hosts/mw131.yaml [16:48:39] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:48:44] PROBLEM - ping6 on mw143 is CRITICAL: PING CRITICAL - Packet loss = 100% [16:48:46] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/50859b4cb051...923929bb6f2f [16:48:47] [02miraheze/puppet] 07Universal-Omega 03923929b - Delete hieradata/hosts/mw132.yaml [16:48:50] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:48:59] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/923929bb6f2f...819c1bc95e96 [16:49:02] [02miraheze/puppet] 07Universal-Omega 03819c1bc - Delete hieradata/hosts/mw133.yaml [16:49:05] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:49:08] PROBLEM - mw142 SSH on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:10] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.51, 7.69, 7.65 [16:49:12] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/819c1bc95e96...abc1b57b346b [16:49:13] [02miraheze/puppet] 07Universal-Omega 03abc1b57 - Delete hieradata/hosts/mw134.yaml [16:49:15] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:49:19] PROBLEM - mw131 NTP time on mw131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:20] PROBLEM - mem141 memcached on mem141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:22] PROBLEM - mw133 PowerDNS Recursor on mw133 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:22] PROBLEM - mw133 conntrack_table_size on mw133 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:25] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:28] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/abc1b57b346b...497b3c0d323c [16:49:29] PROBLEM - mw131 Disk Space on mw131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:31] [02miraheze/puppet] 07Universal-Omega 03497b3c0 - Delete hieradata/hosts/mw141.yaml [16:49:33] PROBLEM - db142 SSH on db142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:33] PROBLEM - mem131 Disk Space on mem131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:34] PROBLEM - mw132 ferm_active on mw132 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:34] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:49:37] PROBLEM - mw141 SSH on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:38] PROBLEM - Host db142 is DOWN: PING CRITICAL - Packet loss = 100% [16:49:41] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:44] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/497b3c0d323c...ae94fac33e5e [16:49:44] PROBLEM - ping6 on mw142 is CRITICAL: PING CRITICAL - Packet loss = 100% [16:49:45] [02miraheze/puppet] 07Universal-Omega 03ae94fac - Delete hieradata/hosts/mw142.yaml [16:49:46] PROBLEM - mem131 NTP time on mem131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:48] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:49:50] PROBLEM - Host mw131 is DOWN: PING CRITICAL - Packet loss = 100% [16:50:02] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/ae94fac33e5e...12126b31ee0e [16:50:05] [02miraheze/puppet] 07Universal-Omega 0312126b3 - Delete hieradata/hosts/mw143.yaml [16:50:08] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:50:13] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/12126b31ee0e...70c8fe08c089 [16:50:15] [02miraheze/puppet] 07Universal-Omega 0370c8fe0 - Delete hieradata/hosts/mwtask141.yaml [16:50:18] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:50:22] PROBLEM - Host mem131 is DOWN: PING CRITICAL - Packet loss = 100% [16:50:24] PROBLEM - Host mem141 is DOWN: PING CRITICAL - Packet loss = 100% [16:50:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/70c8fe08c089...4b18bf306248 [16:50:45] [02miraheze/puppet] 07Universal-Omega 034b18bf3 - Delete hieradata/hosts/db121.yaml [16:50:46] PROBLEM - mw133 Disk Space on mw133 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:50:46] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:50:52] PROBLEM - mw132 NTP time on mw132 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:50:56] PROBLEM - mw133 NTP time on mw133 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:50:57] PROBLEM - Host mw133 is DOWN: PING CRITICAL - Packet loss = 100% [16:51:11] PROBLEM - Host mw132 is DOWN: PING CRITICAL - Packet loss = 100% [16:51:11] PROBLEM - Host mw143 is DOWN: PING CRITICAL - Packet loss = 100% [16:51:42] PROBLEM - mw141 ferm_active on mw141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:51:51] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:52:02] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:52:10] PROBLEM - ping6 on mw141 is CRITICAL: PING CRITICAL - Packet loss = 100% [16:52:25] PROBLEM - mw142 Disk Space on mw142 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:52:26] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4b18bf306248...2130defc9df4 [16:52:27] [02miraheze/puppet] 07Universal-Omega 032130def - Update site.pp [16:52:29] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [16:52:34] PROBLEM - mw141 Disk Space on mw141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:52:35] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:52:56] PROBLEM - Host mw142 is DOWN: PING CRITICAL - Packet loss = 100% [16:53:06] PROBLEM - mw141 PowerDNS Recursor on mw141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:53:10] PROBLEM - mw141 NTP time on mw141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:53:19] PROBLEM - mw141 php-fpm on mw141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:53:36] PROBLEM - Host mw141 is DOWN: PING CRITICAL - Packet loss = 100% [16:58:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03remove-scsvg [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/24c3baf5f412 [16:59:01] [02miraheze/mw-config] 07Universal-Omega 0324c3baf - Remove support for scsvg [16:59:03] [02mw-config] 07Universal-Omega created branch 03remove-scsvg - 13https://github.com/miraheze/mw-config [16:59:04] [02mw-config] 07Universal-Omega opened pull request 03#5449: Remove support for scsvg - 13https://github.com/miraheze/mw-config/pull/5449 [16:59:55] miraheze/mw-config - Universal-Omega the build passed. [17:02:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03remove-scsvg [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/24c3baf5f412...dc46697cd26e [17:02:02] [02miraheze/mw-config] 07Universal-Omega 03dc46697 - - [17:02:05] [02mw-config] 07Universal-Omega synchronize pull request 03#5449: Remove support for scsvg - 13https://github.com/miraheze/mw-config/pull/5449 [17:02:50] miraheze/mw-config - Universal-Omega the build passed. [17:02:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03remove-scsvg [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/dc46697cd26e...089025b3aac5 [17:02:58] [02miraheze/mw-config] 07Universal-Omega 03089025b - - [17:03:01] [02mw-config] 07Universal-Omega synchronize pull request 03#5449: Remove support for scsvg - 13https://github.com/miraheze/mw-config/pull/5449 [17:03:58] miraheze/mw-config - Universal-Omega the build passed. [17:04:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03remove-scsvg [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/089025b3aac5...33511d3e7251 [17:04:22] [02miraheze/mw-config] 07Universal-Omega 0333511d3 - - [17:04:25] [02mw-config] 07Universal-Omega synchronize pull request 03#5449: Remove support for scsvg - 13https://github.com/miraheze/mw-config/pull/5449 [17:05:14] miraheze/mw-config - Universal-Omega the build passed. [17:05:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03remove-scsvg [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/33511d3e7251...1ef754f8c882 [17:05:52] [02miraheze/mw-config] 07Universal-Omega 031ef754f - - [17:05:55] [02mw-config] 07Universal-Omega synchronize pull request 03#5449: Remove support for scsvg - 13https://github.com/miraheze/mw-config/pull/5449 [17:06:44] miraheze/mw-config - Universal-Omega the build passed. [17:10:04] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.54, 6.29, 6.70 [17:11:40] [02mw-config] 07Universal-Omega closed pull request 03#5449: Remove support for scsvg - 13https://github.com/miraheze/mw-config/pull/5449 [17:11:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/mw-config/compare/4a688dcd15d3...97ab82959130 [17:11:44] [02miraheze/mw-config] 07Universal-Omega 0397ab829 - Remove support for scsvg (#5449) [17:11:47] [02miraheze/mw-config] 07Universal-Omega deleted branch 03remove-scsvg [17:11:48] [02mw-config] 07Universal-Omega deleted branch 03remove-scsvg - 13https://github.com/miraheze/mw-config [17:12:37] miraheze/mw-config - Universal-Omega the build passed. [17:15:28] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 152.42 ms [17:15:33] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:18:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b401900227a0...c553ad0e55f4 [17:18:21] [02miraheze/puppet] 07paladox 03c553ad0 - Update services.conf [17:18:32] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 146.30 ms [17:20:33] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:23:47] !log [@test151] starting deploy of {'config': True} to test151 [17:23:48] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [17:24:54] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [17:44:30] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/2130defc9df4...08c46ca3993b [17:44:31] [02miraheze/puppet] 07Universal-Omega 0308c46ca - Delete hieradata/hosts/swiftac111.yaml [17:44:33] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [17:45:01] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/08c46ca3993b...73809d1af9c9 [17:45:02] [02miraheze/puppet] 07Universal-Omega 0373809d1 - Update site.pp [17:45:03] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [17:59:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03rw [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/5b69950e067a [17:59:57] [02miraheze/mw-config] 07Universal-Omega 035b69950 - Unset read only on everything except c4 [17:59:58] [02mw-config] 07Universal-Omega created branch 03rw - 13https://github.com/miraheze/mw-config [17:59:59] [02mw-config] 07Universal-Omega opened pull request 03#5450: Unset read only on everything except c4 - 13https://github.com/miraheze/mw-config/pull/5450 [18:00:52] miraheze/mw-config - Universal-Omega the build passed. [18:08:10] [02mw-config] 07Universal-Omega closed pull request 03#5450: Unset read only on everything except c4 - 13https://github.com/miraheze/mw-config/pull/5450 [18:08:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/97ab82959130...6b1d0f7b7cba [18:08:16] [02miraheze/mw-config] 07Universal-Omega 036b1d0f7 - Unset read only on everything except c4 (#5450) [18:08:19] [02miraheze/mw-config] 07Universal-Omega deleted branch 03rw [18:08:20] [02mw-config] 07Universal-Omega deleted branch 03rw - 13https://github.com/miraheze/mw-config [18:09:18] miraheze/mw-config - Universal-Omega the build passed. [18:21:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.90, 6.61, 6.04 [18:23:13] !log [@test151] starting deploy of {'config': True} to test151 [18:23:14] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [18:24:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.73, 6.34, 6.05 [18:25:56] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.andreijiroh.uk.eu.org could not be found [18:31:45] !log [@mwtask181] starting deploy of {'config': True} to all [18:31:46] !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [18:36:02] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [18:47:28] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#24: T11699: Use jobs and hooks to automate checking that a domain is pointed - 13https://github.com/miraheze/RequestSSL/pull/24 [18:50:43] miraheze/RequestSSL - redbluegreenhat the build has errored. [18:50:44] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 151.70 ms [19:02:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6b1d0f7b7cba...af85f989a524 [19:02:24] !log [@mwtask181] starting deploy of {'config': True} to all [19:02:25] [02miraheze/mw-config] 07Universal-Omega 03af85f98 - Remove c4 from $wgDatabaseClustersMaintenance [19:02:25] !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [19:02:46] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 147.74 ms [19:03:18] miraheze/mw-config - Universal-Omega the build passed. [19:03:51] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [19:03:55] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/af85f989a524...9b4a36db92c8 [19:03:58] [02miraheze/mw-config] 07Universal-Omega 039b4a36d - Unset read only on c4 [19:04:45] miraheze/mw-config - Universal-Omega the build passed. [19:06:52] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#24: T11699: Use jobs and hooks to automate checking that a domain is pointed - 13https://github.com/miraheze/RequestSSL/pull/24 [19:10:11] miraheze/RequestSSL - redbluegreenhat the build has errored. [19:11:36] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#24: T11699: Use jobs and hooks to automate checking that a domain is pointed - 13https://github.com/miraheze/RequestSSL/pull/24 [19:15:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/9d80d637a7d3 [19:15:31] [02miraheze/puppet] 07paladox 039d80d63 - db151/db161/db171/db181 increase buffer pool to 30g [19:15:33] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [19:15:34] [02puppet] 07paladox opened pull request 03#3717: db151/db161/db171/db181 increase buffer pool to 30g - 13https://github.com/miraheze/puppet/pull/3717 [19:15:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9d80d637a7d3...037b138cf55f [19:15:50] [02miraheze/puppet] 07paladox 03037b138 - Update db151.yaml [19:15:51] [02puppet] 07paladox synchronize pull request 03#3717: db151/db161/db171/db181 increase buffer pool to 30g - 13https://github.com/miraheze/puppet/pull/3717 [19:15:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/037b138cf55f...1ef574917dbe [19:15:58] [02miraheze/puppet] 07paladox 031ef5749 - Update db171.yaml [19:15:58] [02puppet] 07paladox synchronize pull request 03#3717: db151/db161/db171/db181 increase buffer pool to 30g - 13https://github.com/miraheze/puppet/pull/3717 [19:16:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1ef574917dbe...e5f43090c6ea [19:16:07] [02miraheze/puppet] 07paladox 03e5f4309 - Update db181.yaml [19:16:08] [02puppet] 07paladox synchronize pull request 03#3717: db151/db161/db171/db181 increase buffer pool to 30g - 13https://github.com/miraheze/puppet/pull/3717 [19:16:15] [02puppet] 07paladox closed pull request 03#3717: db151/db161/db171/db181 increase buffer pool to 30g - 13https://github.com/miraheze/puppet/pull/3717 [19:16:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/c553ad0e55f4...fa382246e8c5 [19:16:19] [02miraheze/puppet] 07paladox 03fa38224 - db151/db161/db171/db181 increase buffer pool to 30g (#3717) [19:16:21] miraheze/RequestSSL - redbluegreenhat the build has errored. [19:16:22] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [19:16:24] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [19:17:52] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.852 second response time [19:19:13] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [19:19:28] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [19:20:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/fb4a72fee87c [19:20:30] [02miraheze/puppet] 07paladox 03fb4a72f - Increase mem151/mem161 size to 63 gb [19:20:32] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [19:20:34] [02puppet] 07paladox opened pull request 03#3718: Increase mem151/mem161 size to 63 gb - 13https://github.com/miraheze/puppet/pull/3718 [19:20:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fb4a72fee87c...eb82d96ff93a [19:20:49] [02miraheze/puppet] 07paladox 03eb82d96 - Update mem161.yaml [19:20:51] [02puppet] 07paladox synchronize pull request 03#3718: Increase mem151/mem161 size to 63 gb - 13https://github.com/miraheze/puppet/pull/3718 [19:20:56] [02puppet] 07paladox closed pull request 03#3718: Increase mem151/mem161 size to 63 gb - 13https://github.com/miraheze/puppet/pull/3718 [19:21:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/fa382246e8c5...6b198d25d7f0 [19:21:00] [02miraheze/puppet] 07paladox 036b198d2 - Increase mem151/mem161 size to 63 gb (#3718) [19:21:02] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [19:21:05] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [19:21:15] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php (START) [19:21:29] @cosmicalpha run it with --procs [19:21:34] maybe --procs 24 or something [19:21:41] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php (END - exit=2) [19:21:48] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --procs 24 (START) [19:22:15] PROBLEM - mw172 HTTPS on mw172 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.057 second response time [19:22:29] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.919 second response time [19:29:48] @agentisai tbh, it was coping amazingly well without [19:29:55] And task has way more runners [19:30:12] So we could do way more runners [19:32:11] yep [19:33:23] Miraheze was reforged in fire [19:34:29] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [19:35:12] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:40:18] Yeah [19:40:22] It’s now Miratide [19:40:33] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:41:55] MiraForge??? 🤯🤯🤯🤯 [19:43:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.39, 7.13, 6.23 [19:44:11] Haha yes [19:47:34] I ammend [19:47:50] It was reforged at the bottom of a hydrothermic vent in the ocean [19:49:02] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/73809d1af9c9...28a6df06b2bb [19:49:03] [02miraheze/puppet] 07Universal-Omega 0328a6df0 - Update site.pp [19:49:06] [02puppet] 07Universal-Omega synchronize pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [19:49:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.00, 6.33, 6.20 [19:55:33] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:58:44] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/invalidateUserSessions.php --wiki=metawiki --user=CircleyDoesTest (END - exit=0) [19:58:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:01] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found [20:13:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6b198d25d7f0...8f8b7675971d [20:13:52] [02miraheze/puppet] 07paladox 038f8b767 - php: increase pm.max_requests to 6k [20:15:08] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.21, 6.72, 6.40 [20:17:54] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9b4a36db92c8...ecff5d2f491e [20:17:55] [02miraheze/mw-config] 07AgentIsai 03ecff5d2 - Update sitenotice [20:18:50] miraheze/mw-config - AgentIsai the build passed. [20:18:59] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.69, 6.76, 6.50 [20:20:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.09, 6.75, 6.33 [20:20:53] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/dns/commit/3df8656fb30c [20:20:54] [02miraheze/dns] 07Universal-Omega 033df8656 - Update 0.0.0.0.0.4.7.6.0.1.a.2.ip6.arpa [20:20:56] [02dns] 07Universal-Omega created branch 03decom - 13https://github.com/miraheze/dns [20:21:06] [02dns] 07Universal-Omega opened pull request 03#482: Decom servers - 13https://github.com/miraheze/dns/pull/482 [20:21:19] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.30, 6.88, 6.48 [20:22:22] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/dns/compare/3df8656fb30c...cbbcd99cc438 [20:22:23] [02miraheze/dns] 07Universal-Omega 03cbbcd99 - Update miraheze.org [20:22:24] [02dns] 07Universal-Omega synchronize pull request 03#482: Decom servers - 13https://github.com/miraheze/dns/pull/482 [20:22:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.56, 6.59, 6.32 [20:22:51] [02puppet] 07Universal-Omega closed pull request 03#3716: Decom servers - 13https://github.com/miraheze/puppet/pull/3716 [20:22:54] [02miraheze/puppet] 07Universal-Omega deleted branch 03decom [20:22:57] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-18/±1] 13https://github.com/miraheze/puppet/compare/8f8b7675971d...1a5cda551745 [20:23:00] [02miraheze/puppet] 07Universal-Omega 031a5cda5 - Decom servers (#3716) [20:23:01] [02puppet] 07Universal-Omega deleted branch 03decom - 13https://github.com/miraheze/puppet [20:23:15] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.54, 6.75, 6.47 [20:23:20] !log [@test151] starting deploy of {'config': True} to test151 [20:23:21] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [20:23:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:23:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:26:48] [02dns] 07Universal-Omega closed pull request 03#482: Decom servers - 13https://github.com/miraheze/dns/pull/482 [20:26:51] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/760951085db4...68691e681b9e [20:26:52] [02miraheze/dns] 07Universal-Omega 0368691e6 - Decom servers (#482) [20:26:53] [02miraheze/dns] 07Universal-Omega deleted branch 03decom [20:26:55] [02dns] 07Universal-Omega deleted branch 03decom - 13https://github.com/miraheze/dns [20:32:22] !log [@mwtask181] starting deploy of {'config': True} to all [20:32:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:32:28] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 7s [20:32:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:35:23] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mahdiruiz.line.pm could not be found [20:36:51] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/1a5cda551745...66433f61af7a [20:36:53] [02miraheze/puppet] 07AgentIsai 0366433f6 - Create cp27 [20:37:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.02, 6.56, 6.33 [20:39:46] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.58, 6.22, 6.23 [20:42:38] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/66433f61af7a...3baef58dbb8d [20:42:40] [02miraheze/puppet] 07Universal-Omega 033baef58 - varnish: remove SCSVG from acls [20:44:55] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/68691e681b9e...2e6c93a6b8cb [20:44:57] [02miraheze/dns] 07AgentIsai 032e6c93a - Add cp27 to dns [20:45:03] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.71, 7.06, 6.77 [20:45:34] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/2e6c93a6b8cb...a33d112eac61 [20:45:35] [02miraheze/dns] 07AgentIsai 03a33d112 - Add cp27 to config [20:47:21] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03runners [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/3cf2120ec5dd [20:47:23] [02miraheze/puppet] 07Universal-Omega 033cf2120 - jobrunner-hi: increase runners [20:47:25] [02puppet] 07Universal-Omega created branch 03runners - 13https://github.com/miraheze/puppet [20:47:26] [02puppet] 07Universal-Omega opened pull request 03#3719: jobrunner-hi: increase runners - 13https://github.com/miraheze/puppet/pull/3719 [20:48:06] PROBLEM - cp27 HTTPS on cp27 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [20:49:35] PROBLEM - cp27 Nginx Backend for mwtask181 on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:49:36] PROBLEM - cp27 Nginx Backend for mw151 on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:49:39] PROBLEM - cp27 conntrack_table_size on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:49:40] PROBLEM - cp27 NTP time on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:49:41] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:49:42] PROBLEM - cp27 PowerDNS Recursor on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:49:44] PROBLEM - cp27 Nginx Backend for phorge171 on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:49:45] PROBLEM - cp27 Nginx Backend for mw161 on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:49:47] PROBLEM - cp27 Disk Space on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:49:51] PROBLEM - cp27 Nginx Backend for mw181 on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:49:52] PROBLEM - cp27 Nginx Backend for mon181 on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:49:53] PROBLEM - cp27 Nginx Backend for mw171 on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:49:55] [02puppet] 07Universal-Omega closed pull request 03#3719: jobrunner-hi: increase runners - 13https://github.com/miraheze/puppet/pull/3719 [20:49:58] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3baef58dbb8d...cd196aaf1560 [20:49:58] PROBLEM - cp27 ferm_active on cp27 is UNKNOWN: NRPE: Unable to read output [20:50:00] [02miraheze/puppet] 07Universal-Omega 03cd196aa - jobrunner-hi: increase runners (#3719) [20:50:02] PROBLEM - cp27 Nginx Backend for mw162 on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:50:03] [02miraheze/puppet] 07Universal-Omega deleted branch 03runners [20:50:04] PROBLEM - cp27 Puppet on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:50:04] [02puppet] 07Universal-Omega deleted branch 03runners - 13https://github.com/miraheze/puppet [20:50:08] PROBLEM - cp27 Nginx Backend for swiftproxy171 on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:50:10] PROBLEM - cp27 Nginx Backend for mw182 on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:50:11] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:50:12] PROBLEM - cp27 Nginx Backend for mw172 on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:50:12] PROBLEM - cp27 APT on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:50:14] PROBLEM - cp27 Nginx Backend for matomo151 on cp27 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:51:33] PROBLEM - cp27 conntrack_table_size on cp27 is UNKNOWN: NRPE: Unable to read output [20:51:34] RECOVERY - cp27 Nginx Backend for mw151 on cp27 is OK: TCP OK - 0.000 second response time on localhost port 8113 [20:51:35] RECOVERY - cp27 Nginx Backend for mwtask181 on cp27 is OK: TCP OK - 0.000 second response time on localhost port 8160 [20:51:38] RECOVERY - cp27 NTP time on cp27 is OK: NTP OK: Offset -0.003527611494 secs [20:51:41] RECOVERY - cp27 Nginx Backend for phorge171 on cp27 is OK: TCP OK - 0.000 second response time on localhost port 8202 [20:51:41] RECOVERY - cp27 PowerDNS Recursor on cp27 is OK: DNS OK: 0.079 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,2a02:c206:2162:6187::1,37.60.225.206,84.247.188.12 [20:51:42] RECOVERY - cp27 Nginx Backend for mw161 on cp27 is OK: TCP OK - 0.000 second response time on localhost port 8115 [20:51:44] RECOVERY - cp27 Disk Space on cp27 is OK: DISK OK - free space: / 18682MiB (20% inode=98%); [20:51:47] RECOVERY - cp27 Nginx Backend for mw171 on cp27 is OK: TCP OK - 0.000 second response time on localhost port 8117 [20:51:48] RECOVERY - cp27 Nginx Backend for mon181 on cp27 is OK: TCP OK - 0.000 second response time on localhost port 8201 [20:51:49] RECOVERY - cp27 Nginx Backend for mw181 on cp27 is OK: TCP OK - 0.000 second response time on localhost port 8119 [20:52:00] RECOVERY - cp27 Nginx Backend for mw162 on cp27 is OK: TCP OK - 0.000 second response time on localhost port 8116 [20:52:04] PROBLEM - cp27 Puppet on cp27 is UNKNOWN: NRPE: Unable to read output [20:52:07] RECOVERY - cp27 Nginx Backend for mw182 on cp27 is OK: TCP OK - 0.000 second response time on localhost port 8120 [20:52:08] RECOVERY - cp27 APT on cp27 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:52:08] RECOVERY - cp27 Nginx Backend for swiftproxy171 on cp27 is OK: TCP OK - 0.000 second response time on localhost port 8207 [20:52:09] RECOVERY - cp27 Nginx Backend for mw172 on cp27 is OK: TCP OK - 0.000 second response time on localhost port 8118 [20:52:11] PROBLEM - cp27 HTTPS on cp27 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 0.924 second response time [20:52:14] RECOVERY - cp27 Nginx Backend for matomo151 on cp27 is OK: TCP OK - 0.000 second response time on localhost port 8203 [20:54:11] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 47% [20:56:11] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 36% [20:57:36] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 18 backends are healthy [20:58:33] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.33, 6.64, 6.75 [21:00:17] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ecff5d2f491e...2b9c970704d4 [21:00:19] [02miraheze/mw-config] 07paladox 032b9c970 - Add cp27 to wgCdnServers [21:00:38] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [21:00:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:00:46] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [21:00:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:01:20] miraheze/mw-config - paladox the build passed. [21:05:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.17, 6.79, 6.43 [21:05:43] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.91, 6.81, 6.39 [21:06:21] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.09, 6.62, 6.23 [21:07:25] RECOVERY - cp27 conntrack_table_size on cp27 is OK: OK: nf_conntrack is 7 % full [21:07:39] RECOVERY - cp27 ferm_active on cp27 is OK: OK ferm input default policy is set [21:08:09] RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [21:08:21] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.34, 6.51, 6.25 [21:10:15] PROBLEM - cp27 HTTPS on cp27 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [21:10:58] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.79, 6.79, 6.54 [21:12:14] PROBLEM - cp27 HTTPS on cp27 is WARNING: HTTP WARNING: HTTP/2 404 - 3456 bytes in 0.907 second response time [21:12:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.34, 6.98, 6.87 [21:13:31] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.72, 6.78, 6.56 [21:15:14] [02ManageWiki] 07RhinosF1 opened pull request 03#443: ManageWikiDefaultPermissions: fix description i18n - 13https://github.com/miraheze/ManageWiki/pull/443 [21:15:49] [02ManageWiki] 07RhinosF1 synchronize pull request 03#443: ManageWikiDefaultPermissions: fix description i18n - 13https://github.com/miraheze/ManageWiki/pull/443 [21:18:12] miraheze/ManageWiki - RhinosF1 the build has errored. [21:18:45] miraheze/ManageWiki - RhinosF1 the build has errored. [21:23:14] !log [@test151] starting deploy of {'config': True} to test151 [21:23:15] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [21:23:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:23:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:24:14] [02ManageWiki] 07RhinosF1 commented on pull request 03#443: ManageWikiDefaultPermissions: fix description i18n - 13https://github.com/miraheze/ManageWiki/pull/443#issuecomment-1913339790 [21:32:41] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.90, 6.83, 6.50 [21:34:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.51, 6.96, 6.44 [21:34:38] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.32, 6.63, 6.47 [21:36:19] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.50, 6.33, 6.76 [21:36:21] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.13, 6.75, 6.40 [21:36:30] !log [macfan@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/updateSpecialPages.php (END - exit=2) [21:36:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:36:54] !log [macfan@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/updateSpecialPages.php (START) [21:36:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:38:21] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.73, 6.76, 6.45 [21:40:08] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.08, 6.48, 6.42 [21:45:54] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.95, 7.16, 6.71 [21:47:28] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.09, 20.56, 18.59 [21:49:23] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.42, 19.66, 18.51 [21:51:40] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.14, 7.43, 6.96 [21:52:31] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cd196aaf1560...3bba68e1d76e [21:52:34] [02miraheze/puppet] 07Universal-Omega 033bba68e - Lower to 10 [21:53:35] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.03, 7.37, 6.99 [21:54:49] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [21:57:07] [02ManageWiki] 07OAuthority commented on pull request 03#443: ManageWikiDefaultPermissions: fix description i18n - 13https://github.com/miraheze/ManageWiki/pull/443#issuecomment-1913346677 [21:57:56] [02ManageWiki] 07OAuthority deleted a comment on pull request 03#443: ManageWikiDefaultPermissions: fix description i18n - 13https://github.com/miraheze/ManageWiki/pull/443#issuecomment-1913346677 [21:59:27] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.36, 7.00, 6.59 [22:06:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.89, 7.02, 6.45 [22:07:15] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.53, 6.75, 6.67 [22:11:15] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.96, 7.39, 6.95 [22:14:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.18, 6.67, 6.57 [22:15:15] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.54, 6.41, 6.64 [22:16:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.88, 6.86, 6.62 [22:17:31] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.14, 7.39, 7.11 [22:18:26] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.72, 6.79, 6.63 [22:18:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.49, 7.78, 7.02 [22:18:51] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.94, 6.68, 6.51 [22:19:31] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.54, 7.50, 7.19 [22:20:49] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.35, 6.55, 6.49 [22:24:19] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.andreijiroh.uk.eu.org could not be found [22:24:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.39, 7.88, 7.36 [22:27:31] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.07, 6.09, 6.73 [22:32:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.13, 6.70, 6.55 [22:44:39] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.35, 6.80, 6.76 [22:48:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.03, 7.58, 7.26 [22:50:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.93, 7.77, 7.37 [22:58:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3bba68e1d76e...87e2eee5538e [22:58:03] [02miraheze/puppet] 07paladox 0387e2eee - squid: allow private network [23:00:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.37, 6.27, 6.78 [23:03:40] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_srv-ssl] [23:08:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.93, 7.10, 6.95 [23:09:25] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/c06e6c5a40b5 [23:09:26] [02miraheze/MirahezeMagic] 07paladox 03c06e6c5 - Fix "Implicit conversion from float 0.02 to int loses precision in" [23:09:30] [02MirahezeMagic] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/MirahezeMagic [23:09:32] [02MirahezeMagic] 07paladox opened pull request 03#456: Fix "Implicit conversion from float 0.02 to int loses precision in" - 13https://github.com/miraheze/MirahezeMagic/pull/456 [23:15:19] miraheze/MirahezeMagic - paladox the build passed. [23:16:14] !log [oa@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/oa/rss.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (START) [23:16:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:16:26] !log [oa@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/oa/rss.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (END - exit=2) [23:16:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.82, 6.33, 6.70 [23:29:40] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:30:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.04, 7.06, 6.79 [23:32:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.19, 7.13, 6.85 [23:35:00] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.02, 6.72, 6.38 [23:35:55] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.53, 7.82, 7.01 [23:36:57] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.63, 6.36, 6.29 [23:43:07] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.91, 7.98, 7.05 [23:44:48] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.97, 7.11, 6.56 [23:45:33] !log [macfan@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/updateSpecialPages.php (END - exit=0) [23:45:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:46:45] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.07, 7.47, 6.76 [23:48:42] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.59, 7.54, 6.88 [23:49:34] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.28, 7.92, 7.79