[00:00:02] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:00:31] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.02, 11.63, 10.96 [00:01:21] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.35, 13.07, 11.37 [00:01:25] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [00:01:50] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 12.13, 9.10, 8.14 [00:01:58] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.300 second response time [00:02:31] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 15.65, 12.78, 11.44 [00:02:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/148e97bd2771...e0f6d7454b25 [00:02:50] [02miraheze/puppet] 07paladox 03e0f6d74 - openldap: use ldap:// not ldaps:// [00:03:09] PROBLEM - cp26 Disk Space on cp26 is CRITICAL: DISK CRITICAL - free space: / 5287MiB (5% inode=98%); [00:03:10] RECOVERY - mw132 Disk Space on mw132 is OK: DISK OK - free space: / 2906MiB (12% inode=69%); [00:03:24] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.027 second response time [00:03:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/205d32bd8946 [00:03:31] [02miraheze/puppet] 07paladox 03205d32b - ldapvi: allow customising host [00:03:32] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [00:03:35] [02puppet] 07paladox opened pull request 03#3702: ldapvi: allow customising host - 13https://github.com/miraheze/puppet/pull/3702 [00:03:43] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.21, 9.86, 8.58 [00:03:44] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.07, 3.80, 3.11 [00:03:48] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 9.03, 8.86, 8.16 [00:03:56] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 17.61, 13.80, 12.15 [00:04:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/205d32bd8946...1922e54e2870 [00:04:49] [02miraheze/puppet] 07paladox 031922e54 - Update openldap.pp [00:04:51] [02puppet] 07paladox synchronize pull request 03#3702: ldapvi: allow customising host - 13https://github.com/miraheze/puppet/pull/3702 [00:04:56] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:05:09] RECOVERY - cp26 Disk Space on cp26 is OK: DISK OK - free space: / 14366MiB (15% inode=98%); [00:05:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/1922e54e2870...5fb8b519e8ea [00:05:34] [02miraheze/puppet] 07paladox 035fb8b51 - Create ldap171.yaml [00:05:35] [02puppet] 07paladox synchronize pull request 03#3702: ldapvi: allow customising host - 13https://github.com/miraheze/puppet/pull/3702 [00:05:40] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.10, 9.93, 8.72 [00:05:46] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 6.05, 4.50, 3.16 [00:05:46] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.27, 3.37, 3.02 [00:05:51] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:06:03] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:06:05] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:06:24] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/0b96ff7b0c9f...16cc6660a0a2 [00:06:25] [02miraheze/dns] 07paladox 0316cc666 - Add ldap171-private to dns [00:06:36] [02puppet] 07paladox closed pull request 03#3702: ldapvi: allow customising host - 13https://github.com/miraheze/puppet/pull/3702 [00:06:37] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [00:06:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/puppet/compare/e0f6d7454b25...91eb92bd2235 [00:06:40] [02miraheze/puppet] 07paladox 0391eb92b - ldapvi: allow customising host (#3702) [00:06:42] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [00:06:56] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.102 second response time [00:07:38] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.52, 9.11, 8.58 [00:07:48] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.721 second response time [00:07:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/91eb92bd2235...1ca3479facd6 [00:07:53] [02miraheze/puppet] 07paladox 031ca3479 - Fix [00:08:00] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.576 second response time [00:08:02] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.294 second response time [00:10:06] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:10:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1ca3479facd6...5f1cd0a8f582 [00:10:44] [02miraheze/puppet] 07paladox 035f1cd0a - Fix ldap monitoring [00:11:47] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.21, 3.42, 3.16 [00:12:01] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.695 second response time [00:14:09] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:14:09] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:15:43] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.26, 3.36, 3.19 [00:16:01] PROBLEM - ldap141 Puppet on ldap141 is CRITICAL: connect to address 2a10:6740::6:506 port 5666: Connection refusedconnect to host 2a10:6740::6:506 port 5666: Connection refused [00:16:08] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.634 second response time [00:16:10] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 4.495 second response time [00:19:08] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 6.93, 10.39, 11.73 [00:19:41] PROBLEM - ldap141 SSH on ldap141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:19:58] PROBLEM - ldap141 Current Load on ldap141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:20:16] PROBLEM - ping6 on ldap141 is CRITICAL: PING CRITICAL - Packet loss = 100% [00:20:41] PROBLEM - Host ldap141 is DOWN: PING CRITICAL - Packet loss = 100% [00:21:05] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.29, 11.09, 11.94 [00:21:51] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:22:59] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.88, 11.59, 11.90 [00:23:09] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.09, 11.70, 12.04 [00:23:29] PROBLEM - db112 MariaDB Connections on db112 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db112.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connection [00:23:29] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db112.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [00:23:32] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:23:42] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.97, 3.73, 3.37 [00:23:49] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.509 second response time [00:24:07] PROBLEM - ping6 on db112 is CRITICAL: PING CRITICAL - Packet loss = 100% [00:24:15] PROBLEM - db112 SSH on db112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:30] PROBLEM - db112 PowerDNS Recursor on db112 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:24:35] PROBLEM - db112 Backups SQL icingaweb2 on db112 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:25:20] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:25:22] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:25:33] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 3 backends are down. mw132 mw133 mw134 [00:25:41] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.62, 3.40, 3.30 [00:25:43] PROBLEM - mw134 MediaWiki Rendering on mw134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:25:47] PROBLEM - Host db112 is DOWN: PING CRITICAL - Packet loss = 100% [00:26:03] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 4 backends are down. mw131 mw132 mw133 mw134 [00:26:14] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:26:32] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:26:37] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 14.61, 11.07, 8.91 [00:26:57] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:27:04] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [00:27:20] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.612 second response time [00:27:23] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 13.60, 11.27, 9.30 [00:27:23] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw132 mw133 [00:27:24] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 6.323 second response time [00:27:31] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw133 [00:27:41] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.972 second response time [00:28:11] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.440 second response time [00:28:14] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:28:36] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 7.239 second response time [00:28:55] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.392 second response time [00:29:02] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.504 second response time [00:29:23] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.27, 11.07, 9.51 [00:29:38] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.86, 3.49, 3.39 [00:29:48] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:30:11] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:30:55] PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized [00:30:59] PROBLEM - swiftproxy171 HTTP on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 401 Unauthorized [00:31:01] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.18, 6.78, 6.28 [00:31:18] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [00:31:30] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:31:31] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 22 backends are healthy [00:31:34] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 14.48, 12.52, 9.74 [00:31:36] PROBLEM - swiftproxy161 HTTPS on swiftproxy161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized [00:31:38] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.34, 3.95, 3.55 [00:31:39] PROBLEM - swiftproxy161 HTTP on swiftproxy161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 401 Unauthorized [00:31:47] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.564 second response time [00:31:54] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 26.75, 15.35, 11.19 [00:32:01] RECOVERY - mw134 MediaWiki Rendering on mw134 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.884 second response time [00:32:14] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.785 second response time [00:32:21] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.205 second response time [00:32:34] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized [00:32:41] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 401 Unauthorized [00:33:00] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:33:00] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.72, 6.36, 6.18 [00:33:01] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:33:28] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [00:33:29] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.990 second response time [00:33:54] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:34:11] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 401 Unauthorized [00:34:14] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized [00:34:58] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.431 second response time [00:35:00] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.747 second response time [00:35:28] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [00:35:29] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 3.523 second response time [00:35:34] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 7.98, 11.83, 10.25 [00:35:54] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [00:35:58] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.375 second response time [00:36:13] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 41% [00:36:14] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:36:21] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [00:36:27] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:36:40] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:36:50] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:37:05] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:37:09] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:37:12] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:37:34] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 14.44, 12.55, 10.68 [00:37:36] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [00:38:10] RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 25% [00:38:32] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 7.868 second response time [00:38:41] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 3.224 second response time [00:38:46] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw133 [00:39:03] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.140 second response time [00:39:04] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw133 [00:39:12] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 4.158 second response time [00:39:13] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.582 second response time [00:39:34] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.91, 11.61, 10.52 [00:39:34] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.725 second response time [00:40:12] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw141 [00:40:16] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw141 mw143 [00:40:17] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 2.186 second response time [00:40:21] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.596 second response time [00:41:09] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.829 second response time [00:41:09] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:41:11] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:42:30] PROBLEM - swiftproxy161 NTP time on swiftproxy161 is WARNING: NTP WARNING: Offset -0.1028498113 secs [00:42:39] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:42:45] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:43:03] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 22 backends are healthy [00:43:14] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 6.439 second response time [00:43:28] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 3 backends are down. mw131 mw134 mw143 [00:43:43] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 2 backends are down. mw131 mw134 [00:44:16] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [00:44:30] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 22 backends are healthy [00:44:33] RECOVERY - swiftproxy161 NTP time on swiftproxy161 is OK: NTP OK: Offset -0.09811499715 secs [00:44:38] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.673 second response time [00:44:42] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.569 second response time [00:45:13] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.614 second response time [00:45:28] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [00:45:34] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.35, 9.46, 10.13 [00:45:42] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [00:45:50] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.25, 10.38, 11.11 [00:45:52] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:45:59] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.49, 11.30, 11.35 [00:46:04] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [00:46:19] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:46:21] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.373 second response time [00:46:33] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:46:47] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:46:49] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:47:02] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.463 second response time [00:47:05] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:47:10] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.471 second response time [00:47:10] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3362 bytes in 0.509 second response time [00:47:50] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.663 second response time [00:48:14] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw133 [00:48:16] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.392 second response time [00:48:21] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw133 [00:48:24] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 4.794 second response time [00:48:31] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.559 second response time [00:48:46] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.523 second response time [00:48:47] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.572 second response time [00:49:00] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.743 second response time [00:49:05] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.696 second response time [00:49:07] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.581 second response time [00:49:08] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.336 second response time [00:49:43] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.81, 3.47, 3.93 [00:49:48] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 5.47, 8.16, 10.08 [00:49:51] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 5.01, 7.56, 9.77 [00:50:14] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [00:50:16] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 22 backends are healthy [00:51:15] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:53:13] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.294 second response time [00:54:15] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:54:29] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:54:33] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:55:08] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:55:39] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.85, 2.53, 3.39 [00:56:14] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 3.202 second response time [00:56:31] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.783 second response time [00:56:56] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 3 backends are down. mw131 mw132 mw133 [00:57:05] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [00:57:23] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:57:23] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:57:26] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:57:28] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 5 backends are down. mw131 mw132 mw141 mw133 mw134 [00:57:34] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 16.47, 10.55, 9.72 [00:57:34] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove [+0/-0/±1] 13https://github.com/miraheze/dns/commit/092ba699516a [00:57:35] [02miraheze/dns] 07Universal-Omega 03092ba69 - Remove migrated servers [00:57:36] [02dns] 07Universal-Omega created branch 03remove - 13https://github.com/miraheze/dns [00:57:40] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [00:57:40] [02dns] 07Universal-Omega opened pull request 03#470: Remove migrated servers - 13https://github.com/miraheze/dns/pull/470 [00:57:43] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 18.99, 12.00, 10.57 [00:57:48] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:57:48] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:57:56] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [00:57:59] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [00:58:12] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 5 backends are down. mw131 mw141 mw142 mw134 mw143 [00:58:18] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 3 backends are down. mw142 mw134 mw143 [00:58:43] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 7.924 second response time [00:58:54] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [00:59:00] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove [+0/-0/±1] 13https://github.com/miraheze/dns/compare/092ba699516a...53eebb41384e [00:59:01] [02miraheze/dns] 07Universal-Omega 0353eebb4 - Remove [00:59:03] [02dns] 07Universal-Omega synchronize pull request 03#470: Remove migrated servers - 13https://github.com/miraheze/dns/pull/470 [00:59:10] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.624 second response time [00:59:22] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.483 second response time [00:59:22] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 2.162 second response time [00:59:26] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.377 second response time [00:59:27] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [00:59:34] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.00, 10.03, 9.61 [00:59:38] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.702 second response time [00:59:39] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.74, 3.47, 3.59 [00:59:40] [02puppet] 07Universal-Omega closed pull request 03#3651: Remove test131 - 13https://github.com/miraheze/puppet/pull/3651 [00:59:42] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.76, 9.80, 9.93 [00:59:44] [02miraheze/puppet] 07Universal-Omega deleted branch 03remove-test131 [00:59:46] [02puppet] 07Universal-Omega deleted branch 03remove-test131 - 13https://github.com/miraheze/puppet [00:59:46] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.492 second response time [00:59:46] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.615 second response time [00:59:55] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 22 backends are healthy [00:59:56] [02puppet] 07Universal-Omega closed pull request 03#3698: varnish: update bastion hosts - 13https://github.com/miraheze/puppet/pull/3698 [00:59:57] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5f1cd0a8f582...47020456403c [00:59:58] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 22 backends are healthy [00:59:59] [02miraheze/puppet] 07Universal-Omega 034702045 - varnish: update bastion hosts (#3698) [01:00:02] [02miraheze/puppet] 07Universal-Omega deleted branch 03varnish-bast [01:00:04] [02puppet] 07Universal-Omega deleted branch 03varnish-bast - 13https://github.com/miraheze/puppet [01:00:12] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [01:00:16] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [01:00:23] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove [+0/-1/±0] 13https://github.com/miraheze/puppet/commit/5e50b9a9b6fc [01:00:26] [02miraheze/puppet] 07Universal-Omega 035e50b9a - Delete hieradata/hosts/bast121.yaml [01:00:29] [02puppet] 07Universal-Omega created branch 03remove - 13https://github.com/miraheze/puppet [01:00:36] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:00:44] [02puppet] 07Universal-Omega opened pull request 03#3703: Remove migrated servers - 13https://github.com/miraheze/puppet/pull/3703 [01:01:02] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [01:01:16] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/5e50b9a9b6fc...b4bbf1b67bd9 [01:01:19] [02miraheze/puppet] 07Universal-Omega 03b4bbf1b - Delete hieradata/hosts/bast141.yaml [01:01:21] [02puppet] 07Universal-Omega synchronize pull request 03#3703: Remove migrated servers - 13https://github.com/miraheze/puppet/pull/3703 [01:01:34] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.13, 8.86, 9.23 [01:02:37] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.483 second response time [01:02:52] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/b4bbf1b67bd9...4cc9a2e112b9 [01:02:54] [02miraheze/puppet] 07Universal-Omega 034cc9a2e - Delete hieradata/hosts/cp24.yaml [01:02:57] [02puppet] 07Universal-Omega synchronize pull request 03#3703: Remove migrated servers - 13https://github.com/miraheze/puppet/pull/3703 [01:03:09] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/4cc9a2e112b9...01e6a581d605 [01:03:12] [02miraheze/puppet] 07Universal-Omega 0301e6a58 - Delete hieradata/hosts/cp25.yaml [01:03:14] [02puppet] 07Universal-Omega synchronize pull request 03#3703: Remove migrated servers - 13https://github.com/miraheze/puppet/pull/3703 [01:03:27] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/01e6a581d605...7d8a82cf9745 [01:03:30] [02miraheze/puppet] 07Universal-Omega 037d8a82c - Delete hieradata/hosts/db112.yaml [01:03:31] [02puppet] 07Universal-Omega synchronize pull request 03#3703: Remove migrated servers - 13https://github.com/miraheze/puppet/pull/3703 [01:03:37] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.85, 3.00, 3.35 [01:03:52] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/7d8a82cf9745...c098634da9df [01:03:53] [02miraheze/puppet] 07Universal-Omega 03c098634 - Delete hieradata/hosts/test131.yaml [01:03:55] [02puppet] 07Universal-Omega synchronize pull request 03#3703: Remove migrated servers - 13https://github.com/miraheze/puppet/pull/3703 [01:04:10] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/c098634da9df...433807d8b830 [01:04:12] [02miraheze/puppet] 07Universal-Omega 03433807d - Delete hieradata/hosts/matomo121.yaml [01:04:15] [02puppet] 07Universal-Omega synchronize pull request 03#3703: Remove migrated servers - 13https://github.com/miraheze/puppet/pull/3703 [01:04:36] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/433807d8b830...f10997756ce0 [01:04:38] [02miraheze/puppet] 07Universal-Omega 03f109977 - Delete hieradata/hosts/phab121.yaml [01:04:39] [02puppet] 07Universal-Omega synchronize pull request 03#3703: Remove migrated servers - 13https://github.com/miraheze/puppet/pull/3703 [01:05:10] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/f10997756ce0...cd93041ac3df [01:05:12] [02miraheze/puppet] 07Universal-Omega 03cd93041 - Delete hieradata/hosts/reports121.yaml [01:05:14] [02puppet] 07Universal-Omega synchronize pull request 03#3703: Remove migrated servers - 13https://github.com/miraheze/puppet/pull/3703 [01:05:18] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:36] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.81, 3.99, 3.67 [01:09:35] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.94, 3.91, 3.68 [01:11:36] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.23, 3.95, 3.71 [01:12:04] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 6.16, 9.25, 11.93 [01:13:13] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.97, 9.28, 11.86 [01:13:34] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.90, 3.62, 3.63 [01:13:56] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:14:35] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cd93041ac3df...b529125be48b [01:14:38] [02miraheze/puppet] 07Universal-Omega 03b529125 - Update site.pp [01:14:40] [02puppet] 07Universal-Omega synchronize pull request 03#3703: Remove migrated servers - 13https://github.com/miraheze/puppet/pull/3703 [01:15:26] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.04, 9.56, 11.70 [01:15:55] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.124 second response time [01:15:57] [02dns] 07Universal-Omega synchronize pull request 03#470: Remove migrated servers - 13https://github.com/miraheze/dns/pull/470 [01:15:58] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove [+0/-0/±1] 13https://github.com/miraheze/dns/compare/53eebb41384e...14f1999502d6 [01:15:59] [02miraheze/dns] 07Universal-Omega 0314f1999 - - [01:16:08] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 6.90, 8.95, 11.85 [01:18:04] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.92, 10.51, 11.44 [01:19:35] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.29, 3.73, 3.64 [01:20:04] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.51, 9.59, 10.96 [01:21:39] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.42, 3.63, 3.61 [01:23:26] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 5.48, 7.42, 9.90 [01:23:41] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.39, 4.03, 3.77 [01:24:15] !log [dbcopy@db131] Began to dump db131 to db171 [01:24:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:25:14] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.21, 9.75, 10.66 [01:25:15] !log [agent@db131] Began to dump db131 to db171 [01:25:29] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [01:25:35] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:25:56] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 16.99, 10.90, 11.04 [01:26:00] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:26:04] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.06, 10.57, 10.86 [01:26:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:52] !log [agent@db131] Began to dump db131 to db171 [01:27:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:15] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.31, 10.52, 10.90 [01:27:26] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.75, 11.50, 11.03 [01:27:29] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 2.364 second response time [01:27:32] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.165 second response time [01:27:59] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.681 second response time [01:28:04] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.19, 10.89, 10.98 [01:28:21] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.53, 9.95, 8.41 [01:29:01] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.875 second response time [01:29:15] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 17.24, 12.88, 11.71 [01:29:36] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [01:29:38] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.99, 3.73, 3.76 [01:29:47] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:29:51] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10000 milliseconds [01:30:04] PROBLEM - cp25 Puppet on cp25 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:30:11] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 20.17, 13.78, 11.99 [01:30:16] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:30:17] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.49, 9.71, 8.45 [01:30:27] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 2.204 second response time [01:30:39] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:30:53] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3362 bytes in 0.510 second response time [01:31:03] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:31:14] !log [agent@db131] MySQL dump to db171 halted [01:31:20] !log [agent@db131] Began to dump db131 to db171 [01:31:40] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 6.170 second response time [01:31:42] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.04, 4.50, 4.03 [01:31:44] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw131 mw142 [01:31:45] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.582 second response time [01:31:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:31:53] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw131 [01:31:53] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.680 second response time [01:31:55] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw131 [01:31:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:32:13] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.80, 9.10, 8.44 [01:32:14] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.545 second response time [01:32:24] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.173 second response time [01:32:25] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw141 [01:32:37] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.516 second response time [01:32:51] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.463 second response time [01:33:06] PROBLEM - bast141 Puppet on bast141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:33:06] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.553 second response time [01:33:26] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.05, 11.95, 11.51 [01:33:42] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [01:33:51] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [01:33:55] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 22 backends are healthy [01:34:20] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 22 backends are healthy [01:35:31] !log [agent@db142] Began to dump db142 to db181 [01:35:42] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.06, 3.80, 3.86 [01:35:58] !log [agent@db142] Began to dump db142 to db181 [01:36:02] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:36:18] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:36:28] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [01:36:30] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:36:47] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3362 bytes in 0.509 second response time [01:36:51] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.470 second response time [01:37:26] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.02, 12.66, 11.89 [01:37:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:41] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.57, 3.96, 3.90 [01:37:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:38:02] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.533 second response time [01:38:15] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.548 second response time [01:38:25] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.739 second response time [01:38:28] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.620 second response time [01:38:45] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.462 second response time [01:38:51] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.177 second response time [01:39:27] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [01:39:42] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 2.960 second response time [01:39:43] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [01:40:23] PROBLEM - cp24 Puppet on cp24 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:40:41] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [01:41:28] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 5.521 second response time [01:41:39] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.64, 3.68, 3.85 [01:41:42] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.136 second response time [01:41:44] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 3.302 second response time [01:42:45] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:44:04] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.84, 10.85, 11.68 [01:44:38] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [01:45:26] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 7.52, 10.96, 11.80 [01:45:37] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.24, 3.51, 3.70 [01:47:35] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.58, 3.23, 3.59 [01:48:04] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 15.88, 12.49, 12.09 [01:48:47] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:48:58] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:49:47] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:50:13] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [01:50:45] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.297 second response time [01:50:57] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.532 second response time [01:51:02] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw132 mw133 [01:51:26] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 15.08, 11.90, 11.79 [01:51:45] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.622 second response time [01:52:15] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.574 second response time [01:52:59] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [01:53:26] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 7.48, 10.18, 11.17 [01:56:12] !log [agent@db121] Dumping db121 to db161 [01:56:26] [02puppet] 07Universal-Omega closed pull request 03#3703: Remove migrated servers - 13https://github.com/miraheze/puppet/pull/3703 [01:56:27] [02miraheze/puppet] 07Universal-Omega deleted branch 03remove [01:56:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:56:29] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-9/±1] 13https://github.com/miraheze/puppet/compare/47020456403c...8b6d8f9060e9 [01:56:31] [02miraheze/puppet] 07Universal-Omega 038b6d8f9 - Remove migrated servers (#3703) [01:56:33] [02puppet] 07Universal-Omega deleted branch 03remove - 13https://github.com/miraheze/puppet [01:56:48] PROBLEM - ping6 on bast141 is CRITICAL: PING CRITICAL - Packet loss = 100% [01:56:49] [02dns] 07Universal-Omega closed pull request 03#470: Remove migrated servers - 13https://github.com/miraheze/dns/pull/470 [01:56:51] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/16cc6660a0a2...476ea5321b11 [01:56:53] [02miraheze/dns] 07Universal-Omega 03476ea53 - Remove migrated servers (#470) [01:56:56] [02miraheze/dns] 07Universal-Omega deleted branch 03remove [01:56:57] [02dns] 07Universal-Omega deleted branch 03remove - 13https://github.com/miraheze/dns [01:57:18] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.98, 10.40, 11.71 [01:57:20] PROBLEM - bast141 SSH on bast141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:57:29] PROBLEM - ping6 on bast121 is CRITICAL: PING CRITICAL - Packet loss = 100% [01:57:35] PROBLEM - bast121 SSH on bast121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:57:44] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:57:44] PROBLEM - Host bast121 is DOWN: PING CRITICAL - Packet loss = 100% [01:57:50] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.79, 4.19, 3.73 [01:57:51] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [01:58:02] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:58:03] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3392 bytes in 0.589 second response time [01:58:08] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 6.46, 10.31, 11.58 [01:58:15] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.594 second response time [01:58:36] PROBLEM - Host bast141 is DOWN: PING CRITICAL - Packet loss = 100% [01:58:51] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.470 second response time [01:58:54] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:59:28] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw133 mw134 [01:59:29] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 2 backends are down. mw133 mw134 [01:59:31] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw133 mw134 [01:59:40] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:40] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 6.43, 9.17, 11.47 [01:59:44] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.032 second response time [01:59:55] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw134 [01:59:56] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 7.357 second response time [02:00:04] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw142 [02:00:17] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.612 second response time [02:00:29] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:00:44] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:00:53] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [02:01:03] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.394 second response time [02:01:20] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 6 backends are down. mw131 mw132 mw141 mw133 mw134 mw143 [02:01:23] !log [agent@db121] Dumping db121 to db161 [02:01:26] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 2.88, 6.87, 9.47 [02:02:08] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.535 second response time [02:02:15] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 5.305 second response time [02:02:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:02:29] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.540 second response time [02:02:43] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.321 second response time [02:02:50] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 2.290 second response time [02:02:50] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [02:02:52] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.739 second response time [02:03:01] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.135 second response time [02:03:19] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 22 backends are healthy [02:03:25] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [02:03:29] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [02:03:31] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 22 backends are healthy [02:03:53] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.34, 7.92, 9.97 [02:03:55] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [02:04:02] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:04:06] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.843 second response time [02:05:01] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:05:04] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 19.33, 12.28, 11.02 [02:05:05] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:05:21] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 17.18, 12.24, 11.50 [02:05:40] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 16.88, 11.55, 11.43 [02:06:00] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.669 second response time [02:06:23] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:06:45] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 1.290 second response time [02:06:56] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:07:01] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:07:09] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:07:19] !log [agent@db101] Dumping db101 to db151 [02:07:47] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw134 [02:07:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:07:57] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 18.60, 13.23, 11.65 [02:08:20] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.583 second response time [02:08:52] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.218 second response time [02:09:21] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [02:09:36] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:09:51] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:10:16] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:10:38] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.45, 12.53, 10.14 [02:10:48] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:10:51] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [02:11:15] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.506 second response time [02:11:15] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.614 second response time [02:11:21] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.296 second response time [02:11:47] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw131 mw133 [02:11:51] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 2.991 second response time [02:12:13] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw141 [02:12:21] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw141 [02:12:35] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw132 mw141 [02:12:36] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.66, 11.46, 10.02 [02:12:53] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 4.833 second response time [02:13:14] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.91, 10.34, 8.87 [02:13:16] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 14.50, 13.51, 10.71 [02:13:46] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 2.467 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [02:14:13] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [02:14:17] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [02:14:25] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 2.621 second response time [02:14:35] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [02:14:35] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 14.90, 12.73, 10.65 [02:14:51] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 1.724 second response time [02:15:14] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.53, 9.89, 8.93 [02:15:14] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.87, 11.48, 10.29 [02:15:31] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:17:24] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:17:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:17:56] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [02:18:06] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:18:11] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 5 backends are down. mw131 mw132 mw141 mw133 mw134 [02:18:18] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:18:24] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3392 bytes in 0.449 second response time [02:18:28] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.372 second response time [02:18:56] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:19:11] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.57, 11.94, 10.73 [02:19:22] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.660 second response time [02:19:24] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.479 second response time [02:19:26] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.23, 3.61, 3.94 [02:19:40] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.316 second response time [02:19:55] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.508 second response time [02:20:03] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.540 second response time [02:20:07] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [02:20:16] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.820 second response time [02:20:22] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.575 second response time [02:20:27] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.505 second response time [02:20:32] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 6.77, 10.16, 10.24 [02:20:53] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.282 second response time [02:21:09] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.34, 10.01, 10.18 [02:21:22] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 22 backends are healthy [02:21:25] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [02:21:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:21:30] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:22:31] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 8.29, 9.51, 9.99 [02:23:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.523 second response time [02:23:34] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.698 second response time [02:24:24] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:24:30] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [02:26:29] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.745 second response time [02:26:32] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3392 bytes in 1.193 second response time [02:26:40] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:26:56] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 5 backends are down. mw131 mw141 mw133 mw134 mw143 [02:27:07] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 5 backends are down. mw131 mw141 mw133 mw134 mw143 [02:27:09] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 5 backends are down. mw131 mw141 mw133 mw134 mw143 [02:27:17] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.91, 3.82, 3.76 [02:27:18] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 5 backends are down. mw131 mw141 mw133 mw134 mw143 [02:27:33] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 6 backends are down. mw131 mw132 mw141 mw142 mw134 mw143 [02:27:56] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 10.44, 10.30, 9.48 [02:27:57] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:28:05] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.73, 11.57, 10.74 [02:28:34] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 7.014 second response time [02:28:37] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 6.397 second response time [02:28:38] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 14.50, 13.66, 11.49 [02:28:43] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 6.014 second response time [02:29:15] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [02:29:33] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [02:29:51] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 12.82, 11.10, 9.85 [02:29:55] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.545 second response time [02:30:05] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 15.86, 13.52, 11.57 [02:31:46] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.10, 10.48, 9.78 [02:32:04] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.07, 11.40, 11.01 [02:32:37] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:32:40] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [02:32:46] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:32:53] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:33:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:33:32] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 3 backends are down. mw131 mw132 mw134 [02:33:43] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 18.04, 13.60, 11.03 [02:34:05] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 14.95, 13.11, 11.69 [02:34:05] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:34:33] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.484 second response time [02:34:42] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:35:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:35:11] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:35:11] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:35:16] PROBLEM - os131 PowerDNS Recursor on os131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:35:19] PROBLEM - mw134 MediaWiki Rendering on mw134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:36:02] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:36:05] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.06, 11.80, 11.39 [02:36:08] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 4 backends are down. mw141 mw142 mw133 mw134 [02:36:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03infra [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/a2570512c4bb [02:36:12] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 8.537 second response time [02:36:13] [02miraheze/mw-config] 07Universal-Omega 03a257051 - Support using new infra based on hostname [02:36:16] [02mw-config] 07Universal-Omega created branch 03infra - 13https://github.com/miraheze/mw-config [02:36:19] [02mw-config] 07Universal-Omega opened pull request 03#5444: Support using new infra based on hostname - 13https://github.com/miraheze/mw-config/pull/5444 [02:36:38] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.330 second response time [02:36:38] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.319 second response time [02:36:48] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 2.902 second response time [02:36:52] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03infra [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a2570512c4bb...da103b100c65 [02:36:53] [02miraheze/mw-config] 07github-actions 03da103b1 - CI: lint code to MediaWiki standards [02:36:53] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 4.883 second response time [02:36:54] [02mw-config] 07github-actions[bot] synchronize pull request 03#5444: Support using new infra based on hostname - 13https://github.com/miraheze/mw-config/pull/5444 [02:37:06] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.662 second response time [02:37:10] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.158 second response time [02:37:12] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.01, 3.75, 3.92 [02:37:15] RECOVERY - mw134 MediaWiki Rendering on mw134 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.913 second response time [02:37:21] miraheze/mw-config - Universal-Omega the build passed. [02:37:24] RECOVERY - os131 PowerDNS Recursor on os131 is OK: DNS OK: 1.641 second response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [02:37:30] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.156 second response time [02:37:58] !log [agent@db101] Finished dumping db101 to db151 [02:38:01] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.690 second response time [02:38:05] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 13.09, 12.12, 11.54 [02:38:05] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [02:38:29] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:38:43] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:38:46] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [02:38:54] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [02:38:59] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:39:10] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.79, 4.41, 4.13 [02:39:10] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:39:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:40:05] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.10, 11.87, 11.57 [02:40:28] PROBLEM - graylog131 PowerDNS Recursor on graylog131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:40:31] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 4.606 second response time [02:40:43] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 5.336 second response time [02:40:46] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.830 second response time [02:40:51] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.649 second response time [02:40:56] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.537 second response time [02:41:06] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.69, 3.82, 3.95 [02:41:09] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 4.451 second response time [02:41:24] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 7.56, 11.29, 11.03 [02:41:54] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:42:02] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw132 [02:42:28] RECOVERY - graylog131 PowerDNS Recursor on graylog131 is OK: DNS OK: 0.322 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [02:42:58] Huh [02:43:02] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [02:43:23] oh, broken pipe [02:43:26] \:/ [02:43:31] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [02:43:51] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.534 second response time [02:44:19] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [02:44:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03infra [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/da103b100c65...8c24c176947f [02:44:26] [02miraheze/mw-config] 07Universal-Omega 038c24c17 - Support [02:44:29] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 7.10, 10.07, 11.33 [02:44:29] [02mw-config] 07Universal-Omega synchronize pull request 03#5444: Support using new infra based on hostname - 13https://github.com/miraheze/mw-config/pull/5444 [02:44:34] !log [agent@db101] Dumping db101 to db151 [02:44:43] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 22 backends are healthy [02:44:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:44:53] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [02:45:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:45:14] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.07, 8.28, 9.88 [02:45:18] miraheze/mw-config - Universal-Omega the build passed. [02:45:23] PROBLEM - os131 PowerDNS Recursor on os131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:45:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03infra [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8c24c176947f...ffd5af296b43 [02:45:26] [02miraheze/mw-config] 07Universal-Omega 03ffd5af2 - Support no SSL [02:45:27] [02mw-config] 07Universal-Omega synchronize pull request 03#5444: Support using new infra based on hostname - 13https://github.com/miraheze/mw-config/pull/5444 [02:45:27] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [02:45:34] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.47, 6.35, 5.96 [02:45:56] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [02:46:04] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 3.18, 6.80, 9.46 [02:46:18] miraheze/mw-config - Universal-Omega the build passed. [02:46:37] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [02:46:38] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 7.295 second response time [02:46:51] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.496 second response time [02:46:51] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [02:47:24] RECOVERY - os131 PowerDNS Recursor on os131 is OK: DNS OK: 0.954 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [02:47:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:47:31] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 3 backends are down. mw141 mw142 mw134 [02:47:34] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.43, 6.31, 6.00 [02:47:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03infra [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ffd5af296b43...e604c3730d26 [02:47:46] [02miraheze/mw-config] 07Universal-Omega 03e604c37 - Support [02:47:48] [02mw-config] 07Universal-Omega synchronize pull request 03#5444: Support using new infra based on hostname - 13https://github.com/miraheze/mw-config/pull/5444 [02:47:56] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3392 bytes in 0.469 second response time [02:48:13] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 6 backends are down. mw131 mw141 mw142 mw133 mw134 mw143 [02:48:29] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:48:37] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 4.160 second response time [02:48:40] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 6.738 second response time [02:48:43] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw132 mw133 [02:48:44] miraheze/mw-config - Universal-Omega the build passed. [02:48:56] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.65, 3.69, 3.67 [02:48:56] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:48:58] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [02:49:00] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw132 mw133 [02:49:13] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 16.42, 12.20, 11.06 [02:49:32] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 7.705 second response time [02:49:33] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.549 second response time [02:50:04] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.41, 10.39, 10.40 [02:50:05] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 6.185 second response time [02:50:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03infra [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e604c3730d26...d53e3e0a97f0 [02:50:12] [02miraheze/mw-config] 07Universal-Omega 03d53e3e0 - Support [02:50:13] [02mw-config] 07Universal-Omega synchronize pull request 03#5444: Support using new infra based on hostname - 13https://github.com/miraheze/mw-config/pull/5444 [02:50:42] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:50:53] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.16, 3.56, 3.63 [02:50:53] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.663 second response time [02:50:53] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.283 second response time [02:50:57] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.616 second response time [02:51:03] miraheze/mw-config - Universal-Omega the build passed. [02:51:13] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.91, 11.28, 10.88 [02:51:30] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [02:51:34] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.81, 6.65, 6.21 [02:51:46] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [02:52:02] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.660 second response time [02:52:05] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.40, 9.12, 9.92 [02:52:05] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [02:52:33] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.545 second response time [02:52:38] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.139 second response time [02:52:43] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 22 backends are healthy [02:52:53] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.55, 4.13, 3.82 [02:53:15] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.29, 11.20, 10.85 [02:53:22] [02mw-config] 07Universal-Omega synchronize pull request 03#5444: Support using new infra based on hostname - 13https://github.com/miraheze/mw-config/pull/5444 [02:53:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03infra [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d53e3e0a97f0...35b70b337bbf [02:53:27] [02miraheze/mw-config] 07Universal-Omega 0335b70b3 - Fix [02:53:32] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [02:53:34] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.70, 6.36, 6.16 [02:54:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03infra [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/35b70b337bbf...dc97f23c756b [02:54:04] [02miraheze/mw-config] 07Universal-Omega 03dc97f23 - Fix [02:54:06] [02mw-config] 07Universal-Omega synchronize pull request 03#5444: Support using new infra based on hostname - 13https://github.com/miraheze/mw-config/pull/5444 [02:54:14] miraheze/mw-config - Universal-Omega the build passed. [02:54:23] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 14.45, 11.43, 11.15 [02:54:52] miraheze/mw-config - Universal-Omega the build passed. [02:54:55] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03infra [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/dc97f23c756b...d8517a239746 [02:54:57] [02miraheze/mw-config] 07Universal-Omega 03d8517a2 - Fix [02:54:58] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [02:54:59] [02mw-config] 07Universal-Omega synchronize pull request 03#5444: Support using new infra based on hostname - 13https://github.com/miraheze/mw-config/pull/5444 [02:55:13] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 8.95, 11.03, 10.89 [02:55:28] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.133 second response time [02:55:53] miraheze/mw-config - Universal-Omega the build passed. [02:56:04] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.32, 11.77, 11.07 [02:56:19] [02mw-config] 07Universal-Omega synchronize pull request 03#5444: Support using new infra based on hostname - 13https://github.com/miraheze/mw-config/pull/5444 [02:56:22] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 6.35, 9.37, 10.43 [02:56:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03infra [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d8517a239746...2af2416804ad [02:56:23] [02miraheze/mw-config] 07Universal-Omega 032af2416 - Make new infra read-only [02:57:13] miraheze/mw-config - Universal-Omega the build passed. [02:58:21] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.57, 8.89, 10.15 [02:59:13] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.74, 9.15, 10.16 [03:00:20] RECOVERY - db151 Backups SQL on db151 is OK: FILE_AGE OK: /var/log/sql-backup.log is 19 seconds old and 92 bytes [03:00:46] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.63, 3.89, 3.89 [03:01:50] RECOVERY - db101 Backups SQL on db101 is OK: FILE_AGE OK: /var/log/sql-backup.log is 109 seconds old and 0 bytes [03:03:22] [02mw-config] 07Universal-Omega synchronize pull request 03#5444: Support using new infra based on hostname - 13https://github.com/miraheze/mw-config/pull/5444 [03:03:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03infra [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2af2416804ad...90f377ad12ec [03:03:26] [02miraheze/mw-config] 07Universal-Omega 0390f377a - Add mwtask181 to scsvg [03:04:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 6.91, 9.81, 11.70 [03:04:05] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 5.74, 7.96, 9.63 [03:04:17] miraheze/mw-config - Universal-Omega the build passed. [03:05:08] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:05:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:06:40] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.93, 4.57, 4.11 [03:07:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.765 second response time [03:07:31] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.40, 9.30, 11.60 [03:08:25] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 7.62, 9.21, 11.62 [03:09:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.27, 9.68, 11.82 [03:10:02] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.00, 7.71, 10.17 [03:10:05] [02miraheze/ErrorPages] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/styfle/cancel-workflow-action-0.12.1 [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/commit/b40195eba0d9 [03:10:06] [02miraheze/ErrorPages] 07dependabot[bot] 03b40195e - Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 [03:10:08] [02ErrorPages] 07dependabot[bot] created branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.12.1 - 13https://github.com/miraheze/ErrorPages [03:10:09] [02ErrorPages] 07dependabot[bot] opened pull request 03#40: Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/ErrorPages/pull/40 [03:10:10] [02ErrorPages] 07dependabot[bot] labeled pull request 03#40: Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/ErrorPages/pull/40 [03:10:55] miraheze/ErrorPages - dependabot[bot] the build passed. [03:11:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.77, 10.89, 12.01 [03:11:17] [02mw-config] 07Universal-Omega synchronize pull request 03#5444: Support using new infra based on hostname - 13https://github.com/miraheze/mw-config/pull/5444 [03:11:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03infra [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/90f377ad12ec...acd8ed20f2ba [03:11:20] [02miraheze/mw-config] 07Universal-Omega 03acd8ed2 - - [03:12:12] miraheze/mw-config - Universal-Omega the build passed. [03:12:38] [02mw-config] 07Universal-Omega synchronize pull request 03#5444: Support using new infra based on hostname - 13https://github.com/miraheze/mw-config/pull/5444 [03:12:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03infra [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/acd8ed20f2ba...337a87bf0cdc [03:12:42] [02miraheze/mw-config] 07Universal-Omega 03337a87b - + [03:13:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 7.68, 9.54, 11.37 [03:13:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [03:13:39] [02mw-config] 07Universal-Omega closed pull request 03#5444: Support using new infra based on hostname - 13https://github.com/miraheze/mw-config/pull/5444 [03:13:41] [02miraheze/mw-config] 07Universal-Omega deleted branch 03infra [03:13:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/mw-config/compare/a425dc1befad...10c4821b1061 [03:13:45] [02miraheze/mw-config] 07Universal-Omega 0310c4821 - Support using new infra based on hostname (#5444) [03:13:47] [02mw-config] 07Universal-Omega deleted branch 03infra - 13https://github.com/miraheze/mw-config [03:13:50] miraheze/mw-config - Universal-Omega the build passed. [03:13:58] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [03:14:23] rip [03:14:26] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.50, 9.95, 11.04 [03:14:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:37] miraheze/mw-config - Universal-Omega the build passed. [03:15:09] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 70s [03:15:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.68, 11.13, 11.71 [03:15:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:15:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.505 second response time [03:15:32] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.06, 10.74, 11.17 [03:15:50] !log [universalomega@test151] starting deploy of {'pull': 'config', 'config': True, 'force': True} to test151 [03:15:51] !log [universalomega@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [03:16:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:16:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:16:26] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.66, 9.61, 10.79 [03:16:40] !log [universalomega@test151] starting deploy of {'pull': 'config', 'config': True, 'force': True} to test151 [03:16:43] !log [universalomega@test151] finished deploy of {'pull': 'config', 'config': True, 'force': True} to test151 - SUCCESS in 2s [03:16:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:11] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 987 bytes in 0.653 second response time [03:17:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.89, 10.77, 11.48 [03:17:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:32] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.12, 10.59, 11.05 [03:17:47] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 987 bytes in 0.744 second response time [03:17:53] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 987 bytes in 0.781 second response time [03:18:04] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 987 bytes in 0.785 second response time [03:18:07] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 987 bytes in 0.770 second response time [03:18:22] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 987 bytes in 0.741 second response time [03:18:32] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 987 bytes in 0.648 second response time [03:18:41] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 987 bytes in 0.777 second response time [03:18:46] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 8191 bytes in 0.156 second response time [03:20:41] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [03:22:01] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [03:22:04] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:22:17] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:22:27] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 19.47, 12.40, 10.95 [03:22:27] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.26, 10.58, 10.79 [03:22:37] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.330 second response time [03:22:43] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:23:20] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 16.92, 14.29, 12.69 [03:23:33] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 17.98, 13.42, 11.90 [03:23:58] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.507 second response time [03:24:05] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.818 second response time [03:24:14] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 1.293 second response time [03:24:44] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.618 second response time [03:25:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [03:26:11] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:27:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.481 second response time [03:28:09] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.770 second response time [03:30:15] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:32:13] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.614 second response time [03:36:39] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.05, 2.93, 3.73 [03:36:46] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:38:43] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 2.686 second response time [03:39:54] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [03:40:39] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.86, 2.19, 3.23 [03:41:55] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 4.684 second response time [03:42:15] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [03:44:13] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.852 second response time [03:48:46] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [03:50:41] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.587 second response time [03:52:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.18, 10.69, 11.83 [03:55:37] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.90, 10.63, 11.71 [03:59:37] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 14.95, 11.20, 11.60 [04:03:56] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp34.miraheze.org - CNAME OK [04:04:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 16.76, 12.60, 11.61 [04:06:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.03, 11.40, 11.31 [04:07:31] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [04:08:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.73, 11.88, 11.49 [04:09:20] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.587 second response time [04:09:29] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.596 second response time [04:09:32] RECOVERY - swiftproxy161 HTTPS on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.273 second response time [04:09:45] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 1.97, 2.93, 3.91 [04:09:51] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.778 second response time [04:10:05] RECOVERY - swiftproxy161 HTTP on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.265 second response time [04:10:12] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [04:10:26] RECOVERY - swiftproxy171 HTTP on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.258 second response time [04:10:35] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.763 second response time [04:10:43] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.27, 3.94, 2.88 [04:10:49] RECOVERY - swiftproxy171 HTTPS on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.269 second response time [04:11:04] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.83, 10.60, 9.00 [04:11:05] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.524 second response time [04:11:42] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.30, 11.26, 11.84 [04:12:09] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 1.366 second response time [04:12:40] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.62, 3.34, 2.79 [04:13:03] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.03, 8.96, 8.59 [04:14:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 6.13, 11.07, 11.64 [04:14:31] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 6.99, 9.98, 11.86 [04:15:34] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.93, 6.28, 5.41 [04:17:34] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.12, 5.97, 5.39 [04:17:45] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.92, 2.37, 3.24 [04:21:43] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.80, 8.22, 9.91 [04:23:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.98, 10.54, 11.84 [04:24:37] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [04:24:38] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [04:24:39] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.61, 4.12, 3.24 [04:25:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.41, 11.26, 11.92 [04:25:27] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.92, 9.14, 7.91 [04:26:02] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.34, 8.56, 9.90 [04:26:31] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.25, 10.84, 10.88 [04:26:33] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.489 second response time [04:26:37] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.716 second response time [04:26:39] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.46, 3.82, 3.25 [04:27:26] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.42, 8.43, 7.80 [04:28:41] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.40, 3.33, 3.13 [04:30:32] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.32, 10.10, 10.62 [04:31:13] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [04:31:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.62, 11.21, 11.76 [04:31:20] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.437 second response time [04:32:04] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:32:32] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.40, 10.48, 10.66 [04:32:39] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.12, 3.82, 3.37 [04:32:42] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [04:33:09] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.484 second response time [04:33:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 18.17, 13.59, 12.55 [04:33:20] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.490 second response time [04:33:22] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw133 [04:33:22] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw133 [04:33:26] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw133 [04:34:01] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.77, 11.22, 8.54 [04:34:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.87, 10.24, 10.00 [04:34:10] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 7.726 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [04:34:32] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.51, 11.21, 10.94 [04:34:41] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.642 second response time [04:35:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.94, 11.66, 11.97 [04:35:20] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [04:35:22] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [04:35:25] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [04:36:03] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 5.81, 8.72, 9.48 [04:37:20] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.02, 11.41, 9.52 [04:37:38] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.34, 10.59, 9.48 [04:37:51] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 8.06, 10.03, 8.72 [04:39:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.03, 11.43, 11.76 [04:39:19] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.68, 9.87, 9.20 [04:39:36] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 5.95, 8.91, 8.99 [04:42:26] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.19, 3.83, 3.03 [04:42:40] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.31, 3.89, 3.80 [04:43:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.28, 11.11, 11.56 [04:46:21] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.65, 3.81, 3.20 [04:48:33] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.66, 8.64, 9.74 [04:50:16] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.75, 3.21, 3.09 [04:50:35] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.005236506462 secs [04:52:39] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.27, 3.52, 3.65 [04:54:39] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.71, 2.84, 3.39 [04:59:17] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.54, 8.84, 9.87 [05:15:12] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [05:15:20] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:15:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [05:17:09] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.545 second response time [05:17:18] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.573 second response time [05:17:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.496 second response time [05:26:01] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:26:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 17.28, 10.96, 8.55 [05:26:09] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 17.58, 11.74, 9.80 [05:26:12] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [05:26:22] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [05:26:32] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3392 bytes in 1.286 second response time [05:26:35] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 19.03, 12.50, 9.94 [05:26:37] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:26:46] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:26:52] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [05:27:03] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.375 second response time [05:27:10] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.448 second response time [05:27:46] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.374 second response time [05:28:09] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.351 second response time [05:28:33] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 4 backends are down. mw131 mw132 mw133 mw134 [05:28:40] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 8.811 second response time [05:28:41] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 7.799 second response time [05:28:43] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 4 backends are down. mw131 mw132 mw133 mw134 [05:28:50] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 9.820 second response time [05:28:51] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 14.46, 12.61, 10.06 [05:28:51] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.639 second response time [05:29:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.88, 3.42, 3.97 [05:29:09] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 6.826 second response time [05:29:12] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.761 second response time [05:29:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [05:29:45] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.584 second response time [05:30:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.57, 11.01, 9.26 [05:30:05] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.519 second response time [05:30:29] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.790 second response time [05:30:36] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.85, 11.30, 10.08 [05:30:43] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 22 backends are healthy [05:30:51] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.15, 10.87, 9.70 [05:31:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.670 second response time [05:32:02] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 6.44, 9.64, 8.99 [05:32:04] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 7.55, 10.62, 10.12 [05:32:33] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [05:32:51] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.73, 9.24, 9.23 [05:34:36] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.18, 9.39, 9.57 [05:37:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.80, 3.72, 3.82 [05:38:36] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.24, 10.79, 10.15 [05:39:58] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.79, 11.68, 10.78 [05:40:37] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.65, 11.96, 10.67 [05:41:12] PROBLEM - wonderfuleveryday.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wonderfuleveryday.org' expires in 15 day(s) (Sun 11 Feb 2024 05:35:40 AM GMT +0000). [05:41:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6f043ed3abd6...c42b812d5809 [05:41:24] [02miraheze/ssl] 07MirahezeSSLBot 03c42b812 - Bot: Update SSL cert for wonderfuleveryday.org [05:41:56] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.75, 10.86, 10.61 [05:42:37] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.96, 11.81, 10.77 [05:43:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.16, 3.83, 3.89 [05:45:53] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.26, 9.49, 10.09 [05:47:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.98, 4.36, 4.09 [05:49:48] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.16, 10.38, 10.36 [05:51:46] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.22, 11.29, 10.69 [05:52:37] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.10, 11.06, 10.62 [05:54:37] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.97, 11.07, 10.69 [05:55:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.61, 3.92, 4.00 [05:55:43] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.61, 11.64, 11.02 [05:57:42] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.84, 12.07, 11.24 [06:00:38] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.01, 10.81, 10.58 [06:01:06] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.43, 3.85, 3.91 [06:01:55] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.50, 10.50, 9.45 [06:03:06] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.39, 3.72, 3.86 [06:04:38] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 7.78, 9.94, 10.35 [06:05:12] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [06:06:38] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.39, 11.33, 10.80 [06:07:10] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.993 second response time [06:07:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [06:07:55] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.79, 9.35, 9.32 [06:08:05] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 19.91, 12.89, 9.34 [06:08:05] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.19, 11.11, 9.31 [06:09:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.48, 4.46, 4.08 [06:09:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.511 second response time [06:09:55] RECOVERY - wonderfuleveryday.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wonderfuleveryday.org' will expire on Thu 25 Apr 2024 04:41:18 AM GMT +0000. [06:10:04] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.95, 10.83, 8.99 [06:10:05] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.12, 9.63, 9.00 [06:10:38] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.48, 10.55, 10.61 [06:10:50] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.00, 7.02, 6.26 [06:12:04] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.02, 9.30, 8.64 [06:14:38] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 16.41, 12.16, 11.14 [06:16:38] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.18, 6.54, 6.38 [06:16:38] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.39, 11.33, 10.97 [06:19:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.71, 3.75, 3.96 [06:19:25] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 5.47, 10.27, 11.67 [06:20:38] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.77, 9.35, 10.20 [06:29:17] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.85, 8.73, 10.16 [06:30:37] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [06:31:05] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.33, 2.80, 3.37 [06:31:11] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.16, 7.17, 6.68 [06:32:05] [02miraheze/landing] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/styfle/cancel-workflow-action-0.12.1 [+0/-0/±1] 13https://github.com/miraheze/landing/commit/4397de30cce6 [06:32:06] [02miraheze/landing] 07dependabot[bot] 034397de3 - Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 [06:32:07] [02landing] 07dependabot[bot] created branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.12.1 - 13https://github.com/miraheze/landing [06:32:09] [02landing] 07dependabot[bot] labeled pull request 03#129: Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/landing/pull/129 [06:32:11] [02landing] 07dependabot[bot] opened pull request 03#129: Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/landing/pull/129 [06:32:33] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.481 second response time [06:32:43] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw141 [06:32:46] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw141 [06:32:47] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw141 [06:32:54] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 15.62, 10.78, 8.60 [06:32:58] miraheze/landing - dependabot[bot] the build passed. [06:33:06] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.98, 6.78, 6.59 [06:33:09] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw141 [06:33:15] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.98, 11.79, 8.67 [06:34:43] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [06:34:43] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 22 backends are healthy [06:34:46] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [06:34:53] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.43, 10.03, 8.56 [06:35:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.14, 3.62, 3.57 [06:35:09] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [06:35:13] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 8.83, 10.72, 8.65 [06:35:59] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [06:37:13] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.26, 9.90, 8.63 [06:37:58] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.569 second response time [06:38:50] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.37, 9.52, 8.85 [06:39:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.81, 10.53, 10.25 [06:40:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.02, 10.50, 9.00 [06:42:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.62, 10.39, 9.15 [06:42:05] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [06:43:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.55, 4.00, 3.86 [06:44:02] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.72, 9.66, 9.03 [06:45:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.60, 11.41, 10.66 [06:45:40] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.50, 10.75, 9.95 [06:47:40] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.33, 11.67, 10.38 [06:49:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.08, 3.61, 3.69 [06:49:19] [02miraheze/phabricator-extensions] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/styfle/cancel-workflow-action-0.12.1 [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/commit/12eb94cc4686 [06:49:20] [02miraheze/phabricator-extensions] 07dependabot[bot] 0312eb94c - Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 [06:49:21] [02phabricator-extensions] 07dependabot[bot] created branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.12.1 - 13https://github.com/miraheze/phabricator-extensions [06:49:24] [02phabricator-extensions] 07dependabot[bot] labeled pull request 03#43: Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/phabricator-extensions/pull/43 [06:49:27] [02phabricator-extensions] 07dependabot[bot] labeled pull request 03#43: Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/phabricator-extensions/pull/43 [06:49:29] [02phabricator-extensions] 07dependabot[bot] opened pull request 03#43: Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/phabricator-extensions/pull/43 [06:49:40] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.19, 11.30, 10.42 [06:50:01] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.24, 9.96, 9.39 [06:50:04] miraheze/phabricator-extensions - dependabot[bot] the build passed. [06:51:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.59, 3.27, 3.56 [06:51:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.18, 11.23, 10.89 [06:51:40] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.26, 12.35, 10.90 [06:52:01] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 9.78, 10.07, 9.51 [06:52:28] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.33, 7.64, 6.71 [06:53:12] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [06:54:24] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.88, 6.53, 6.41 [06:54:28] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.24, 8.87, 7.70 [06:55:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.62, 3.70, 3.65 [06:55:09] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.478 second response time [06:56:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 15.17, 11.32, 9.83 [06:56:02] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.52, 11.16, 10.00 [06:56:27] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.05, 7.62, 7.38 [06:57:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.54, 3.56, 3.60 [06:57:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.03, 12.09, 11.33 [06:58:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.65, 10.71, 9.77 [06:58:18] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.36, 6.99, 6.60 [07:00:24] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=superwariobroswiki --no-updates --username-prefix=mh:superwariobros /home/reception/ImportDump/superwariobroswiki-20240126060516.xml (START) [07:00:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.65, 11.28, 10.22 [07:03:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 6.33, 4.70, 4.02 [07:03:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.78, 11.96, 11.73 [07:03:20] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.375 second response time [07:04:06] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.50, 6.05, 6.33 [07:05:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.08, 12.74, 12.08 [07:05:17] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [07:06:17] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.76, 11.67, 9.12 [07:06:38] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.68, 10.24, 8.88 [07:07:07] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=eleniswiki (END - exit=0) [07:07:14] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.622 second response time [07:07:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.490 second response time [07:07:32] !log [reception@mwtask181] starting deploy of {'force': True, 'files': '../mediawiki/1.40/extensions/SemanticMediaWiki/.smw.json'} to all [07:07:52] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [07:08:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.39, 11.35, 10.74 [07:08:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:37] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 18.84, 12.72, 9.88 [07:08:37] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [07:08:39] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 28.44, 16.59, 11.18 [07:09:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.10, 3.81, 3.84 [07:09:48] !log [reception@mwtask181] finished deploy of {'force': True, 'files': '../mediawiki/1.40/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 135s [07:09:50] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.657 second response time [07:10:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 15.52, 13.13, 11.47 [07:10:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:10:27] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 7.95, 10.91, 8.83 [07:10:33] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.521 second response time [07:10:35] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.96, 10.03, 9.22 [07:11:02] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=321nailswiki --no-updates --username-prefix=millennium /home/reception/ImportDump/321nailswiki-20240120031140.xml (START) [07:11:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.09, 4.27, 4.01 [07:11:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:11:58] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [07:12:46] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [07:13:14] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [07:13:14] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=superwariobroswiki --no-updates --username-prefix=mh:superwariobros /home/reception/ImportDump/superwariobroswiki-20240126060516.xml (END - exit=256) [07:13:14] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.474 second response time [07:13:15] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=superwariobroswiki (START) [07:14:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.16, 7.06, 6.73 [07:14:04] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 7.703 second response time [07:14:12] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [07:14:32] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.18, 11.61, 10.08 [07:14:43] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw131 mw132 [07:14:44] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [07:14:45] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw131 mw132 [07:14:53] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 2 backends are down. mw131 mw132 [07:14:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:15:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.49, 3.98, 3.97 [07:16:04] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw141 [07:16:12] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 18.29, 14.24, 10.82 [07:16:13] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 5.957 second response time [07:16:31] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 10.08, 12.02, 10.48 [07:16:41] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [07:16:46] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.242 second response time [07:16:50] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [07:17:17] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.500 second response time [07:17:20] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.625 second response time [07:18:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.16, 6.11, 6.41 [07:18:03] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [07:18:08] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 7.14, 11.35, 10.16 [07:18:30] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.66, 11.03, 10.28 [07:18:31] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 6.94, 11.71, 11.30 [07:18:43] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 22 backends are healthy [07:18:45] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.129 second response time [07:20:03] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.53, 9.47, 9.62 [07:20:28] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.87, 9.88, 9.94 [07:22:05] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.21, 10.00, 11.74 [07:22:28] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 4.92, 8.16, 9.94 [07:24:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.41, 10.15, 11.65 [07:29:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.24, 10.50, 11.95 [07:31:05] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.62, 3.00, 3.36 [07:31:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.70, 11.37, 12.08 [07:34:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.35, 11.18, 11.14 [07:35:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.91, 3.80, 3.62 [07:35:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.28, 11.21, 11.90 [07:36:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.91, 10.70, 10.95 [07:37:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.89, 11.61, 11.94 [07:37:33] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.02, 7.32, 6.90 [07:38:08] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.07, 11.26, 10.87 [07:38:39] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=superwariobroswiki (END - exit=0) [07:38:41] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=superwariobroswiki --active --update (END - exit=0) [07:38:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:39:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:39:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.17, 3.97, 3.73 [07:39:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.55, 11.22, 11.78 [07:39:29] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.27, 6.42, 6.61 [07:39:44] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.31, 11.33, 11.94 [07:40:08] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.89, 11.07, 10.88 [07:41:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.85, 3.94, 3.75 [07:42:02] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.19, 8.95, 10.15 [07:42:35] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1015660465 secs [07:44:09] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.37, 11.41, 11.02 [07:45:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.26, 11.06, 11.39 [07:46:09] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.73, 10.78, 10.83 [07:47:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.59, 10.32, 11.07 [07:47:18] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.29, 7.91, 7.04 [07:47:54] PROBLEM - mw132 Disk Space on mw132 is WARNING: DISK WARNING - free space: / 2609MiB (10% inode=69%); [07:48:09] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.11, 11.10, 10.93 [07:49:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.66, 4.06, 3.80 [07:50:09] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.04, 10.40, 10.70 [07:51:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.88, 3.94, 3.79 [07:51:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.92, 11.09, 11.13 [07:53:06] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.10, 7.66, 7.26 [07:53:13] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=321nailswiki --no-updates --username-prefix=millennium /home/reception/ImportDump/321nailswiki-20240120031140.xml (END - exit=0) [07:53:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.23, 11.10, 11.12 [07:53:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:54:10] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.28, 9.20, 10.15 [07:56:59] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.35, 8.90, 7.84 [07:57:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.19, 3.80, 3.75 [07:57:45] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.35, 9.20, 10.16 [07:59:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.76, 3.76, 3.74 [08:01:15] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:01:17] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.05, 8.98, 10.15 [08:01:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [08:03:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.37, 4.19, 3.88 [08:03:12] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.514 second response time [08:03:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.577 second response time [08:05:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.17, 10.15, 10.45 [08:09:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.40, 3.64, 3.80 [08:11:17] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.34, 9.59, 10.16 [08:19:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.56, 3.66, 3.66 [08:21:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.94, 3.33, 3.53 [08:23:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.45, 3.99, 3.74 [08:24:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.33, 7.51, 7.84 [08:24:53] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.40, 3.15, 2.01 [08:25:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.38, 3.68, 3.65 [08:26:12] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.82, 10.71, 10.27 [08:26:50] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.66, 3.34, 2.23 [08:28:10] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 15.65, 12.54, 10.99 [08:28:46] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.94, 3.31, 2.35 [08:29:08] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.15, 10.95, 9.89 [08:29:46] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.25, 3.44, 2.62 [08:29:47] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.56, 10.34, 10.01 [08:30:03] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.77, 7.56, 7.67 [08:30:14] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.63, 10.72, 9.67 [08:31:46] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.92, 3.55, 2.76 [08:31:47] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.32, 11.44, 10.47 [08:32:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.80, 7.30, 7.57 [08:32:14] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.40, 11.44, 10.08 [08:32:43] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.98, 3.51, 2.68 [08:33:00] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.24, 11.70, 10.48 [08:33:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.96, 4.01, 3.74 [08:33:20] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:33:47] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 7.16, 10.25, 10.19 [08:33:49] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.62, 4.10, 3.05 [08:33:50] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [08:34:12] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [08:34:15] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.68, 10.75, 10.02 [08:34:41] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [08:34:41] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.20, 4.45, 3.12 [08:34:46] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:34:50] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [08:34:55] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 17.52, 13.14, 11.09 [08:35:13] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [08:35:23] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 5.743 second response time [08:35:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [08:35:45] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 15.82, 11.18, 8.47 [08:35:47] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.53, 11.89, 10.78 [08:35:50] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.553 second response time [08:35:54] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:36:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.56, 7.16, 7.37 [08:36:06] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw132 mw141 [08:36:09] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.309 second response time [08:36:15] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.61, 12.48, 10.78 [08:36:34] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw132 [08:36:37] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.234 second response time [08:36:39] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.59, 12.82, 9.41 [08:36:41] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.566 second response time [08:36:43] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw132 [08:36:50] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.079 second response time [08:36:55] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw132 [08:37:09] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.486 second response time [08:37:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.960 second response time [08:37:41] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.67, 9.78, 8.29 [08:37:51] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.766 second response time [08:38:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.83, 7.09, 7.33 [08:38:05] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [08:38:33] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [08:38:37] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 7.72, 11.18, 9.24 [08:38:43] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 22 backends are healthy [08:38:54] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [08:40:12] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [08:40:15] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.44, 11.27, 10.80 [08:40:36] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 15.93, 11.98, 9.68 [08:42:11] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 3.737 second response time [08:42:16] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.86, 12.14, 11.18 [08:42:34] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 8.43, 10.19, 9.29 [08:42:40] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.59, 12.46, 9.70 [08:43:25] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:43:57] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [08:44:05] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [08:44:08] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3392 bytes in 1.236 second response time [08:45:24] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.485 second response time [08:45:55] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.630 second response time [08:46:03] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.596 second response time [08:46:05] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3478 bytes in 1.323 second response time [08:46:38] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.22, 10.18, 9.49 [08:48:03] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [08:48:09] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [08:48:12] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [08:48:43] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.78, 3.81, 3.70 [08:50:04] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 2.469 second response time [08:50:09] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.313 second response time [08:50:09] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.165 second response time [08:50:37] PROBLEM - graylog131 PowerDNS Recursor on graylog131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:50:42] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.36, 3.92, 3.75 [08:52:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.93, 6.35, 6.77 [08:52:19] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.32, 10.63, 11.26 [08:52:38] RECOVERY - graylog131 PowerDNS Recursor on graylog131 is OK: DNS OK: 0.898 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [08:52:39] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.93, 3.87, 3.74 [08:54:13] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 7.66, 11.04, 11.75 [08:56:39] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.46, 4.22, 3.89 [08:57:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [08:58:04] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 17.80, 12.98, 12.24 [08:58:19] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 19.53, 13.93, 12.26 [08:59:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.602 second response time [09:00:03] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.55, 8.83, 7.73 [09:02:51] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:08:20] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 5.92, 10.47, 11.76 [09:10:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 6.41, 10.20, 11.62 [09:10:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.51, 7.37, 7.71 [09:11:50] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.50, 10.66, 11.78 [09:15:50] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 15.87, 12.00, 11.99 [09:16:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.22, 6.81, 7.30 [09:17:50] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.60, 11.46, 11.81 [09:18:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.96, 11.36, 11.38 [09:18:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.95, 6.07, 6.97 [09:20:03] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.92, 11.93, 11.64 [09:21:21] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:21:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:22:22] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 9.92, 9.48, 10.20 [09:23:20] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.130 second response time [09:23:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.535 second response time [09:23:46] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 1.50, 3.00, 3.88 [09:23:52] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.64, 10.61, 9.27 [09:25:50] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 8.14, 9.80, 9.16 [09:26:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.46, 11.55, 11.40 [09:27:45] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.01, 2.04, 3.29 [09:28:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 7.38, 10.03, 10.88 [09:28:22] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.90, 10.44, 10.39 [09:29:50] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.94, 11.55, 11.32 [09:30:23] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.54, 11.86, 10.92 [09:31:12] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:31:51] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 7.48, 10.08, 10.82 [09:32:23] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.37, 9.96, 10.31 [09:33:09] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.503 second response time [09:33:31] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.33, 10.38, 11.70 [09:34:23] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.16, 8.72, 9.82 [09:35:51] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.34, 8.76, 10.13 [09:36:02] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 9.44, 9.32, 10.14 [09:37:28] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.47, 11.47, 11.76 [09:38:03] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.98, 6.06, 6.77 [09:39:30] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:39:50] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:39:51] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 21.23, 14.32, 11.98 [09:39:54] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:40:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 22.37, 15.71, 12.50 [09:40:03] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3392 bytes in 1.267 second response time [09:40:24] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.82, 12.71, 11.16 [09:40:32] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3392 bytes in 1.210 second response time [09:40:37] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:40:39] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 0.78, 2.75, 3.87 [09:41:05] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:41:06] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.95, 11.11, 9.21 [09:41:22] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 10.27, 10.01, 8.22 [09:41:35] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 14.47, 11.66, 9.67 [09:41:38] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 8.768 second response time [09:41:45] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw131 mw134 [09:41:46] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 3 backends are down. mw131 mw133 mw134 [09:42:33] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.490 second response time [09:42:46] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [09:43:10] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw131 mw133 [09:43:20] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 3 backends are down. mw131 mw133 mw134 [09:43:21] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 3 backends are down. mw131 mw133 mw134 [09:43:57] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.966 second response time [09:43:57] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.332 second response time [09:44:09] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.329 second response time [09:44:28] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [09:44:37] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.534 second response time [09:44:39] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.29, 1.49, 3.10 [09:44:41] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.701 second response time [09:45:06] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 22 backends are healthy [09:45:11] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.503 second response time [09:45:13] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 4.99, 8.88, 8.33 [09:45:18] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [09:45:20] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [09:45:32] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.28, 10.79, 9.88 [09:45:43] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [09:45:46] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [09:46:03] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.21, 7.67, 7.20 [09:46:25] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.99, 10.98, 11.03 [09:47:03] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 5.26, 8.88, 9.07 [09:47:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.22, 3.28, 3.91 [09:47:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.39, 6.98, 6.36 [09:47:30] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.37, 9.36, 9.46 [09:48:04] PROBLEM - swiftproxy161 NTP time on swiftproxy161 is WARNING: NTP WARNING: Offset 0.1041423976 secs [09:48:30] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [09:49:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.91, 6.93, 6.42 [09:49:09] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.22, 3.52, 3.92 [09:49:51] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.12, 10.90, 11.85 [09:50:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.46, 10.84, 11.68 [09:50:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.38, 7.96, 7.46 [09:50:04] RECOVERY - swiftproxy161 NTP time on swiftproxy161 is OK: NTP OK: Offset 0.09600889683 secs [09:50:25] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.40, 8.65, 10.06 [09:51:07] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.56, 3.51, 3.87 [09:52:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.91, 8.37, 7.66 [09:52:53] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.38, 6.39, 6.35 [09:53:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.47, 10.47, 11.92 [09:54:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.15, 7.99, 7.62 [09:57:05] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.19, 2.66, 3.36 [09:57:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.31, 10.92, 11.70 [09:57:53] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.37, 10.29, 10.94 [09:57:54] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:58:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.75, 8.42, 7.87 [09:59:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.33, 9.96, 11.26 [09:59:51] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.109 second response time [09:59:53] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.75, 10.63, 11.03 [10:00:03] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.98, 7.67, 7.69 [10:01:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.65, 10.47, 11.25 [10:01:38] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp35.miraheze.org - CNAME OK [10:02:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.23, 8.01, 7.80 [10:03:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.06, 3.60, 3.54 [10:03:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.99, 9.73, 10.89 [10:04:02] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 9.28, 9.38, 10.20 [10:04:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.30, 7.01, 7.49 [10:04:10] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp35.miraheze.org - CNAME OK [10:05:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.80, 3.48, 3.49 [10:07:53] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.26, 8.65, 9.97 [10:09:05] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.24, 2.89, 3.26 [10:09:17] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.21, 8.59, 10.10 [10:10:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.93, 7.89, 7.65 [10:12:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.38, 7.22, 7.43 [10:16:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.28, 7.31, 7.40 [10:20:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.83, 6.76, 7.20 [10:21:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.96, 3.59, 3.43 [10:23:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.02, 3.87, 3.56 [10:23:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.63, 10.25, 9.62 [10:24:01] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.11, 9.85, 9.34 [10:25:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.76, 3.45, 3.44 [10:25:17] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 9.24, 9.99, 9.61 [10:26:01] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 10.20, 9.93, 9.43 [10:26:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.14, 6.95, 7.18 [10:28:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.84, 6.61, 7.03 [10:30:03] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.78, 7.16, 7.16 [10:30:47] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.11, 9.79, 8.98 [10:31:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.30, 10.68, 9.98 [10:32:01] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.35, 10.54, 9.87 [10:32:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.19, 6.27, 6.86 [10:32:40] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.84, 3.33, 2.21 [10:34:33] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.08, 10.57, 9.39 [10:34:38] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.80, 9.40, 9.06 [10:34:39] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.18, 3.08, 2.24 [10:35:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.79, 11.07, 10.34 [10:36:02] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.73, 9.81, 9.80 [10:36:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.56, 7.27, 7.14 [10:36:33] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.68, 10.14, 9.40 [10:37:05] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.24, 3.05, 3.29 [10:38:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.48, 5.96, 6.66 [10:39:17] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.70, 9.79, 10.02 [10:46:56] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 12.39, 9.06, 7.67 [10:50:49] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.22, 7.34, 7.29 [10:51:02] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.82, 10.56, 10.01 [10:51:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.88, 10.11, 9.84 [10:53:03] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.51, 9.46, 9.68 [10:53:17] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.84, 9.41, 9.61 [10:55:21] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 15.67, 9.84, 7.42 [10:56:41] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.89, 3.58, 2.48 [10:57:20] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.86, 8.60, 7.24 [10:58:33] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.10, 5.93, 6.65 [10:58:39] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.23, 2.75, 2.32 [11:02:36] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.35, 10.14, 9.49 [11:03:03] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.72, 10.22, 10.05 [11:03:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.35, 3.47, 3.27 [11:03:17] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.96, 10.84, 10.21 [11:03:36] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [11:04:24] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.39, 6.94, 6.78 [11:05:05] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.05, 3.35, 3.25 [11:06:36] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 9.54, 9.98, 9.62 [11:07:03] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.54, 9.51, 9.85 [11:07:17] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.79, 10.10, 10.13 [11:08:16] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.16, 7.03, 6.92 [11:12:08] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.69, 6.58, 6.76 [11:17:23] PROBLEM - mw134 Disk Space on mw134 is WARNING: DISK WARNING - free space: / 2608MiB (10% inode=70%); [11:22:55] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.71, 3.43, 3.29 [11:26:53] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.03, 3.44, 3.31 [11:28:01] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.39, 10.38, 9.83 [11:28:51] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.93, 3.23, 3.25 [11:30:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 16.61, 12.32, 9.87 [11:30:39] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 17.11, 13.05, 10.20 [11:31:05] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 16.72, 11.55, 9.66 [11:32:42] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [11:32:46] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [11:32:47] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.24, 4.12, 3.60 [11:33:03] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp34.miraheze.org - CNAME OK [11:34:59] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 19.77, 13.08, 8.96 [11:35:13] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.89, 9.65, 7.16 [11:36:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.23, 7.22, 6.79 [11:36:05] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 18.61, 13.08, 8.80 [11:36:45] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.584 second response time [11:36:58] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.42, 12.00, 9.04 [11:37:13] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 9.07, 9.57, 7.45 [11:38:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.53, 6.71, 6.67 [11:38:05] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.45, 11.63, 8.81 [11:38:45] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.36, 3.87, 3.67 [11:40:04] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 5.24, 9.43, 8.35 [11:40:40] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.19, 11.67, 11.60 [11:40:55] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.87, 9.55, 8.75 [11:43:05] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 6.98, 10.96, 11.68 [11:43:48] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.76, 10.61, 11.80 [11:44:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 5.33, 9.28, 11.21 [11:44:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.21, 7.47, 7.06 [11:44:41] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.65, 3.00, 3.34 [11:46:03] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.90, 6.63, 6.80 [11:46:40] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.11, 7.56, 9.71 [11:49:43] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.18, 10.32, 11.18 [11:50:02] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 9.19, 8.40, 10.16 [11:50:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.34, 6.90, 6.93 [11:51:06] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.12, 8.24, 10.02 [11:51:42] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.01, 10.10, 11.03 [11:54:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.10, 6.23, 6.67 [11:54:42] PROBLEM - cp26 Disk Space on cp26 is WARNING: DISK WARNING - free space: / 10222MiB (10% inode=98%); [12:03:07] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.98, 10.66, 10.33 [12:05:20] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:05:20] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.30, 4.10, 2.58 [12:05:30] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.37, 9.39, 10.13 [12:06:35] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5158033073 secs [12:07:18] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.652 second response time [12:07:39] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.26, 11.29, 8.73 [12:08:24] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.78, 3.46, 3.26 [12:08:59] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.41, 11.02, 8.87 [12:09:07] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.85, 9.35, 9.93 [12:09:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [12:09:47] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.04, 7.05, 6.77 [12:09:50] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:10:22] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.23, 3.65, 3.34 [12:11:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.489 second response time [12:11:37] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.52, 11.35, 9.44 [12:11:43] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.22, 6.68, 6.66 [12:11:49] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.591 second response time [12:12:56] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 6.82, 11.00, 9.61 [12:13:08] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.73, 3.89, 3.21 [12:13:36] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 5.66, 9.24, 8.89 [12:14:54] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 5.22, 8.97, 9.03 [12:15:06] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.45, 3.40, 3.11 [12:16:18] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.37, 10.16, 10.23 [12:17:04] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 1.676 second response time [12:18:16] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 10.11, 10.01, 10.15 [12:19:03] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.480 second response time [12:19:28] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.47, 7.79, 7.11 [12:22:16] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.18, 3.89, 3.80 [12:23:20] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.54, 7.44, 7.14 [12:25:16] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.73, 8.05, 7.39 [12:26:48] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.71, 10.98, 9.81 [12:27:12] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.07, 7.44, 7.26 [12:28:08] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.42, 9.43, 9.24 [12:28:49] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.81, 10.69, 9.87 [12:30:11] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.38, 3.82, 3.74 [12:30:28] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:30:49] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 14.57, 12.14, 10.49 [12:32:02] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.53, 10.87, 10.26 [12:32:05] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.23, 10.05, 9.23 [12:32:49] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.29, 9.91, 9.88 [12:34:02] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 9.98, 10.01, 9.31 [12:35:59] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.40, 11.47, 10.61 [12:36:07] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.80, 3.98, 3.84 [12:36:08] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.61, 10.13, 9.75 [12:37:23] RECOVERY - mw134 Disk Space on mw134 is OK: DISK OK - free space: / 3058MiB (12% inode=70%); [12:38:48] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.92, 6.30, 6.76 [12:38:50] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.88, 10.52, 10.12 [12:42:09] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.50, 11.51, 10.48 [12:42:50] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.19, 10.97, 10.36 [12:44:09] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 11.90, 12.19, 10.89 [12:44:50] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [12:44:50] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.97, 10.91, 10.44 [12:46:09] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.52, 11.46, 10.78 [12:46:48] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.516 second response time [12:46:51] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.28, 9.96, 10.14 [12:48:09] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.82, 12.27, 11.15 [12:52:09] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.39, 11.77, 11.30 [12:52:22] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.83, 7.48, 6.96 [12:53:45] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.12, 11.98, 11.90 [12:54:19] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.14, 7.38, 6.99 [12:54:36] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.07, 10.98, 10.06 [12:55:44] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.97, 13.22, 12.38 [12:55:53] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.31, 3.79, 3.66 [12:56:09] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.74, 11.96, 11.37 [12:56:15] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.20, 7.84, 7.21 [12:56:32] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.25, 11.25, 10.28 [12:56:42] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.67, 3.51, 2.73 [12:57:52] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.53, 11.61, 10.67 [12:57:53] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.16, 3.67, 3.64 [12:58:11] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.43, 7.48, 7.14 [12:58:28] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.78, 12.77, 10.99 [12:58:39] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.43, 3.96, 2.97 [13:00:07] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.47, 7.77, 7.26 [13:00:22] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [13:03:13] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:03:49] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.81, 4.17, 3.84 [13:04:40] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.95, 3.73, 3.24 [13:07:47] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.24, 3.78, 3.77 [13:08:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.48, 7.84, 7.49 [13:08:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.75, 11.94, 11.61 [13:10:39] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.70, 3.19, 3.17 [13:11:44] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.03, 3.98, 3.86 [13:11:53] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.10, 11.24, 11.88 [13:12:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.65, 8.88, 7.93 [13:13:43] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.15, 3.76, 3.79 [13:16:13] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 6.12, 9.33, 11.34 [13:18:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.17, 7.42, 7.74 [13:20:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 16.80, 12.22, 11.50 [13:20:13] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.09, 10.77, 11.43 [13:21:37] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.14, 2.72, 3.28 [13:22:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.34, 11.65, 11.40 [13:22:13] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.52, 9.88, 11.01 [13:23:55] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.84, 11.73, 11.44 [13:24:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.41, 11.76, 11.46 [13:24:13] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 15.85, 11.76, 11.53 [13:27:32] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.23, 3.77, 3.58 [13:31:29] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.00, 3.56, 3.55 [13:32:35] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [13:33:28] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.92, 4.02, 3.72 [13:34:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.55, 6.17, 6.77 [13:36:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.17, 11.38, 11.89 [13:39:25] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.33, 3.69, 3.71 [13:40:15] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.62, 11.20, 11.97 [13:47:19] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.22, 3.48, 3.56 [13:48:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.69, 11.74, 11.49 [13:48:15] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.98, 11.08, 11.47 [13:50:03] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.79, 7.19, 6.94 [13:51:16] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.70, 3.77, 3.67 [13:53:15] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.19, 3.87, 3.72 [13:56:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.25, 8.13, 7.37 [13:57:13] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.76, 3.96, 3.81 [13:58:03] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.52, 7.92, 7.38 [13:59:12] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.08, 4.42, 4.00 [14:01:11] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.28, 3.91, 3.86 [14:02:00] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp34.miraheze.org - CNAME OK [14:03:10] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.94, 4.46, 4.07 [14:04:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.54, 7.79, 7.49 [14:04:46] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [14:05:09] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.70, 3.87, 3.91 [14:05:12] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [14:06:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.42, 7.11, 7.29 [14:06:13] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=superwariobroswiki --no-updates --username-prefix=mh:superwariobros /home/reception/ImportDump/superwariobroswiki-20240126140339.xml (START) [14:06:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:06:44] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 3.697 second response time [14:07:07] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.61, 4.12, 3.99 [14:07:09] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.495 second response time [14:10:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 3.39, 5.16, 6.45 [14:11:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.91, 3.92, 3.94 [14:21:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8b6d8f9060e9...78bc3fc6169a [14:21:00] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:21:02] [02miraheze/puppet] 07paladox 0378bc3fc - swift: temporarily increase rsync_timeout to 1h for object-server [14:21:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.67, 4.10, 3.96 [14:27:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.45, 3.94, 3.98 [14:28:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.68, 7.15, 6.66 [14:30:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.78, 8.02, 7.03 [14:30:22] PROBLEM - wikitide.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikitide.net All nameservers failed to answer the query. [14:32:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.73, 7.33, 6.89 [14:34:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.34, 6.65, 6.70 [14:37:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.20, 3.91, 3.89 [14:40:03] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.08, 10.57, 11.98 [14:40:45] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/cfdbbcab4d00 [14:40:47] [02miraheze/dns] 07paladox 03cfdbbca - Change swift-lb backend to swiftproxy161/171 [14:40:49] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [14:40:51] [02dns] 07paladox opened pull request 03#471: Change swift-lb backend to swiftproxy161/171 - 13https://github.com/miraheze/dns/pull/471 [14:40:56] [02dns] 07paladox closed pull request 03#471: Change swift-lb backend to swiftproxy161/171 - 13https://github.com/miraheze/dns/pull/471 [14:40:59] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/476ea5321b11...2ac3b37b377f [14:41:01] [02miraheze/dns] 07paladox 032ac3b37 - Change swift-lb backend to swiftproxy161/171 (#471) [14:41:03] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [14:41:04] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [14:41:05] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.57, 3.83, 3.89 [14:42:03] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.46, 11.17, 12.00 [14:42:58] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [14:43:12] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [14:43:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/78bc3fc6169a...2b6c5b50156a [14:43:47] [02miraheze/puppet] 07paladox 032b6c5b5 - Set role::swift::stats_reporter_host to swiftproxy171 [14:44:03] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.83, 10.82, 11.76 [14:45:09] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.486 second response time [14:46:03] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.02, 11.50, 11.89 [14:47:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.60, 3.65, 3.73 [14:48:02] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.39, 7.67, 6.60 [14:48:03] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.23, 10.86, 11.63 [14:49:41] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [14:50:01] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.64, 7.60, 6.71 [14:50:04] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 17.75, 13.16, 12.36 [14:50:56] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.02, 7.10, 6.83 [14:51:26] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.26, 6.89, 6.04 [14:51:43] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.907 second response time [14:52:00] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.65, 8.09, 7.00 [14:52:13] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10001 milliseconds [14:52:16] [02miraheze/mw-config] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/styfle/cancel-workflow-action-0.12.1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/fe524d629b74 [14:52:18] [02miraheze/mw-config] 07dependabot[bot] 03fe524d6 - Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 [14:52:21] [02mw-config] 07dependabot[bot] opened pull request 03#5445: Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/mw-config/pull/5445 [14:52:22] [02mw-config] 07dependabot[bot] created branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.12.1 - 13https://github.com/miraheze/mw-config [14:52:24] [02mw-config] 07dependabot[bot] labeled pull request 03#5445: Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/mw-config/pull/5445 [14:52:26] [02mw-config] 07dependabot[bot] labeled pull request 03#5445: Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/mw-config/pull/5445 [14:52:54] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.86, 6.46, 6.61 [14:53:08] miraheze/mw-config - dependabot[bot] the build passed. [14:53:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.33, 7.54, 6.94 [14:54:39] PROBLEM - cp51 SSH on cp51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:06] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 100% [14:55:18] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.77, 6.38, 6.04 [14:56:34] PROBLEM - cp51 Disk Space on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:56:36] PROBLEM - cp51 Nginx Backend for mw142 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:56:36] PROBLEM - cp51 Nginx Backend for puppet181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:56:54] PROBLEM - cp51 Nginx Backend for reports171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:56:54] PROBLEM - cp51 Nginx Backend for mw181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:56:57] PROBLEM - cp51 Nginx Backend for mw172 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:57:08] PROBLEM - Host cp51 is DOWN: PING CRITICAL - Packet loss = 100% [14:59:18] [02mw-config] 07paladox commented on commit 0310c4821b1061b150e195fc29570bba92b2a10589 - 13https://github.com/miraheze/mw-config/commit/10c4821b1061b150e195fc29570bba92b2a10589#r137860969 [15:00:22] PROBLEM - wikitide.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikitide.net could not be found [15:00:23] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [15:00:50] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:01:55] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.22, 6.59, 6.76 [15:02:48] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.503 second response time [15:03:29] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/ee32da12a272 [15:03:30] [02miraheze/mw-config] 07paladox 03ee32da1 - Use private ips for new db rather than hostname [15:03:32] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [15:03:33] [02mw-config] 07paladox opened pull request 03#5446: Use private ips for new db rather than hostname - 13https://github.com/miraheze/mw-config/pull/5446 [15:04:37] miraheze/mw-config - paladox the build passed. [15:04:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ee32da12a272...670d8bd7044c [15:04:53] [02miraheze/mw-config] 07paladox 03670d8bd - Update GlobalCache.php [15:04:54] [02mw-config] 07paladox synchronize pull request 03#5446: Use private ips for new db rather than hostname - 13https://github.com/miraheze/mw-config/pull/5446 [15:05:54] miraheze/mw-config - paladox the build passed. [15:06:08] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/670d8bd7044c...56099e68e4e1 [15:06:09] [02miraheze/mw-config] 07paladox 0356099e6 - Update GlobalCache.php [15:06:10] [02mw-config] 07paladox synchronize pull request 03#5446: Use private ips for new db rather than hostname - 13https://github.com/miraheze/mw-config/pull/5446 [15:07:10] miraheze/mw-config - paladox the build passed. [15:07:27] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.25, 8.23, 7.21 [15:09:12] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:11:09] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.508 second response time [15:13:16] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.98, 6.96, 7.06 [15:16:36] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=superwariobroswiki --no-updates --username-prefix=mh:superwariobros /home/reception/ImportDump/superwariobroswiki-20240126140339.xml (END - exit=0) [15:16:37] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=superwariobroswiki (START) [15:17:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:17:08] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.03, 5.98, 6.68 [15:17:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:19:15] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.80, 3.11, 2.42 [15:19:50] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:20:07] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 157.43 ms [15:21:12] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.00, 2.81, 2.40 [15:21:49] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.591 second response time [15:24:10] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 181.73 ms [15:24:56] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.64, 10.98, 8.54 [15:26:50] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.31, 6.37, 5.62 [15:28:49] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.54, 6.22, 5.64 [15:29:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:30:45] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.12, 7.32, 7.76 [15:31:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.491 second response time [15:32:17] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [15:32:22] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 12.52, 10.88, 8.97 [15:34:17] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.18, 9.44, 8.67 [15:37:22] [02mw-config] 07StemmlerSisters commented on pull request 03#5441: add drafts extension T11509 - 13https://github.com/miraheze/mw-config/pull/5441#issuecomment-1912265028 [15:38:30] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.21, 8.07, 7.68 [15:42:23] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.23, 7.32, 7.51 [15:43:56] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:45:55] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.785 second response time [15:46:53] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 12.46, 11.19, 9.80 [15:46:59] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 13.37, 12.45, 10.50 [15:47:01] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.97, 10.88, 9.53 [15:48:03] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:48:14] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.02, 7.34, 7.38 [15:48:32] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [15:48:42] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.51, 3.88, 3.00 [15:48:48] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.33, 11.20, 9.96 [15:48:58] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.38, 11.98, 10.59 [15:49:00] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.39, 12.02, 10.14 [15:50:04] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 2.508 second response time [15:50:10] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.82, 6.79, 7.15 [15:50:12] PROBLEM - swiftproxy161 NTP time on swiftproxy161 is WARNING: NTP WARNING: Offset 0.112296164 secs [15:50:30] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.312 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [15:50:39] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.48, 3.76, 3.07 [15:50:59] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.76, 10.92, 9.92 [15:51:12] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:51:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:51:42] PROBLEM - psycho.engineering - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'psycho.engineering' expires in 7 day(s) (Sat 03 Feb 2024 03:34:03 PM GMT +0000). [15:52:05] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.13, 7.10, 7.21 [15:52:39] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.32, 4.04, 3.26 [15:52:54] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.02, 12.90, 11.23 [15:53:09] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.509 second response time [15:53:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.671 second response time [15:54:03] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.42, 7.20, 7.24 [15:54:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:54:44] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.83, 3.77, 3.25 [15:54:46] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:56:03] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.32, 7.90, 7.51 [15:56:28] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.02, 6.60, 6.09 [15:56:42] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.873 second response time [15:56:56] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.70, 9.79, 9.95 [15:58:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.71, 6.81, 7.16 [15:58:23] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.81, 6.29, 6.03 [15:58:24] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.54, 9.35, 9.86 [15:58:39] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.83, 2.83, 2.99 [16:01:40] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:02:51] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.44, 10.59, 10.08 [16:03:12] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [16:03:29] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10000 milliseconds [16:04:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 3.97, 5.68, 6.61 [16:04:08] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [16:04:42] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.11, 4.13, 3.41 [16:05:09] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.665 second response time [16:05:13] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 10.66, 10.66, 10.24 [16:05:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.831 second response time [16:06:05] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.638 second response time [16:06:51] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.59, 11.04, 10.54 [16:08:39] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.66, 3.98, 3.53 [16:08:43] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.54, 11.69, 11.81 [16:09:13] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [16:09:14] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.04, 8.88, 9.66 [16:09:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:10:12] RECOVERY - swiftproxy161 NTP time on swiftproxy161 is OK: NTP OK: Offset 0.09517627954 secs [16:10:39] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.98, 3.28, 3.33 [16:11:09] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.501 second response time [16:14:43] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.71, 3.51, 3.40 [16:15:34] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.03, 6.97, 6.60 [16:17:34] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.01, 6.64, 6.52 [16:18:39] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.73, 2.78, 3.15 [16:20:43] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 8.81, 9.33, 10.12 [16:23:54] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [16:24:41] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [16:25:53] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.780 second response time [16:26:28] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.99, 9.42, 10.16 [16:26:39] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 2.939 second response time [16:41:05] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:44:27] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.80, 11.13, 10.10 [16:46:08] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.46, 9.85, 9.49 [16:48:06] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.97, 9.41, 9.39 [16:48:24] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 8.29, 9.61, 9.75 [16:53:45] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.64, 3.38, 2.65 [16:55:45] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 3.32, 3.38, 2.74 [16:56:18] [02mw-config] 07Universal-Omega closed pull request 03#5446: Use private ips for new db rather than hostname - 13https://github.com/miraheze/mw-config/pull/5446 [16:56:21] [02miraheze/mw-config] 07Universal-Omega deleted branch 03paladox-patch-4 [16:56:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/10c4821b1061...2405dc3a71f1 [16:56:23] [02miraheze/mw-config] 07paladox 032405dc3 - Use private ips for new db rather than hostname (#5446) [16:56:25] [02mw-config] 07Universal-Omega deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [16:57:21] miraheze/mw-config - Universal-Omega the build passed. [17:02:20] !log [@mwtask181] starting deploy of {'config': True} to all [17:02:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:03:20] !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw151.wikitide.net [17:03:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:05:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03db-hostnames [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/6213605cfe15 [17:05:56] [02miraheze/mw-config] 07Universal-Omega 036213605 - Use new hostnames in DB [17:05:57] [02mw-config] 07Universal-Omega created branch 03db-hostnames - 13https://github.com/miraheze/mw-config [17:06:02] [02mw-config] 07Universal-Omega opened pull request 03#5447: Use new hostnames in DB - 13https://github.com/miraheze/mw-config/pull/5447 [17:06:56] miraheze/mw-config - Universal-Omega the build passed. [17:07:16] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [17:08:42] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.33, 3.44, 2.58 [17:09:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03db-hostnames [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6213605cfe15...8b153d07959f [17:09:21] [02miraheze/mw-config] 07Universal-Omega 038b153d0 - Update [17:09:24] [02mw-config] 07Universal-Omega synchronize pull request 03#5447: Use new hostnames in DB - 13https://github.com/miraheze/mw-config/pull/5447 [17:10:13] miraheze/mw-config - Universal-Omega the build passed. [17:10:39] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.08, 3.51, 2.72 [17:12:32] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=superwariobroswiki (END - exit=0) [17:12:35] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=superwariobroswiki --active --update (END - exit=0) [17:12:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:13:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:13:11] PROBLEM - cp26 Current Load on cp26 is WARNING: LOAD WARNING - total load average: 7.08, 6.12, 5.39 [17:14:39] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.01, 3.28, 2.82 [17:15:11] RECOVERY - cp26 Current Load on cp26 is OK: LOAD OK - total load average: 4.90, 5.31, 5.17 [17:17:46] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/commit/068dbb641d6f [17:17:48] [02miraheze/phabricator-extensions] 07paladox 03068dbb6 - Backport https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/16/diffs [17:17:50] [02phabricator-extensions] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/phabricator-extensions [17:17:52] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.65, 3.42, 2.87 [17:17:53] [02phabricator-extensions] 07paladox opened pull request 03#44: Backport https://gitlab.wikimedia.org/repos/phabricator/extensions/-/… - 13https://github.com/miraheze/phabricator-extensions/pull/44 [17:17:58] [02phabricator-extensions] 07paladox edited pull request 03#44: Backport https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/16/diffs - 13https://github.com/miraheze/phabricator-extensions/pull/44 [17:18:01] [02phabricator-extensions] 07paladox edited pull request 03#44: Backport https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/16/diffs - 13https://github.com/miraheze/phabricator-extensions/pull/44 [17:19:08] miraheze/phabricator-extensions - paladox the build has errored. [17:21:49] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.21, 3.42, 3.00 [17:23:18] !log [@test151] starting deploy of {'config': True} to test151 [17:23:19] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [17:23:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:23:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:23:57] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.35, 3.67, 3.14 [17:25:54] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.29, 3.23, 3.06 [17:29:54] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp34.miraheze.org - CNAME OK [17:31:13] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [17:32:39] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.69, 2.62, 2.31 [17:33:10] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.799 second response time [17:33:16] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:33:34] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.57, 9.58, 8.96 [17:34:24] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2b6c5b50156a...f1168f91ea5a [17:34:25] [02miraheze/puppet] 07Universal-Omega 03f1168f9 - Fix host for role::swift::stats_reporter_host [17:37:30] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 9.40, 9.70, 9.16 [17:38:39] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.86, 3.04, 2.61 [17:41:05] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:42:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f1168f91ea5a...9756ee18458e [17:42:45] [02miraheze/puppet] 07Universal-Omega 039756ee1 - Use phorge for HostGroup [17:46:47] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9756ee18458e...3bbb06e4e726 [17:46:50] [02miraheze/puppet] 07Universal-Omega 033bbb06e - Fix host for role::swift::stats_reporter_host [17:47:11] PROBLEM - cp26 Current Load on cp26 is CRITICAL: LOAD CRITICAL - total load average: 8.04, 6.06, 4.91 [17:49:11] RECOVERY - cp26 Current Load on cp26 is OK: LOAD OK - total load average: 5.01, 5.62, 4.89 [18:01:27] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:06:40] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.84, 11.10, 8.93 [18:08:39] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.05, 10.44, 8.96 [18:10:37] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.98, 9.48, 8.78 [18:17:28] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.40, 7.24, 6.11 [18:19:24] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.91, 7.25, 6.24 [18:25:13] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.75, 6.80, 6.42 [18:29:35] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [18:30:53] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp35.miraheze.org - CNAME OK [18:31:30] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.121 second response time [18:40:37] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.40, 9.83, 8.52 [18:42:35] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.65, 8.86, 8.33 [18:44:32] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/068dbb641d6f...284e0800688f [18:44:35] [02miraheze/phabricator-extensions] 07paladox 03284e080 - Update PhutilMediaWikiAuthAdapter.php [18:44:37] [02phabricator-extensions] 07paladox synchronize pull request 03#44: Backport https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/16/diffs - 13https://github.com/miraheze/phabricator-extensions/pull/44 [18:44:57] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.06, 6.70, 6.43 [18:45:26] miraheze/phabricator-extensions - paladox the build has errored. [18:46:34] [02phabricator-extensions] 07paladox synchronize pull request 03#44: Backport https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/16/diffs - 13https://github.com/miraheze/phabricator-extensions/pull/44 [18:46:36] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/284e0800688f...215296fa61e1 [18:46:37] [02miraheze/phabricator-extensions] 07paladox 03215296f - Update PhabricatorMediaWikiAuthProvider.php [18:46:49] [02miraheze/phabricator-extensions] 07github-actions[bot] pushed 031 commit to 03paladox-patch-2 [+0/-0/±2] 13https://github.com/miraheze/phabricator-extensions/compare/215296fa61e1...3fb2e091643c [18:46:50] [02miraheze/phabricator-extensions] 07github-actions 033fb2e09 - CI: lint code to MediaWiki standards [18:46:51] [02phabricator-extensions] 07github-actions[bot] synchronize pull request 03#44: Backport https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/16/diffs - 13https://github.com/miraheze/phabricator-extensions/pull/44 [18:47:21] miraheze/phabricator-extensions - paladox the build passed. [18:47:38] [02phabricator-extensions] 07paladox closed pull request 03#44: Backport https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/16/diffs - 13https://github.com/miraheze/phabricator-extensions/pull/44 [18:47:41] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/phabricator-extensions/compare/aac51b326d9e...63c7961c9b2c [18:47:43] [02miraheze/phabricator-extensions] 07paladox 0363c7961 - Backport https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/16/diffs (#44) [18:47:44] [02miraheze/phabricator-extensions] 07paladox deleted branch 03paladox-patch-2 [18:47:45] [02phabricator-extensions] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/phabricator-extensions [18:48:21] miraheze/phabricator-extensions - paladox the build passed. [18:48:57] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.88, 6.54, 6.45 [18:57:14] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03swiftproxy171 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/342d4dcd9f98 [18:57:15] [02miraheze/puppet] 07Universal-Omega 03342d4dc - swift: use swiftproxy171 [18:57:16] [02puppet] 07Universal-Omega created branch 03swiftproxy171 - 13https://github.com/miraheze/puppet [18:57:18] [02puppet] 07Universal-Omega opened pull request 03#3704: swift: use swiftproxy171 - 13https://github.com/miraheze/puppet/pull/3704 [18:57:52] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03swiftproxy171 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/342d4dcd9f98...f2312d7392b0 [18:57:55] [02miraheze/puppet] 07Universal-Omega 03f2312d7 - Update [18:57:57] [02puppet] 07Universal-Omega synchronize pull request 03#3704: swift: use swiftproxy171 - 13https://github.com/miraheze/puppet/pull/3704 [19:00:22] [02puppet] 07paladox closed pull request 03#3704: swift: use swiftproxy171 - 13https://github.com/miraheze/puppet/pull/3704 [19:00:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/3bbb06e4e726...e04b777d0f1c [19:00:25] [02miraheze/puppet] 07Universal-Omega 03e04b777 - swift: use swiftproxy171 (#3704) [19:00:28] [02miraheze/puppet] 07paladox deleted branch 03swiftproxy171 [19:00:30] [02puppet] 07paladox deleted branch 03swiftproxy171 - 13https://github.com/miraheze/puppet [19:02:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/commit/800281724126 [19:02:35] [02miraheze/puppet] 07Universal-Omega 038002817 - Delete hieradata/hosts/swiftproxy111.yaml [19:02:38] [02puppet] 07Universal-Omega created branch 03decom - 13https://github.com/miraheze/puppet [19:02:49] [02puppet] 07Universal-Omega opened pull request 03#3705: Decom swiftproxy111 and swiftproxy131 - 13https://github.com/miraheze/puppet/pull/3705 [19:03:14] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/800281724126...630776a4c7d7 [19:03:16] [02miraheze/puppet] 07Universal-Omega 03630776a - Delete hieradata/hosts/swiftproxy131.yaml [19:03:17] [02puppet] 07Universal-Omega synchronize pull request 03#3705: Decom swiftproxy111 and swiftproxy131 - 13https://github.com/miraheze/puppet/pull/3705 [19:03:47] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.60, 6.60, 6.04 [19:04:01] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.07, 6.77, 6.48 [19:04:02] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.16, 6.58, 5.80 [19:04:04] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/630776a4c7d7...3b3dc37b1056 [19:04:07] [02miraheze/puppet] 07Universal-Omega 033b3dc37 - Update site.pp [19:04:10] [02puppet] 07Universal-Omega synchronize pull request 03#3705: Decom swiftproxy111 and swiftproxy131 - 13https://github.com/miraheze/puppet/pull/3705 [19:06:01] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.18, 6.34, 6.34 [19:06:36] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.72, 2.67, 1.79 [19:07:47] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.34, 6.57, 6.18 [19:08:05] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.56, 2.53, 1.82 [19:08:09] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.25, 6.65, 6.04 [19:10:18] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/dns/commit/188c4b46b272 [19:10:20] [02miraheze/dns] 07Universal-Omega 03188c4b4 - Update 0.0.0.0.0.4.7.6.0.1.a.2.ip6.arpa [19:10:23] [02dns] 07Universal-Omega created branch 03decom - 13https://github.com/miraheze/dns [19:10:31] [02dns] 07Universal-Omega opened pull request 03#472: Decom swiftproxy111 and swiftproxy131 - 13https://github.com/miraheze/dns/pull/472 [19:11:35] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/dns/compare/188c4b46b272...2e449ffa2208 [19:11:37] [02miraheze/dns] 07Universal-Omega 032e449ff - Update miraheze.org [19:11:39] [02dns] 07Universal-Omega synchronize pull request 03#472: Decom swiftproxy111 and swiftproxy131 - 13https://github.com/miraheze/dns/pull/472 [19:12:19] [02dns] 07Universal-Omega synchronize pull request 03#472: Decom swiftproxy111 and swiftproxy131 - 13https://github.com/miraheze/dns/pull/472 [19:12:19] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/dns/compare/2e449ffa2208...137f2be0407f [19:12:21] [02miraheze/dns] 07Universal-Omega 03137f2be - Update 0.0.0.0.0.4.7.6.0.1.a.2.ip6.arpa [19:12:38] [02puppet] 07Universal-Omega closed pull request 03#3705: Decom swiftproxy111 and swiftproxy131 - 13https://github.com/miraheze/puppet/pull/3705 [19:12:39] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-2/±1] 13https://github.com/miraheze/puppet/compare/e04b777d0f1c...3c70d8086d64 [19:12:42] [02miraheze/puppet] 07Universal-Omega 033c70d80 - Decom swiftproxy111 and swiftproxy131 (#3705) [19:12:43] [02miraheze/puppet] 07Universal-Omega deleted branch 03decom [19:12:45] [02puppet] 07Universal-Omega deleted branch 03decom - 13https://github.com/miraheze/puppet [19:18:00] [02dns] 07Universal-Omega closed pull request 03#472: Decom swiftproxy111 and swiftproxy131 - 13https://github.com/miraheze/dns/pull/472 [19:18:01] [02miraheze/dns] 07Universal-Omega deleted branch 03decom [19:18:04] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/2ac3b37b377f...4c30b19f7d43 [19:18:07] [02miraheze/dns] 07Universal-Omega 034c30b19 - Decom swiftproxy111 and swiftproxy131 (#472) [19:18:10] [02dns] 07Universal-Omega deleted branch 03decom - 13https://github.com/miraheze/dns [19:18:13] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 6.64, 9.97, 11.74 [19:21:36] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.82, 3.33, 3.99 [19:22:09] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 3.93, 6.74, 10.03 [19:23:07] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 4.51, 7.64, 11.12 [19:23:23] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 4.50, 7.23, 11.52 [19:23:24] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [19:23:33] PROBLEM - cp26 Disk Space on cp26 is CRITICAL: DISK CRITICAL - free space: / 5544MiB (5% inode=98%); [19:24:00] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 6.79, 8.35, 11.49 [19:24:51] PROBLEM - graylog131 HTTPS on graylog131 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 113 bytes in 0.373 second response time [19:25:24] PROBLEM - graylog131 Puppet on graylog131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 20 minutes ago with 0 failures [19:25:26] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 0.28, 1.65, 3.16 [19:26:54] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 42% [19:27:05] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.66, 7.08, 10.08 [19:28:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/1224e5e4b404 [19:28:38] [02miraheze/puppet] 07paladox 031224e5e - Switch base::syslog::syslog_host to graylog161 [19:28:40] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [19:28:41] [02puppet] 07paladox opened pull request 03#3706: Switch base::syslog::syslog_host to graylog161 - 13https://github.com/miraheze/puppet/pull/3706 [19:28:52] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [19:29:44] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.97, 7.39, 10.08 [19:30:03] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/a223b7eaf84e [19:30:04] [02miraheze/dns] 07paladox 03a223b7e - Add graylog to wikitide.net [19:30:07] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [19:30:10] [02dns] 07paladox opened pull request 03#473: Add graylog to wikitide.net - 13https://github.com/miraheze/dns/pull/473 [19:30:12] [02dns] 07paladox closed pull request 03#473: Add graylog to wikitide.net - 13https://github.com/miraheze/dns/pull/473 [19:30:13] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [19:30:16] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [19:30:50] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 40% [19:31:13] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 49% [19:31:20] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.05, 8.09, 10.14 [19:31:22] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/3b4f32a8e3ed [19:31:23] [02miraheze/dns] 07paladox 033b4f32a - Update graylog.miraheze.org backend to graylog161 [19:31:24] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [19:31:25] [02dns] 07paladox opened pull request 03#474: Update graylog.miraheze.org backend to graylog161 - 13https://github.com/miraheze/dns/pull/474 [19:31:32] [02dns] 07paladox closed pull request 03#474: Update graylog.miraheze.org backend to graylog161 - 13https://github.com/miraheze/dns/pull/474 [19:31:35] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/4c30b19f7d43...1dfb20ac2d18 [19:31:37] [02miraheze/dns] 07paladox 031dfb20a - Update graylog.miraheze.org backend to graylog161 (#474) [19:31:38] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [19:31:39] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [19:35:07] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [19:37:03] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 21299MiB (10% inode=99%); [19:37:09] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 42% [19:40:57] RECOVERY - os131 Disk Space on os131 is OK: DISK OK - free space: / 25660MiB (12% inode=99%); [19:41:07] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [19:43:07] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 56% [19:43:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3c70d8086d64...a5b86609a9a7 [19:43:38] [02miraheze/puppet] 07paladox 03a5b8660 - opensearch: bind to all network interfaces [19:44:37] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [19:45:07] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [19:46:35] RECOVERY - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is OK: OK - NGINX Error Rate is 36% [19:47:07] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 51% [19:48:09] [02mw-config] 07redbluegreenhat commented on pull request 03#5435: Use SpecialPage instead of global $wgServer - 13https://github.com/miraheze/mw-config/pull/5435#issuecomment-1912604108 [19:49:07] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [19:49:48] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.002529293299 secs [19:50:29] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [19:51:09] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:52:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 59% [19:53:07] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 43% [19:54:45] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/b9efc17da23f [19:54:48] [02miraheze/dns] 07paladox 03b9efc17 - Change opensearch.miraheze.org backend to os151 [19:54:49] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [19:54:52] [02dns] 07paladox opened pull request 03#475: Change opensearch.miraheze.org backend to os151 - 13https://github.com/miraheze/dns/pull/475 [19:54:56] [02dns] 07paladox closed pull request 03#475: Change opensearch.miraheze.org backend to os151 - 13https://github.com/miraheze/dns/pull/475 [19:54:57] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [19:54:59] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/1dfb20ac2d18...52fbc0e5f9e2 [19:55:00] [02miraheze/dns] 07paladox 0352fbc0e - Change opensearch.miraheze.org backend to os151 (#475) [19:55:03] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [19:55:43] PROBLEM - poserdazfreebies.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'orain.org' expires in 7 day(s) (Sat 03 Feb 2024 07:51:10 PM GMT +0000). [19:55:55] PROBLEM - cp26 Current Load on cp26 is CRITICAL: LOAD CRITICAL - total load average: 31.26, 18.60, 9.46 [19:57:07] RECOVERY - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is OK: OK - NGINX Error Rate is 38% [19:58:22] RECOVERY - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is OK: OK - NGINX Error Rate is 39% [19:58:30] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'orain.org' expires in 7 day(s) (Sat 03 Feb 2024 07:51:10 PM GMT +0000). [20:00:43] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.30, 9.33, 7.84 [20:02:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a5b86609a9a7...683458721eee [20:02:30] [02miraheze/puppet] 07paladox 036834587 - opensearch: listen to ipv4 addresses for nginx [20:02:43] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.07, 8.60, 7.74 [20:03:07] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 53% [20:03:55] PROBLEM - cp26 Current Load on cp26 is WARNING: LOAD WARNING - total load average: 3.25, 6.74, 7.12 [20:05:55] RECOVERY - cp26 Current Load on cp26 is OK: LOAD OK - total load average: 4.71, 6.02, 6.80 [20:07:11] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 51% [20:07:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/683458721eee...fd99a426fbdc [20:07:27] [02miraheze/puppet] 07paladox 03fd99a42 - Update graylog161.yaml [20:08:09] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 53% [20:09:06] [02puppet] 07paladox closed pull request 03#3706: Switch base::syslog::syslog_host to graylog161 - 13https://github.com/miraheze/puppet/pull/3706 [20:09:07] RECOVERY - graylog161 HTTPS on graylog161 is OK: HTTP OK: HTTP/2 200 - 646 bytes in 0.013 second response time [20:09:07] RECOVERY - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is OK: OK - NGINX Error Rate is 36% [20:09:08] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [20:09:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fd99a426fbdc...02125b8df772 [20:09:13] [02miraheze/puppet] 07paladox 0302125b8 - Switch base::syslog::syslog_host to graylog161 (#3706) [20:09:16] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [20:11:12] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 25% [20:11:24] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 146 bytes in 0.485 second response time [20:12:05] RECOVERY - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is OK: OK - NGINX Error Rate is 25% [20:13:25] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.660 second response time [20:14:03] PROBLEM - orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'orain.org' expires in 7 day(s) (Sat 03 Feb 2024 07:51:10 PM GMT +0000). [20:15:24] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 146 bytes in 0.510 second response time [20:15:57] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 41% [20:16:00] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 49% [20:17:25] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.625 second response time [20:17:44] PROBLEM - isv.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'orain.org' expires in 7 day(s) (Sat 03 Feb 2024 07:51:10 PM GMT +0000). [20:17:52] RECOVERY - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is OK: OK - NGINX Error Rate is 21% [20:20:27] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 146 bytes in 0.209 second response time [20:21:43] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 43% [20:22:27] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.579 second response time [20:23:30] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.91, 6.28, 5.64 [20:23:37] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [20:23:56] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 84% [20:24:14] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 40% [20:25:28] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.98, 6.11, 5.65 [20:25:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 49% [20:26:14] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [20:27:27] RECOVERY - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is OK: OK - NGINX Error Rate is 33% [20:28:00] [02miraheze/mw-config] 07Universal-Omega pushed 0328 commits to 03SpecialPage [+0/-0/±50] 13https://github.com/miraheze/mw-config/compare/b8ab7f5dd449...af043cd3d472 [20:28:01] [02miraheze/mw-config] 07Universal-Omega 03af043cd - Merge branch 'master' into SpecialPage [20:28:03] [02mw-config] 07Universal-Omega synchronize pull request 03#5435: Use SpecialPage instead of global $wgServer - 13https://github.com/miraheze/mw-config/pull/5435 [20:28:14] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 56% [20:28:57] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.19, 10.36, 8.68 [20:29:00] miraheze/mw-config - Universal-Omega the build passed. [20:29:34] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.50, 11.27, 8.88 [20:29:50] RECOVERY - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is OK: OK - NGINX Error Rate is 33% [20:29:55] PROBLEM - vmcodex.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vmcodex.net could not be found [20:30:56] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.16, 9.33, 8.51 [20:31:28] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.97, 10.19, 8.76 [20:32:15] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 38% [20:32:38] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:33:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/02125b8df772...50e12a4ea80c [20:33:29] [02miraheze/puppet] 07paladox 0350e12a4 - graylog: upgrade to 5.2.3 [20:36:15] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 91% [20:36:43] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [20:37:22] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 2 backends are down. mw131 mw132 [20:38:00] !log restart nginx on mw* to fix swiftproxy connection after swift-lb was changed [20:38:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:38:15] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 19% [20:38:36] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.11, 11.22, 8.92 [20:38:40] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3456 bytes in 0.664 second response time [20:39:09] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.84, 10.61, 9.07 [20:39:21] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [20:39:43] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.81, 10.44, 9.01 [20:40:35] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.97, 11.10, 9.17 [20:40:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/50e12a4ea80c...23c8bc383646 [20:40:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.44, 6.55, 5.93 [20:40:53] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.85, 11.09, 9.74 [20:40:54] [02miraheze/puppet] 07paladox 0323c8bc3 - graylog: increase heap to 3g [20:41:03] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.87, 9.47, 8.84 [20:41:12] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 18.88, 11.77, 9.08 [20:41:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/23c8bc383646...32725744022b [20:41:26] [02miraheze/puppet] 07paladox 033272574 - opensearch: increase heap to 3g [20:41:41] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.36, 10.41, 9.16 [20:41:54] PROBLEM - graylog131 Puppet on graylog131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:42:40] PROBLEM - ping6 on graylog131 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:42:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.97, 6.36, 5.94 [20:43:08] PROBLEM - graylog131 PowerDNS Recursor on graylog131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:43:10] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.22, 10.66, 8.99 [20:43:18] PROBLEM - graylog131 SSH on graylog131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:39] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 9.87, 9.87, 9.09 [20:43:56] PROBLEM - Host graylog131 is DOWN: PING CRITICAL - Packet loss = 100% [20:44:54] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.00, 10.69, 9.51 [20:44:57] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.33, 6.87, 6.29 [20:46:33] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.71, 10.18, 9.44 [20:46:49] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 9.20, 9.99, 9.39 [20:46:51] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.42, 9.69, 9.66 [20:46:57] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.11, 6.53, 6.24 [20:47:08] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 8.28, 9.46, 8.87 [20:49:31] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.13, 10.25, 9.49 [20:50:13] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.24, 10.97, 8.80 [20:50:43] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.25, 11.81, 10.32 [20:51:29] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.12, 9.97, 9.51 [20:52:13] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.78, 9.75, 8.63 [20:52:43] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.40, 11.18, 10.27 [20:53:47] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.61, 10.91, 10.24 [20:55:46] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.89, 10.47, 10.15 [20:56:43] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.61, 9.41, 9.81 [20:57:46] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.83, 9.71, 9.90 [21:02:18] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp35.miraheze.org - CNAME OK [21:02:57] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.13, 6.65, 6.28 [21:03:56] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [21:04:57] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.72, 6.65, 6.32 [21:05:18] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [21:05:55] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.707 second response time [21:06:40] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [21:07:08] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 17.50, 13.18, 10.38 [21:07:14] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.493 second response time [21:08:13] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 10.62, 11.15, 9.41 [21:08:40] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.493 second response time [21:09:04] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 9.41, 12.82, 10.32 [21:09:05] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.86, 11.83, 10.25 [21:10:13] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.24, 9.23, 8.91 [21:11:01] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.25, 9.82, 9.69 [21:11:03] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 5.91, 10.34, 9.71 [21:13:01] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.57, 8.95, 9.27 [21:13:33] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.39, 6.81, 6.37 [21:17:24] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.16, 6.73, 6.45 [21:22:11] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/commit/99572d08a9e6 [21:22:14] [02miraheze/puppet] 07Universal-Omega 0399572d0 - Delete hieradata/hosts/graylog131.yaml [21:22:15] [02puppet] 07Universal-Omega created branch 03decom - 13https://github.com/miraheze/puppet [21:22:18] [02puppet] 07Universal-Omega opened pull request 03#3707: Decom graylog131, os131, and os141 - 13https://github.com/miraheze/puppet/pull/3707 [21:22:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/99572d08a9e6...68795764d7da [21:22:44] [02miraheze/puppet] 07Universal-Omega 036879576 - Delete hieradata/hosts/os131.yaml [21:22:47] [02puppet] 07Universal-Omega synchronize pull request 03#3707: Decom graylog131, os131, and os141 - 13https://github.com/miraheze/puppet/pull/3707 [21:22:57] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/68795764d7da...5187150b3978 [21:23:00] [02miraheze/puppet] 07Universal-Omega 035187150 - Delete hieradata/hosts/os141.yaml [21:23:03] [02puppet] 07Universal-Omega synchronize pull request 03#3707: Decom graylog131, os131, and os141 - 13https://github.com/miraheze/puppet/pull/3707 [21:24:12] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5187150b3978...9650493ea591 [21:24:13] [02miraheze/puppet] 07Universal-Omega 039650493 - Update site.pp [21:24:16] [02puppet] 07Universal-Omega synchronize pull request 03#3707: Decom graylog131, os131, and os141 - 13https://github.com/miraheze/puppet/pull/3707 [21:27:30] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.22, 6.54, 6.33 [21:29:27] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 4.84, 6.02, 6.17 [21:33:23] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.02, 6.66, 6.40 [21:35:19] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.89, 6.40, 6.34 [21:35:40] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.26, 6.89, 6.60 [21:37:36] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.05, 6.52, 6.50 [21:39:15] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.37, 6.88, 6.54 [21:45:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.49, 7.69, 6.98 [21:47:02] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.52, 7.71, 7.08 [21:49:59] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.22, 7.26, 5.85 [21:51:51] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.57, 6.72, 6.80 [21:51:51] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.97, 7.43, 6.08 [21:52:20] [02puppet] 07Universal-Omega closed pull request 03#3707: Decom graylog131, os131, and os141 - 13https://github.com/miraheze/puppet/pull/3707 [21:52:23] [02miraheze/puppet] 07Universal-Omega deleted branch 03decom [21:52:25] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-3/±1] 13https://github.com/miraheze/puppet/compare/32725744022b...054d5c9fc88d [21:52:26] [02miraheze/puppet] 07Universal-Omega 03054d5c9 - Decom graylog131, os131, and os141 (#3707) [21:52:27] [02puppet] 07Universal-Omega deleted branch 03decom - 13https://github.com/miraheze/puppet [21:53:40] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/dns/commit/5c7216faa494 [21:53:43] [02miraheze/dns] 07Universal-Omega 035c7216f - Update 0.0.0.0.0.4.7.6.0.1.a.2.ip6.arpa [21:53:44] [02dns] 07Universal-Omega created branch 03decom - 13https://github.com/miraheze/dns [21:53:53] [02dns] 07Universal-Omega opened pull request 03#476: Decom graylog131, os131, and os141 - 13https://github.com/miraheze/dns/pull/476 [21:54:59] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/dns/compare/5c7216faa494...1fbb2e4d60e9 [21:55:01] [02miraheze/dns] 07Universal-Omega 031fbb2e4 - Update miraheze.org [21:55:03] [02dns] 07Universal-Omega synchronize pull request 03#476: Decom graylog131, os131, and os141 - 13https://github.com/miraheze/dns/pull/476 [21:56:47] [02dns] 07Universal-Omega closed pull request 03#476: Decom graylog131, os131, and os141 - 13https://github.com/miraheze/dns/pull/476 [21:56:48] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/52fbc0e5f9e2...7f6df5d12c8e [21:56:49] [02miraheze/dns] 07Universal-Omega 037f6df5d - Decom graylog131, os131, and os141 (#476) [21:56:52] [02miraheze/dns] 07Universal-Omega deleted branch 03decom [21:56:55] [02dns] 07Universal-Omega deleted branch 03decom - 13https://github.com/miraheze/dns [22:02:51] PROBLEM - os161 Puppet on os161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Opensearch_template[graylog-internal] [22:03:03] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/054d5c9fc88d...5822cb23d3e6 [22:03:05] [02miraheze/puppet] 07Universal-Omega 035822cb2 - Use test151 for memcached_servers_beta [22:05:38] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.29, 6.63, 6.46 [22:06:46] PROBLEM - os151 Puppet on os151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Opensearch_template[graylog-internal] [22:06:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5822cb23d3e6...294f5654b7ee [22:06:53] [02miraheze/puppet] 07paladox 03294f565 - swift: set conn_timeout to 1.5 for proxy-server [22:07:51] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mem [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/fd730d7fd2c8 [22:07:54] [02miraheze/puppet] 07Universal-Omega 03fd730d7 - Use mem151/mem161 on new infra [22:07:57] [02puppet] 07Universal-Omega created branch 03mem - 13https://github.com/miraheze/puppet [22:07:59] [02puppet] 07Universal-Omega opened pull request 03#3708: Use mem151/mem161 on new infra - 13https://github.com/miraheze/puppet/pull/3708 [22:08:36] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mem [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fd730d7fd2c8...f5ca6e34e869 [22:08:37] [02miraheze/puppet] 07Universal-Omega 03f5ca6e3 - Update mw131.yaml [22:08:39] [02puppet] 07Universal-Omega synchronize pull request 03#3708: Use mem151/mem161 on new infra - 13https://github.com/miraheze/puppet/pull/3708 [22:09:00] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mem [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f5ca6e34e869...e97db037a3ca [22:09:02] [02miraheze/puppet] 07Universal-Omega 03e97db03 - Update mw132.yaml [22:09:05] [02puppet] 07Universal-Omega synchronize pull request 03#3708: Use mem151/mem161 on new infra - 13https://github.com/miraheze/puppet/pull/3708 [22:09:36] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mem [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e97db037a3ca...868e43971b22 [22:09:37] [02miraheze/puppet] 07Universal-Omega 03868e439 - Update mw133.yaml [22:09:40] [02puppet] 07Universal-Omega synchronize pull request 03#3708: Use mem151/mem161 on new infra - 13https://github.com/miraheze/puppet/pull/3708 [22:09:56] RECOVERY - Host cp51 is UP: PING OK - Packet loss = 0%, RTA = 160.20 ms [22:10:05] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mem [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/868e43971b22...5e3c8f43bcb9 [22:10:07] [02miraheze/puppet] 07Universal-Omega 035e3c8f4 - Update mw134.yaml [22:10:07] RECOVERY - cp51 Nginx Backend for mw172 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8118 [22:10:08] RECOVERY - cp51 Nginx Backend for reports171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8205 [22:10:13] [02puppet] 07Universal-Omega synchronize pull request 03#3708: Use mem151/mem161 on new infra - 13https://github.com/miraheze/puppet/pull/3708 [22:10:18] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.82, 6.46, 6.01 [22:10:23] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mem [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5e3c8f43bcb9...58f388db716a [22:10:26] [02miraheze/puppet] 07Universal-Omega 0358f388d - Update mw141.yaml [22:10:29] [02puppet] 07Universal-Omega synchronize pull request 03#3708: Use mem151/mem161 on new infra - 13https://github.com/miraheze/puppet/pull/3708 [22:10:47] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mem [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/58f388db716a...2f6732ac6fec [22:10:50] [02miraheze/puppet] 07Universal-Omega 032f6732a - Update mw142.yaml [22:10:52] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 159.93 ms [22:10:52] [02puppet] 07Universal-Omega synchronize pull request 03#3708: Use mem151/mem161 on new infra - 13https://github.com/miraheze/puppet/pull/3708 [22:10:58] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3477 bytes in 1.313 second response time [22:11:02] RECOVERY - cp51 SSH on cp51 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [22:11:26] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mem [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2f6732ac6fec...283d5dcd3b4f [22:11:28] [02miraheze/puppet] 07Universal-Omega 03283d5dc - Update mw143.yaml [22:11:30] [02puppet] 07Universal-Omega synchronize pull request 03#3708: Use mem151/mem161 on new infra - 13https://github.com/miraheze/puppet/pull/3708 [22:11:53] RECOVERY - cp51 Nginx Backend for mw181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8119 [22:11:54] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mem [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/283d5dcd3b4f...122de9303575 [22:11:57] RECOVERY - cp51 Nginx Backend for mw142 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8109 [22:11:57] [02miraheze/puppet] 07Universal-Omega 03122de93 - Update mwtask141.yaml [22:11:59] [02puppet] 07Universal-Omega synchronize pull request 03#3708: Use mem151/mem161 on new infra - 13https://github.com/miraheze/puppet/pull/3708 [22:12:04] RECOVERY - cp51 Disk Space on cp51 is OK: DISK OK - free space: / 17615MiB (18% inode=98%); [22:12:16] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.40, 6.45, 6.06 [22:12:20] RECOVERY - cp51 Nginx Backend for puppet181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8204 [22:13:44] [02puppet] 07paladox reviewed pull request 03#3708 commit - 13https://github.com/miraheze/puppet/pull/3708#discussion_r1468195415 [22:16:16] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.49, 6.94, 6.33 [22:18:16] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.15, 6.77, 6.35 [22:18:49] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03jobchron171 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ceb55fbe662b [22:18:51] [02miraheze/puppet] 07Universal-Omega 03ceb55fb - Use jobchron171 on new infra [22:18:54] [02puppet] 07Universal-Omega created branch 03jobchron171 - 13https://github.com/miraheze/puppet [22:18:55] [02puppet] 07Universal-Omega opened pull request 03#3709: Use jobchron171 on new infra - 13https://github.com/miraheze/puppet/pull/3709 [22:19:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03jobchron171 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ceb55fbe662b...ad0a7abdc52e [22:19:35] [02miraheze/puppet] 07Universal-Omega 03ad0a7ab - Update mw152.yaml [22:19:36] [02puppet] 07Universal-Omega synchronize pull request 03#3709: Use jobchron171 on new infra - 13https://github.com/miraheze/puppet/pull/3709 [22:20:27] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03jobchron171 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ad0a7abdc52e...383386e7d42e [22:20:30] [02miraheze/puppet] 07Universal-Omega 03383386e - Update mw161.yaml [22:20:31] [02puppet] 07Universal-Omega synchronize pull request 03#3709: Use jobchron171 on new infra - 13https://github.com/miraheze/puppet/pull/3709 [22:20:55] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03jobchron171 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/383386e7d42e...952f5163a28a [22:20:57] [02miraheze/puppet] 07Universal-Omega 03952f516 - Update mw162.yaml [22:21:00] [02puppet] 07Universal-Omega synchronize pull request 03#3709: Use jobchron171 on new infra - 13https://github.com/miraheze/puppet/pull/3709 [22:21:19] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03jobchron171 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/952f5163a28a...4f081bc7384e [22:21:21] [02miraheze/puppet] 07Universal-Omega 034f081bc - Update mw171.yaml [22:21:23] [02puppet] 07Universal-Omega synchronize pull request 03#3709: Use jobchron171 on new infra - 13https://github.com/miraheze/puppet/pull/3709 [22:21:46] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03jobchron171 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4f081bc7384e...ac2aa71f1930 [22:21:49] [02miraheze/puppet] 07Universal-Omega 03ac2aa71 - Update mw172.yaml [22:21:50] [02puppet] 07Universal-Omega synchronize pull request 03#3709: Use jobchron171 on new infra - 13https://github.com/miraheze/puppet/pull/3709 [22:22:07] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03jobchron171 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ac2aa71f1930...5dbcab75965d [22:22:10] [02miraheze/puppet] 07Universal-Omega 035dbcab7 - Update mw181.yaml [22:22:12] [02puppet] 07Universal-Omega synchronize pull request 03#3709: Use jobchron171 on new infra - 13https://github.com/miraheze/puppet/pull/3709 [22:22:35] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03jobchron171 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5dbcab75965d...ebfa7cfddba1 [22:22:37] [02miraheze/puppet] 07Universal-Omega 03ebfa7cf - Update mw182.yaml [22:22:38] [02puppet] 07Universal-Omega synchronize pull request 03#3709: Use jobchron171 on new infra - 13https://github.com/miraheze/puppet/pull/3709 [22:22:58] PROBLEM - mw142 Disk Space on mw142 is WARNING: DISK WARNING - free space: / 2563MiB (10% inode=70%); [22:23:04] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03jobchron171 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ebfa7cfddba1...80986082f4d0 [22:23:07] [02miraheze/puppet] 07Universal-Omega 038098608 - Update mwtask181.yaml [22:23:08] [02puppet] 07Universal-Omega synchronize pull request 03#3709: Use jobchron171 on new infra - 13https://github.com/miraheze/puppet/pull/3709 [22:23:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03jobchron171 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/80986082f4d0...c4c9ced42541 [22:23:44] [02miraheze/puppet] 07Universal-Omega 03c4c9ced - Update jobchron171.yaml [22:23:47] [02puppet] 07Universal-Omega synchronize pull request 03#3709: Use jobchron171 on new infra - 13https://github.com/miraheze/puppet/pull/3709 [22:27:49] PROBLEM - mw134 Disk Space on mw134 is WARNING: DISK WARNING - free space: / 2606MiB (10% inode=70%); [22:28:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/9b3515c45d18 [22:28:23] [02miraheze/puppet] 07paladox 039b3515c - mediawiki: use a variable for swift-lb in nginx [22:28:26] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [22:28:27] [02puppet] 07paladox opened pull request 03#3710: mediawiki: use a variable for swift-lb in nginx - 13https://github.com/miraheze/puppet/pull/3710 [22:29:06] [02puppet] 07paladox closed pull request 03#3710: mediawiki: use a variable for swift-lb in nginx - 13https://github.com/miraheze/puppet/pull/3710 [22:29:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/294f5654b7ee...b359b2f8f0f8 [22:29:09] [02miraheze/puppet] 07paladox 03b359b2f - mediawiki: use a variable for swift-lb in nginx (#3710) [22:29:12] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [22:29:13] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [22:30:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2405dc3a71f1...f6a38c698574 [22:30:11] [02miraheze/mw-config] 07Universal-Omega 03f6a38c6 - Use private IP for jobchron171 [22:30:37] [02puppet] 07Universal-Omega closed pull request 03#3709: Use jobchron171 on new infra - 13https://github.com/miraheze/puppet/pull/3709 [22:30:39] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±10] 13https://github.com/miraheze/puppet/compare/b359b2f8f0f8...c0278236617f [22:30:41] [02miraheze/puppet] 07Universal-Omega 03c027823 - Use jobchron171 on new infra (#3709) [22:30:43] [02miraheze/puppet] 07Universal-Omega deleted branch 03jobchron171 [22:30:45] [02puppet] 07Universal-Omega deleted branch 03jobchron171 - 13https://github.com/miraheze/puppet [22:31:03] miraheze/mw-config - Universal-Omega the build passed. [22:31:44] !log [@mwtask181] starting deploy of {'config': True} to all [22:31:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:32:15] !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw151.wikitide.net [22:32:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:32:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c0278236617f...79ae99615250 [22:32:47] [02miraheze/puppet] 07paladox 0379ae996 - Fix [22:33:56] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [22:34:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:35:05] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 68s [22:35:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:38:34] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/79ae99615250...3c039e364eec [22:38:35] [02miraheze/puppet] 07paladox 032936789 - Revert "Fix" [22:38:38] [02miraheze/puppet] 07paladox 033c039e3 - Revert "mediawiki: use a variable for swift-lb in nginx (#3710)" [22:40:42] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [22:42:37] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.592 second response time [22:42:52] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.91, 10.48, 8.39 [22:43:43] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1505880058 secs [22:44:48] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.68, 9.70, 8.37 [22:46:16] [02puppet] 07Universal-Omega synchronize pull request 03#3708: Use mem151/mem161 on new infra - 13https://github.com/miraheze/puppet/pull/3708 [22:46:19] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mem [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/122de9303575...437eb2e4d6be [22:46:20] [02miraheze/puppet] 07Universal-Omega 03437eb2e - Use private IPs [22:47:17] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mem [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/437eb2e4d6be...af7d2e5441aa [22:47:20] [02miraheze/puppet] 07Universal-Omega 03af7d2e5 - Use private IPs [22:47:23] [02puppet] 07Universal-Omega synchronize pull request 03#3708: Use mem151/mem161 on new infra - 13https://github.com/miraheze/puppet/pull/3708 [22:48:52] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mem [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/af7d2e5441aa...333c1b7e34d9 [22:48:55] [02miraheze/puppet] 07Universal-Omega 03333c1b7 - Update [22:48:58] [02puppet] 07Universal-Omega synchronize pull request 03#3708: Use mem151/mem161 on new infra - 13https://github.com/miraheze/puppet/pull/3708 [22:49:10] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.30, 7.16, 6.51 [22:49:30] [02puppet] 07Universal-Omega reviewed pull request 03#3708 commit - 13https://github.com/miraheze/puppet/pull/3708#discussion_r1468217499 [22:49:55] [02puppet] 07Universal-Omega closed pull request 03#3708: Use mem151/mem161 on new infra - 13https://github.com/miraheze/puppet/pull/3708 [22:49:58] [02miraheze/puppet] 07Universal-Omega deleted branch 03mem [22:49:59] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±10] 13https://github.com/miraheze/puppet/compare/3c039e364eec...cf888d49bf9f [22:50:01] [02miraheze/puppet] 07Universal-Omega 03cf888d4 - Use mem151/mem161 on new infra (#3708) [22:50:02] [02puppet] 07Universal-Omega deleted branch 03mem - 13https://github.com/miraheze/puppet [22:50:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.61, 6.79, 6.21 [22:52:55] !log [@test151] starting deploy of {'config': True} to test151 [22:52:56] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [22:53:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:55:10] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.75, 7.65, 7.02 [22:56:31] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.09, 6.57, 6.33 [23:01:10] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.91, 6.60, 6.77 [23:02:33] PROBLEM - cp26 Current Load on cp26 is CRITICAL: LOAD CRITICAL - total load average: 13.74, 7.54, 4.32 [23:04:32] RECOVERY - cp26 Current Load on cp26 is OK: LOAD OK - total load average: 6.10, 6.79, 4.44 [23:05:03] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3342 bytes in 7.443 second response time [23:05:13] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10000 milliseconds [23:06:34] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3343 bytes in 7.477 second response time [23:07:00] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.589 second response time [23:07:10] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.24, 8.59, 7.47 [23:07:24] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: connect to address 2a10:6740::6:404 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:07:34] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw132 [23:07:44] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw132 [23:07:53] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw132 [23:08:05] PROBLEM - mw142 Disk Space on mw142 is CRITICAL: connect to address 2a10:6740::6:503 port 5666: Connection refusedconnect to host 2a10:6740::6:503 port 5666: Connection refused [23:08:05] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw132 [23:08:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.31, 6.53, 6.13 [23:08:31] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.166 second response time [23:08:34] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3343 bytes in 1.947 second response time [23:09:11] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.487 second response time [23:09:21] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.989 second response time [23:09:31] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [23:09:43] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [23:09:49] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [23:10:04] RECOVERY - mw142 Disk Space on mw142 is OK: DISK OK - free space: / 2766MiB (11% inode=70%); [23:10:06] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 22 backends are healthy [23:10:10] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.71, 6.65, 6.23 [23:10:32] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.918 second response time [23:11:10] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.48, 7.98, 7.49 [23:12:59] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/7f6df5d12c8e...7e09af380ef6 [23:13:01] [02miraheze/dns] 07paladox 037e09af3 - Set multi to true for swift-lb [23:13:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.42, 7.01, 6.48 [23:15:23] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw172.wikitide.net port 443 after 0 ms: Couldn't connect to server [23:15:30] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw141.miraheze.org port 443 after 121 ms: Couldn't connect to server [23:17:21] PROBLEM - mw172 HTTPS on mw172 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.063 second response time [23:17:30] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.503 second response time [23:19:55] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.05, 6.65, 6.51 [23:23:10] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.56, 5.70, 6.56 [23:27:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/47464fb3ee3d [23:27:01] [02miraheze/puppet] 07paladox 0347464fb - varnish: load balance swift and send request directly to swiftproxy [23:27:02] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [23:27:04] [02puppet] 07paladox opened pull request 03#3711: varnish: load balance swift and send request directly to swiftproxy - 13https://github.com/miraheze/puppet/pull/3711 [23:29:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/47464fb3ee3d...ba7ddff11e9a [23:29:04] [02miraheze/puppet] 07paladox 03ba7ddff - Update common.yaml [23:29:07] [02puppet] 07paladox synchronize pull request 03#3711: varnish: load balance swift and send request directly to swiftproxy - 13https://github.com/miraheze/puppet/pull/3711 [23:29:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ba7ddff11e9a...e0462d57d593 [23:29:45] [02miraheze/puppet] 07paladox 03e0462d5 - Update swift.conf [23:29:47] [02puppet] 07paladox synchronize pull request 03#3711: varnish: load balance swift and send request directly to swiftproxy - 13https://github.com/miraheze/puppet/pull/3711 [23:31:36] [02puppet] 07paladox synchronize pull request 03#3711: varnish: load balance swift and send request directly to swiftproxy - 13https://github.com/miraheze/puppet/pull/3711 [23:31:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e0462d57d593...2c4fe737e0fa [23:31:40] [02miraheze/puppet] 07paladox 032c4fe73 - Update common.yaml [23:34:03] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 200 - 320 bytes in 0.012 second response time [23:35:10] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.31, 6.34, 6.24 [23:37:39] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/01da871daa83 [23:37:42] [02miraheze/dns] 07paladox 0301da871 - Add swiftproxy161/171 to miraheze.org for alias [23:37:44] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [23:37:46] [02dns] 07paladox opened pull request 03#477: Add swiftproxy161/171 to miraheze.org for alias - 13https://github.com/miraheze/dns/pull/477 [23:37:48] [02dns] 07paladox closed pull request 03#477: Add swiftproxy161/171 to miraheze.org for alias - 13https://github.com/miraheze/dns/pull/477 [23:37:49] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/7e09af380ef6...a548bcf7fd15 [23:37:52] [02miraheze/dns] 07paladox 03a548bcf - Add swiftproxy161/171 to miraheze.org for alias (#477) [23:37:53] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [23:37:54] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [23:38:03] PROBLEM - mw151 HTTPS on mw151 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.060 second response time [23:39:10] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.02, 6.61, 6.42 [23:39:23] [02puppet] 07paladox closed pull request 03#3711: varnish: load balance swift and send request directly to swiftproxy - 13https://github.com/miraheze/puppet/pull/3711 [23:39:26] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [23:39:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/cf888d49bf9f...dc171c8a9bfa [23:39:30] [02miraheze/puppet] 07paladox 03dc171c8 - varnish: load balance swift and send request directly to swiftproxy (#3711) [23:39:33] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [23:40:55] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.90, 6.71, 6.41 [23:42:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dc171c8a9bfa...b0a6bd953e95 [23:42:33] [02miraheze/puppet] 07paladox 03b0a6bd9 - swift: allow varnish to access [23:42:51] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.02, 6.44, 6.34 [23:49:10] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.72, 7.41, 6.77 [23:51:10] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.22, 7.26, 6.79 [23:57:09] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.70, 6.52, 6.64