[00:17:38] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 69496MiB (15% inode=99%); [03:50:56] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [03:50:59] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [03:51:01] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:51:04] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:51:06] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:51:07] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [03:51:07] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:51:07] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.023 second response time [03:51:08] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:51:08] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:51:11] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.017 second response time [03:51:11] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:51:15] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.013 second response time [03:51:17] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:51:18] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:51:23] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:51:23] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [03:51:26] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [03:51:28] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 174.37, 77.60, 30.42 [03:51:40] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [03:51:44] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [03:51:44] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:51:46] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:51:53] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 6 backends are down. mw151 mw162 mw153 mw163 mw192 mw193 [03:52:03] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [03:52:08] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:52:18] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:52:23] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:52:24] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:52:25] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 6 backends are down. mw151 mw162 mw153 mw163 mw192 mw193 [03:52:26] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:52:27] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:52:27] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:52:28] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:52:29] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [03:52:34] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:52:35] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [03:52:36] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 6 backends are down. mw151 mw162 mw153 mw163 mw192 mw193 [03:52:49] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [03:52:50] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [03:52:50] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [03:53:01] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:53:03] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [03:53:03] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [03:53:21] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [03:54:07] PROBLEM - db171 APT on db171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:55:10] PROBLEM - db171 MariaDB Connections on db171 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: MySQL server has gone away in /usr/lib/nagios/plugins/check_mysql_connections.php:74Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(74): mysqli->query('SHOW STATUS WHE...')#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 74Fatal error: Uncaught mysqli_sql_exception: MySQL server has g [03:55:11] n /usr/lib/nagios/plugins/check_mysql_connections.php:74Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(74): mysqli->query('SHOW STATUS WHE...')#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 74 [03:55:11] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 1.755 second response time [03:55:12] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.266 second response time [03:55:14] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.306 second response time [03:55:15] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 3.525 second response time [03:55:16] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.090 second response time [03:55:18] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 1.130 second response time [03:55:19] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.825 second response time [03:55:23] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 0.056 second response time [03:55:23] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.208 second response time [03:55:32] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.202 second response time [03:55:40] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.197 second response time [03:55:44] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.069 second response time [03:55:44] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.193 second response time [03:55:53] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.067 second response time [03:55:54] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [03:56:03] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.208 second response time [03:56:08] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.068 second response time [03:56:10] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.176 second response time [03:56:18] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.060 second response time [03:56:19] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.064 second response time [03:56:21] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.204 second response time [03:56:22] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.175 second response time [03:56:25] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [03:56:27] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.206 second response time [03:56:27] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.067 second response time [03:56:34] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.063 second response time [03:56:34] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 0.059 second response time [03:56:35] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4328 bytes in 0.064 second response time [03:56:36] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [03:56:50] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.198 second response time [03:56:50] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.203 second response time [03:56:53] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.061 second response time [03:56:55] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.221 second response time [03:56:57] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.060 second response time [03:56:58] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.058 second response time [03:57:01] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.198 second response time [03:57:02] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.066 second response time [03:57:03] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.205 second response time [03:57:07] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.064 second response time [03:57:08] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.191 second response time [03:57:08] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.058 second response time [03:57:09] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.063 second response time [03:57:09] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.053 second response time [03:57:21] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 14% [03:58:00] PROBLEM - db171 Puppet on db171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:58:55] shut up [03:59:06] PROBLEM - db171 MariaDB on db171 is CRITICAL: Can't connect to server on 'db171.fsslc.wtnet' (115) [04:09:12] PROBLEM - db171 PowerDNS Recursor on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: Connection refusedconnect to host 10.0.17.119 port 5666: Connection refused [04:09:26] PROBLEM - db171 Disk Space on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: Connection refusedconnect to host 10.0.17.119 port 5666: Connection refused [04:09:28] PROBLEM - db171 conntrack_table_size on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: Connection refusedconnect to host 10.0.17.119 port 5666: Connection refused [04:09:54] PROBLEM - db171 Check unit status of sql-backup on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: Connection refusedconnect to host 10.0.17.119 port 5666: Connection refused [04:10:08] PROBLEM - db171 ferm_active on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: Connection refusedconnect to host 10.0.17.119 port 5666: Connection refused [04:10:36] PROBLEM - db171 NTP time on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: Connection refusedconnect to host 10.0.17.119 port 5666: Connection refused [04:10:37] PROBLEM - db171 SSH on db171 is CRITICAL: connect to address 10.0.17.119 and port 22: Connection refused [04:12:58] I wonder if there is a problem 🤔 [04:18:08] RECOVERY - db171 ferm_active on db171 is OK: OK ferm input default policy is set [04:18:10] RECOVERY - db171 Check unit status of sql-backup on db171 is OK: OK: Status of the systemd unit sql-backup [04:18:37] RECOVERY - db171 NTP time on db171 is OK: NTP OK: Offset -0.002748459578 secs [04:18:37] RECOVERY - db171 SSH on db171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [04:18:52] RECOVERY - db171 MariaDB on db171 is OK: Uptime: 42 Threads: 218 Questions: 27356 Slow queries: 0 Opens: 1502 Open tables: 1502 Queries per second avg: 651.333 [04:18:56] !log [void@db171] reset db171 [04:19:00] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 30 minutes ago with 0 failures [04:19:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:19:06] RECOVERY - db171 PowerDNS Recursor on db171 is OK: DNS OK: 0.065 seconds response time. db171.fsslc.wtnet returns 10.0.17.119 [04:19:11] RECOVERY - db171 APT on db171 is OK: APT OK: 119 packages available for upgrade (0 critical updates). [04:19:23] RECOVERY - db171 conntrack_table_size on db171 is OK: OK: nf_conntrack is 0 % full [04:19:26] RECOVERY - db171 Disk Space on db171 is OK: DISK OK - free space: / 415278MiB (46% inode=97%); [04:20:24] RECOVERY - db171 MariaDB Connections on db171 is OK: OK connection usage: 0.7%Current connections: 7 [04:21:27] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.34, 6.31, 2.88 [07:45:18] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/bc8576e1b18062c80762fa0eb7bfb6edda8648c4 [07:45:18] 02ssl/03main 07WikiTideBot 03bc8576e Bot: Auto-update domain lists [09:33:38] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49761MiB (10% inode=99%);