[00:14:28] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 52387MiB (11% inode=99%); [01:25:38] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.72, 6.54, 5.96 [01:27:36] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.69, 6.24, 5.93 [01:38:28] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49774MiB (10% inode=99%); [04:21:44] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 249.91, 83.44, 30.81 [04:21:58] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [04:21:58] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [04:22:09] PROBLEM - mw164 MediaWiki Rendering on mw164 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:22:09] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [04:22:09] PROBLEM - mw184 MediaWiki Rendering on mw184 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:22:10] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:22:11] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [04:22:12] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:22:19] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [04:22:19] PROBLEM - mw184 HTTPS on mw184 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [04:22:27] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:22:28] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 15 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw174 mw183 mw184 [04:22:32] PROBLEM - mw164 HTTPS on mw164 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [04:23:57] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.345 second response time [04:23:58] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4108 bytes in 0.072 second response time [04:24:04] PROBLEM - db181 APT on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:24:04] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.177 second response time [04:24:05] RECOVERY - mw164 MediaWiki Rendering on mw164 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.154 second response time [04:24:05] RECOVERY - mw184 MediaWiki Rendering on mw184 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.187 second response time [04:24:07] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.963 second response time [04:24:08] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.052 second response time [04:24:10] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.061 second response time [04:24:11] PROBLEM - db181 MariaDB on db181 is CRITICAL: Received error packet before completion of TLS handshake. The authenticity of the following error cannot be verified: 1040 - Too many connections [04:24:14] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.060 second response time [04:24:14] RECOVERY - mw184 HTTPS on mw184 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.064 second response time [04:24:15] PROBLEM - db181 MariaDB Connections on db181 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Too many connections in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne [04:24:15] on line 66Fatal error: Uncaught mysqli_sql_exception: Too many connections in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [04:24:24] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.237 second response time [04:24:28] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [04:24:29] RECOVERY - mw164 HTTPS on mw164 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.054 second response time [04:27:30] PROBLEM - db181 Puppet on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:57:38] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 57 minutes ago with 0 failures [04:58:42] RECOVERY - db181 APT on db181 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [04:59:54] RECOVERY - db181 MariaDB Connections on db181 is OK: OK connection usage: 0.2%Current connections: 2 [05:00:09] RECOVERY - db181 MariaDB on db181 is OK: Uptime: 134 Threads: 2 Questions: 23349 Slow queries: 0 Opens: 1960 Open tables: 1954 Queries per second avg: 174.246 [05:01:42] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.52, 0.11, 0.04 [11:25:45] PROBLEM - ru-teirailway.f5.si - Cloudflare on sslhost is CRITICAL: connect to address ru-teirailway.f5.si and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [12:08:11] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:08:28] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 11 backends are down. mw151 mw152 mw161 mw162 mw182 mw153 mw154 mw164 mw173 mw174 mw183 [12:09:41] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 10 backends are down. mw151 mw152 mw161 mw162 mw172 mw181 mw182 mw153 mw173 mw174 [12:10:10] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.171 second response time [12:11:25] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [12:16:28] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27032MiB (5% inode=99%); [12:17:03] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:17:31] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4108 bytes in 2.766 second response time [12:20:58] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4108 bytes in 0.423 second response time [12:21:33] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:24:58] PROBLEM - mw154 HTTPS on mw154 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:27:03] RECOVERY - mw154 HTTPS on mw154 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 8.452 second response time [12:28:13] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [12:29:09] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.013 second response time [12:29:34] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:46] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4108 bytes in 2.928 second response time [12:29:58] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 19.26, 12.43, 8.61 [12:31:08] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.424 second response time [12:31:13] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:31:40] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.153 second response time [12:31:44] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:21] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 2.049 second response time [12:33:41] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.989 second response time [12:34:34] !log [paladox@mwtask181] starting deploy of {'config': True} to all [12:34:44] !log [paladox@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw153.wikitide.net [12:34:52] !log [paladox@mwtask181] starting deploy of {'config': True, 'force': True} to all [12:35:35] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.168 second response time [12:35:39] PROBLEM - mw154 MediaWiki Rendering on mw154 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.069 second response time [12:35:55] !log [paladox@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 62s [12:35:58] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 4.28, 10.46, 9.36 [12:36:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:36:11] !log [paladox@mwtask181] starting deploy of {'config': True, 'force': True} to all [12:36:28] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [12:36:31] !log [paladox@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 19s [12:37:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:06] PROBLEM - mw164 MediaWiki Rendering on mw164 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 6.159 second response time [12:37:07] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 7.303 second response time [12:37:58] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 15.20, 13.55, 10.66 [12:39:29] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4108 bytes in 0.057 second response time [12:39:30] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.221 second response time [12:39:37] RECOVERY - mw154 MediaWiki Rendering on mw154 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.195 second response time [12:39:41] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [12:40:56] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.205 second response time [12:40:58] RECOVERY - mw164 MediaWiki Rendering on mw164 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.188 second response time [12:41:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:41:58] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 4.82, 11.85, 10.91 [12:43:58] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.71, 8.79, 9.90 [13:01:36] RECOVERY - db171 Backups SQL on db171 is OK: FILE_AGE OK: /var/log/db-backups/db-backups/db-backups.log is 46 seconds old and 1199633 bytes [13:12:09] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': ['1.42', '1.43'], 'upgrade_extensions': 'CommentStreams'} to all [13:12:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:13:47] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.42', '1.43', '1.44'], 'upgrade_extensions': 'CommentStreams'} to test151 [13:13:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:15:18] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': ['1.42', '1.43'], 'upgrade_extensions': 'CommentStreams'} to all - SUCCESS in 189s [13:15:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:16:47] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.42', '1.43', '1.44'], 'upgrade_extensions': 'CommentStreams'} to test151 - SUCCESS in 180s [13:16:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:43:50] PROBLEM - db182 Disk Space on db182 is CRITICAL: DISK CRITICAL - free space: / 0MiB (0% inode=99%); [13:45:50] RECOVERY - db182 Disk Space on db182 is OK: DISK OK - free space: / 140332MiB (30% inode=99%); [14:10:51] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/435ad06e2a55149dc030ad624d1ece939a7ec548 [14:10:52] 02puppet/03master 07paladox 03435ad06 matomo: update to 5.2.1 [14:11:16] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 100% [14:13:16] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 141.94 ms [15:16:22] PROBLEM - starveil.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [16:27:36] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.87, 6.87, 6.10 [16:29:35] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 6.12, 6.37, 6.00 [19:08:28] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/057056721c5a2a61d4752d389ba25ac45185b5ca [19:08:28] 02puppet/03master 07paladox 030570567 mattermost: upgrade to 10.4.1 [20:34:37] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/0467d4a4bcc7 [20:34:37] 02puppet/03paladox-patch-2 07paladox 030467d4a wikitide-backup: Don't run mysqldump with -C… [20:34:41] [02puppet] 07paladox opened pull request #4139: wikitide-backup: Don't run mysqldump with -C (03master...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4139 [20:34:48] [02puppet] 07coderabbitai[bot] commented on pull request #4139: --- […] 13https://github.com/miraheze/puppet/pull/4139#issuecomment-2601012254 [20:35:02] [02puppet] 07paladox merged pull request #4139: wikitide-backup: Don't run mysqldump with -C (03master...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4139 [20:35:02] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/a534bafd796a736cdf7fd96cd566fb8630612efe [20:35:02] 02puppet/03master 07paladox 03a534baf wikitide-backup: Don't run mysqldump with -C (#4139)… [20:35:03] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 030467d4a 13https://github.com/miraheze/puppet/commit/0467d4a [20:47:12] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/2b6c9d42be55 [20:47:12] 02puppet/03paladox-patch-2 07paladox 032b6c9d4 wikitide-backup: Set -c for sql backup… [20:47:16] [02puppet] 07paladox opened pull request #4140: wikitide-backup: Set -c for sql backup (03master...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4140 [20:47:23] [02puppet] 07coderabbitai[bot] commented on pull request #4140: --- […] 13https://github.com/miraheze/puppet/pull/4140#issuecomment-2601016104 [20:47:41] [02puppet] 07paladox merged pull request #4140: wikitide-backup: Set -c for sql backup (03master...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4140 [20:47:41] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/7ee8a524e1e11aa71317d59a813b166e30374264 [20:47:41] 02puppet/03master 07paladox 037ee8a52 wikitide-backup: Set -c for sql backup (#4140)… [20:47:43] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 032b6c9d4 13https://github.com/miraheze/puppet/commit/2b6c9d4 [21:53:17] [02puppet] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/f2ce677d0ebd20e7bb51871db9e418e01225b6bc [21:53:17] 02puppet/03master 07CosmicAlpha 03f2ce677 Remove evalprime access for activity [21:54:35] miraheze/puppet - Universal-Omega the build passed. [22:47:02] [02mw-config] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/a6ee2195120e2b57d53fde84f4bf51f6df38cfb0 [22:47:03] 02mw-config/03master 07MacFan4000 03a6ee219 rm Evalprime access to staffwiki [22:47:55] !log [macfan@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [22:47:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:47:58] miraheze/mw-config - MacFan4000 the build passed. [22:48:17] !log [macfan@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 21s [22:48:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:32] PROBLEM - sytbay.site - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sytbay.site' expires in 14 day(s) (Mon 03 Feb 2025 10:49:41 PM GMT +0000). [23:03:44] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/4b81765313e7fe3d93ed92ad20eba3f8b5ecc8cd [23:03:44] 02ssl/03master 07WikiTideSSLBot 034b81765 Bot: Update SSL cert for sytbay.site [23:32:54] RECOVERY - sytbay.site - LetsEncrypt on sslhost is OK: OK - Certificate 'sytbay.site' will expire on Sat 19 Apr 2025 10:05:07 PM GMT +0000.