[00:05:01] RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 58916MiB (12% inode=99%); [00:15:54] !log mhglobal: START TRANSACTION; [00:15:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:05] !log mhglobal: GlobalBlocking: patch-globalblocks-add-gb_autoblock_parent_id.sql [00:16:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:47] !log Got "ERROR 1060 (42S21): Duplicate column name 'gb_autoblock_parent_id'" and "ERROR 1061 (42000): Duplicate key name 'gb_autoblock_parent_id'", presumably already ran [00:16:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:56] !log mhglobal: GlobalBlocking: patch-globalblocks-add-gb_create_account.sql [00:16:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:07] !log Got "ERROR 1060 (42S21): Duplicate column name 'gb_create_account'", presumably already ran [00:17:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:20] !log mhglobal: GlobalBlocking: patch-globalblocks-add-gb_enable_autoblock.sql [00:17:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:28] !log "ERROR 1060 (42S21): Duplicate column name 'gb_enable_autoblock'" [00:17:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:34] !log mhglobal: GlobalBlocking: patch-globalblocks-drop-gb_by.sql [00:17:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:43] !log "ERROR 1091 (42000): Can't DROP COLUMN `gb_by`; check that it exists" [00:17:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:49] !log mhglobal: GlobalBlocking: patch-globalblocks-modify-gb_address-index.sql [00:17:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:05] !log "ERROR 1091 (42000): Can't DROP INDEX `gb_address`; check that it exists" and "ERROR 1061 (42000): Duplicate key name 'gb_address_autoblock_parent_id'" [00:18:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:13] !log mhglobal: GlobalBlocking: patch-globalblocks-modify-gb_autoblock_parent_id-default.sql [00:18:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:23] !log "Query OK, 0 rows affected (0.105 sec)" [00:18:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:36] !log mhglobal: GlobalBlocking: patch-global_block_whitelist-default-gbw_address.sql [00:18:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:45] !log "Query OK, 0 rows affected (0.004 sec)" [00:18:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:50] !log mhglobal: COMMIT; [00:18:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:03] !log [blankeclair@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php sql /srv/mediawiki/1.43/extensions/GlobalBlocking/sql/mysql/patch-global_block_whitelist-default-gbw_address.sql (START) [00:30:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:56:46] PROBLEM - www.journeytheword.wiki - Cloudflare on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [00:57:18] PROBLEM - journeytheword.wiki - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [02:21:49] RECOVERY - vylet.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'vylet.wiki' will expire on Tue 17 Jun 2025 02:10:16 AM GMT +0000. [02:22:47] PROBLEM - Host mattermost1 is DOWN: PING CRITICAL - Packet loss = 100% [02:25:13] RECOVERY - Host mattermost1 is UP: PING WARNING - Packet loss = 96%, RTA = 29.94 ms [02:41:29] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/d8e153d710ca678e078aa894c26cc9209327f9cf [02:41:29] 02ssl/03master 07MacFan4000 03d8e153d switch vylet.wiki to cloudflare. [02:41:35] [02dns] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/c5ae081bc959b9c4ab5581edcea3f539d9483cf3 [02:41:36] 02dns/03master 07MacFan4000 03c5ae081 Remove zone for vylet.wiki [02:42:29] miraheze/dns - MacFan4000 the build passed. [02:44:30] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/dfd78ee02de87c9fb9682fdfb163685b9c23977b [02:44:30] 02ssl/03master 07MacFan4000 03dfd78ee switch worldsanskrit.net to cloudflare. [02:44:34] [02dns] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/67d25c6ef5f8205fc78a336bf356ad374478939e [02:44:34] 02dns/03master 07MacFan4000 0367d25c6 Remove zone for worldsanskrit.net [02:45:29] miraheze/dns - MacFan4000 the build passed. [02:47:14] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.65, 15.46, 9.78 [03:09:14] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 8.25, 19.47, 20.08 [03:16:12] !log [blankeclair@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php sql /srv/mediawiki/1.43/extensions/GlobalBlocking/sql/mysql/patch-global_block_whitelist-default-gbw_address.sql (END - exit=0) [03:16:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:19:04] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.28, 17.76, 17.43 [03:21:01] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 6.78, 13.48, 15.92 [04:14:39] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 29.89, 20.67, 12.33 [04:18:39] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 15.25, 21.84, 15.01 [04:20:39] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 7.91, 16.60, 13.89 [04:28:39] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.31, 18.84, 14.98 [04:40:39] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 14.49, 23.14, 20.59 [04:42:39] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 6.52, 17.49, 18.83 [04:46:39] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.81, 23.02, 20.76 [05:10:39] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 13.14, 19.26, 23.14 [05:11:12] PROBLEM - spiceandwolf.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [05:16:39] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 11.63, 14.16, 19.58 [05:53:45] [02mw-config] 07BlankEclair opened pull request #5876: T13386: Fix PHP warnings regarding JsonConfig on ftlmultiversewiki (07miraheze:03master...07BlankEclair:03T13386) 13https://github.com/miraheze/mw-config/pull/5876 [05:54:40] miraheze/mw-config - BlankEclair the build passed. [05:54:44] [02mw-config] 07BlankEclair merged pull request #5876: T13386: Fix PHP warnings regarding JsonConfig on ftlmultiversewiki (07miraheze:03master...07BlankEclair:03T13386) 13https://github.com/miraheze/mw-config/pull/5876 [05:54:44] [02mw-config] 07BlankEclair pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/71b214f1b3844bc3489ed12fa0da45a6d13c012e [05:54:44] 02mw-config/03master 07Claire 0371b214f T13386: Fix PHP warnings regarding JsonConfig on ftlmultiversewiki (#5876) [05:54:46] !log [blankeclair@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [05:54:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:55:08] !log [blankeclair@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 21s [05:55:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:55:40] miraheze/mw-config - BlankEclair the build passed. [07:40:03] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:40:08] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [07:40:10] PROBLEM - cp38 HTTPS on cp38 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [07:40:16] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [07:40:19] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 187.24, 79.45, 30.54 [07:40:21] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [07:40:26] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:40:31] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:40:33] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [07:40:46] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [07:40:49] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:40:49] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [07:40:52] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [07:40:53] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:40:54] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:40:58] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:41:01] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [07:41:03] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [07:41:06] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:41:06] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [07:41:16] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [07:41:24] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:24] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:25] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 49% [07:41:27] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [07:41:28] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:28] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [07:41:29] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [07:41:29] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [07:41:40] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:44] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [07:41:49] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:57] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [07:42:03] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 2.155 second response time [07:42:07] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [07:42:10] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [07:42:15] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [07:43:08] PROBLEM - db181 Puppet on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:43:21] PROBLEM - db181 MariaDB Connections on db181 is UNKNOWN: [07:43:25] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: CRITICAL - NGINX Error Rate is 80% [07:44:03] PROBLEM - db181 MariaDB on db181 is UNKNOWN: [07:44:33] PROBLEM - db181 APT on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:44:36] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 9.219 second response time [07:44:38] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 2.341 second response time [07:44:38] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.152 second response time [07:44:53] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.157 second response time [07:44:53] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.051 second response time [07:44:53] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.056 second response time [07:44:54] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.055 second response time [07:44:54] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.196 second response time [07:44:55] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.167 second response time [07:45:06] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.185 second response time [07:45:06] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.057 second response time [07:45:09] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.061 second response time [07:45:09] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.164 second response time [07:45:13] PROBLEM - mwtask181 jobrunner.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10003 milliseconds [07:45:17] PROBLEM - mwtask161 jobrunner.wikitide.net HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10002 milliseconds [07:45:25] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 9% [07:45:26] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.052 second response time [07:45:26] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.165 second response time [07:45:26] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.158 second response time [07:45:26] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.157 second response time [07:45:27] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4092 bytes in 0.061 second response time [07:45:29] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 31 backends are healthy [07:45:30] PROBLEM - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10003 milliseconds [07:45:30] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.052 second response time [07:45:38] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.164 second response time [07:45:44] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4092 bytes in 0.062 second response time [07:45:48] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:45:51] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [07:45:53] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.163 second response time [07:45:56] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:45:59] PROBLEM - db181 MariaDB on db181 is CRITICAL: Too many connections [07:46:01] PROBLEM - mwtask161 HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [07:46:06] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.057 second response time [07:46:06] RECOVERY - cp38 HTTPS on cp38 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4092 bytes in 0.060 second response time [07:46:08] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.146 second response time [07:46:08] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.066 second response time [07:46:10] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.057 second response time [07:46:19] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.153 second response time [07:46:22] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.058 second response time [07:46:29] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.175 second response time [07:46:36] PROBLEM - mwtask161 jobrunner-high.wikitide.net HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10000 milliseconds [07:46:36] PROBLEM - mwtask161 videoscaler.wikitide.net HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10000 milliseconds [07:46:57] PROBLEM - mwtask181 jobrunner-high.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10003 milliseconds [07:47:01] RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy [07:47:16] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy [07:57:05] RECOVERY - mwtask161 jobrunner-high.wikitide.net HTTPS on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.010 second response time [07:57:05] RECOVERY - mwtask161 videoscaler.wikitide.net HTTPS on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.011 second response time [07:57:30] RECOVERY - mwtask181 jobrunner.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.009 second response time [07:57:32] RECOVERY - mwtask181 jobrunner-high.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.012 second response time [07:57:42] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 28 minutes ago with 0 failures [07:57:43] RECOVERY - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 0.028 second response time [07:57:53] RECOVERY - mwtask161 jobrunner.wikitide.net HTTPS on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.012 second response time [07:58:08] RECOVERY - db181 APT on db181 is OK: APT OK: 32 packages available for upgrade (0 critical updates). [07:58:11] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3972 bytes in 0.063 second response time [07:58:14] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.176 second response time [07:58:16] RECOVERY - mwtask161 HTTPS on mwtask161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3972 bytes in 0.052 second response time [07:58:25] !log attempting to restart MySQL on dn181 [07:58:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:58:39] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.204 second response time [07:58:51] RECOVERY - db181 MariaDB Connections on db181 is OK: OK connection usage: 0.8%Current connections: 8 [07:59:35] RECOVERY - db181 MariaDB on db181 is OK: Uptime: 99 Threads: 10 Questions: 69950 Slow queries: 0 Opens: 4005 Open tables: 3867 Queries per second avg: 706.565 [07:59:52] [02mw-config] 07RhinosF1 pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/651a96efc91b4d2ec668e1035ec921b824d46704 [07:59:52] 02mw-config/03master 07RhinosF1 03651a96e Update LocalSettings.php [08:00:48] miraheze/mw-config - RhinosF1 the build passed. [08:01:03] [02mw-config] 07RhinosF1 pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/c89aeb1dc200f261388ae32d602d1f7a0ac9776a [08:01:03] 02mw-config/03master 07RhinosF1 03c89aeb1 Update LocalSettings.php [08:01:56] miraheze/mw-config - RhinosF1 the build passed. [08:28:44] @rhinosf1 what happened to c4? [08:29:08] Ran out of connections [08:29:13] Had to restart it [08:29:28] I'll see if I can fix that more tomorrow. May see what we can do to increase that. [08:29:40] @cosmicalpha we don't need to increase it [08:29:49] I couldn't see anything useful in logs though [08:29:50] It happens to often. [08:29:57] It's a bug [08:30:04] Increasing it won't stop the problem [08:30:28] I couldn't grab too much useful debugging info cause I was still half asleep and on my phone [08:30:29] Happened twice in the last couple weeks and once is to much for me. So we need to investigate it then. [08:30:34] yes [08:30:47] Extremely difficult to investigate [08:31:33] Bugs like this are bloody evil [08:31:34] is it a mariadb bug or a mediawiki bug spamming to many connections? [08:31:56] Anything in php slowlogs? [08:32:01] of value [08:32:49] Thats how I've debugged issues with the db before because often times very slow connections are way to much db stuff. [08:33:39] Could be either [08:34:06] I'll look through graylog shortly [08:34:44] Are slowlogs on graylog? [08:34:54] I thought they were just on local disk. [08:35:25] oh no I did add to Graylog a long time ago but it doesn't format right on Graylog so each line comes across as a new message on Graylog. [08:35:46] I need to look at fixing that. [08:35:55] Add a formatter for it. [08:52:18] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.64, 0.89, 11.61 [08:56:18] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 1.44, 1.15, 9.26 [09:35:01] !log [rhinos@db181] [09:35:56] !log running mysqlcheck in a screen from salt-ssh on all dbs [09:36:26] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:36:31] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:36:33] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:36:35] oh [09:36:37] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:36:39] that's not good [09:36:46] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:36:49] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:36:49] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:36:53] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.016 second response time [09:36:53] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:36:54] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:36:58] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:37:00] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:01] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [09:37:03] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:37:04] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:06] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:37:06] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:37:14] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.013 second response time [09:37:14] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.015 second response time [09:37:14] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:37:16] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [09:37:18] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:37:18] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:37:19] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:37:25] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 58% [09:37:27] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [09:37:28] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:37:28] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.017 second response time [09:37:29] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.014 second response time [09:37:29] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [09:37:30] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:37:32] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:37:43] !log [rhinos@puppet181] rhinosf1 [09:37:44] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 96% [09:37:44] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [09:37:46] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:37:48] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:49] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:55] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:37:57] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:38:03] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 483.55, 221.89, 89.49 [09:38:04] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:06] PROBLEM - cp38 HTTPS on cp38 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [09:38:07] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:38:08] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:38:10] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:38:13] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:15] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:38:18] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:18] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:39:32] PROBLEM - db171 APT on db171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:39:51] PROBLEM - db171 Puppet on db171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:41:51] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 6.059 second response time [09:41:56] PROBLEM - db171 MariaDB on db171 is CRITICAL: Received error packet before completion of TLS handshake. The authenticity of the following error cannot be verified: 1040 - Too many connections [09:41:58] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.058 second response time [09:42:03] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.053 second response time [09:42:06] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.058 second response time [09:42:07] RECOVERY - cp38 HTTPS on cp38 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4092 bytes in 1.395 second response time [09:42:08] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.058 second response time [09:42:10] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.060 second response time [09:42:12] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.050 second response time [09:42:31] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.056 second response time [09:42:38] PROBLEM - db171 MariaDB Connections on db171 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Too many connections in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne [09:42:38] on line 66Fatal error: Uncaught mysqli_sql_exception: Too many connections in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 1.924 second response time [09:43:15] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.478 second response time [09:43:16] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 6.127 second response time [09:43:26] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.056 second response time [09:43:27] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.055 second response time [09:43:30] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.053 second response time [09:43:43] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.055 second response time [09:43:44] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 38% [09:43:46] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4091 bytes in 2.078 second response time [09:44:18] !log reboot db171 which quite clearly doesn't like mysqlcheck [09:44:38] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.051 second response time [09:44:53] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.055 second response time [09:44:58] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.057 second response time [09:45:01] RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy [09:45:16] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy [09:45:27] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4092 bytes in 0.055 second response time [09:45:29] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 31 backends are healthy [09:47:25] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 39% [09:47:44] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [09:48:27] !log [rhinos@mwtask181] starting deploy of {'config': True} to all [09:48:48] !log [rhinos@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 20s [09:52:15] Well, shit. [09:52:35] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=avidwiki (END - exit=256) [09:55:17] !log [rhinos@puppet181] Upgraded packages on db171: base-files, libc6-dbg, libc-devtools, libc6-dev, libc-dev-bin, linux-libc-dev, libc6, libc-bin, libnss-systemd, systemd-sysv, libpam-systemd, systemd, libsystemd-shared, libsystemd0, udev, libudev1, tzdata, vim, vim-tiny, vim-runtime, vim-common, libc-l10n, locales, wget, curl, libcurl4, dns-root- [09:55:18] data, libcurl3-gnutls, libpq5, linux-image-6.1.0-32-amd64, linux-image-amd64, rsyslog-gnutls, and rsyslog [09:55:25] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 59% [09:59:04] RECOVERY - db171 APT on db171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [09:59:33] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 32 minutes ago with 0 failures [10:00:48] !log reboot and upgrade db171 [10:01:25] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [10:01:36] RECOVERY - db171 MariaDB on db171 is OK: Uptime: 20 Threads: 16 Questions: 314 Slow queries: 0 Opens: 76 Open tables: 70 Queries per second avg: 15.700 [10:01:44] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 16% [10:02:01] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.49, 0.40, 0.14 [10:02:29] RECOVERY - db171 MariaDB Connections on db171 is OK: OK connection usage: 3.2%Current connections: 32 [10:03:25] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 23% [10:05:44] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 82% [10:07:44] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 31% [10:13:44] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [10:15:44] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 48% [10:17:44] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 24% [10:26:00] !log gummypalacewiki.searchindex, hcmwiki.searchindex, hidwehwiki.searchindex, jacksonjudewiki.searchindex, keywordsinfictionwiki.searchindex, metawiki.hit_counter_extension, metawiki.profiling, mobipediatheretrosonicwikiwiki.searchindex, polymediawiki.searchindex, rosettacodewiki.searchindex, threaedimwiki.searchindex, timewastingsimulatorwiki.searchindex, weltseelewiki.searchindex rebuilt on db171 [10:27:44] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 89% [10:29:44] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 13% [10:33:31] !log [rhinos@mwtask181] starting deploy of {'config': True} to all [10:33:52] !log [rhinos@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 20s [10:33:54] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.209 second response time [10:34:00] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.224 second response time [10:34:03] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.174 second response time [10:34:08] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.160 second response time [10:34:14] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.169 second response time [10:34:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:34:28] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.149 second response time [10:34:32] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.226 second response time [10:34:38] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.198 second response time [10:34:47] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.154 second response time [10:34:49] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.162 second response time [10:34:53] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.156 second response time [10:35:03] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.160 second response time [10:35:06] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.181 second response time [10:35:14] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.155 second response time [10:35:14] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.165 second response time [10:35:18] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.156 second response time [10:35:20] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.228 second response time [10:35:28] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.147 second response time [10:35:28] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.164 second response time [10:35:29] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.152 second response time [10:35:30] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.167 second response time [10:35:39] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.168 second response time [11:24:30] PROBLEM - pso2ngs.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pso2ngs.wiki' expires in 14 day(s) (Thu 03 Apr 2025 10:53:33 AM GMT +0000). [11:24:44] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/cffe2870f49779869a42ac89c26ecf8752b926b2 [11:24:44] 02ssl/03master 07WikiTideSSLBot 03cffe287 Bot: Update SSL cert for pso2ngs.wiki [11:53:00] RECOVERY - pso2ngs.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pso2ngs.wiki' will expire on Tue 17 Jun 2025 10:26:08 AM GMT +0000. [12:19:07] [02ssl] 07AgentIsai pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/427de684f25428ef0a0ff63c20027864830f83d7 [12:19:07] 02ssl/03master 07Agent Isai 03427de68 Redirect avid.wiki to www.avid.wiki [12:20:56] [02dns] 07RhinosF1 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/477ff4dc1fcd514053438438d47b48cc776c640e [12:20:56] 02dns/03master 07RhinosF1 03477ff4d Delete zones/mappingwiki.org [12:21:49] [02dns] 07RhinosF1 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/269f9ded6511d7078ff8f6fc1550b9cba9a94993 [12:21:50] 02dns/03master 07RhinosF1 03269f9de Delete zones/xomnipedia.org [12:22:45] miraheze/dns - RhinosF1 the build passed. [12:23:30] miraheze/dns - RhinosF1 the build passed. [12:31:58] [02ssl] 07RhinosF1 created 03RhinosF1-patch-1 (+1 new commit) 13https://github.com/miraheze/ssl/commit/5c1bce3ee1ac [12:31:58] 02ssl/03RhinosF1-patch-1 07RhinosF1 035c1bce3 Delete certificates/xomnipedia.org.crt [12:32:08] [02ssl] 07RhinosF1 opened pull request #832: 2 domains to cf (03master...03RhinosF1-patch-1) 13https://github.com/miraheze/ssl/pull/832 [12:32:50] [02ssl] 07RhinosF1 pushed 1 new commit to 03RhinosF1-patch-1 13https://github.com/miraheze/ssl/commit/ca30b5ab91c48fae153b4904493e72e09bb6a0e3 [12:32:51] 02ssl/03RhinosF1-patch-1 07RhinosF1 03ca30b5a Delete certificates/mappingwiki.org.crt [12:32:53] [02ssl] 07coderabbitai[bot] commented on pull request #832: This pull request removes a file that contained two X.509 certificates for the domain "xomnipedia.org". The removed file included certificates with details such as issuer, subject, validity periods, and public key information. No other changes to […] 13https://github.com/miraheze/ssl/pull/832#issuecomment-2736473226 [12:33:58] [02ssl] 07RhinosF1 pushed 1 new commit to 03RhinosF1-patch-1 13https://github.com/miraheze/ssl/commit/fa2dae67edb54ee6f16ad9cec7d77c9d8e228d37 [12:33:58] 02ssl/03RhinosF1-patch-1 07RhinosF1 03fa2dae6 Update certs.yaml [12:34:22]