[00:11:53] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 73819MiB (16% inode=99%); [00:13:30] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 74171MiB (16% inode=99%); [00:14:15] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 73303MiB (16% inode=99%); [01:05:30] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 26.34, 18.85, 10.86 [01:09:30] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 23.51, 22.04, 14.04 [01:13:30] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 29.40, 24.91, 16.95 [01:15:30] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 16.68, 22.93, 17.28 [01:17:20] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.01, 21.92, 17.42 [01:17:30] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 10.30, 18.36, 16.29 [01:40:56] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 10.56, 20.54, 22.41 [01:42:06] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 26.66, 23.30, 19.61 [01:44:52] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 11.51, 15.86, 20.10 [01:49:42] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 20.71, 23.48, 21.37 [01:51:36] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 27.10, 24.68, 22.03 [01:53:36] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 20.56, 22.54, 21.54 [01:54:40] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.50, 20.52, 20.44 [01:55:36] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 25.26, 23.59, 22.04 [01:57:36] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 22.58, 22.94, 21.97 [02:03:36] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 30.62, 23.80, 22.25 [02:06:40] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 13.40, 17.58, 19.43 [02:11:36] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 17.13, 22.81, 23.17 [02:15:36] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 10.19, 15.80, 20.16 [02:56:51] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 7.51, 10.80, 7.12 [02:58:51] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.38, 7.74, 6.44 [12:35:32] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/43d21091b7e1c629e0378257671803659fd6b908 [12:35:32] 02ssl/03main 07WikiTideBot 0343d2109 Bot: Auto-update domain lists [12:37:33] [02landing] 07OAuthority closed pull request #149: Check if this is a right to left language (07miraheze:03main...07OAuthority:03master) 13https://github.com/miraheze/landing/pull/149 [13:09:37] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 16.70, 12.36, 7.73 [13:11:34] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.70, 9.23, 7.12 [14:48:15] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49739MiB (10% inode=99%); [14:57:53] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49774MiB (10% inode=99%); [15:07:30] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49806MiB (10% inode=99%); [15:28:17] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 115 packages available for upgrade (4 critical updates). [15:29:38] PROBLEM - os202 APT on os202 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (4 critical updates). [15:33:58] PROBLEM - os191 APT on os191 is CRITICAL: APT CRITICAL: 113 packages available for upgrade (4 critical updates). [15:34:03] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 113 packages available for upgrade (4 critical updates). [15:44:28] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 80 packages available for upgrade (4 critical updates). [15:48:25] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 115 packages available for upgrade (4 critical updates). [15:48:28] PROBLEM - os201 APT on os201 is CRITICAL: APT CRITICAL: 85 packages available for upgrade (4 critical updates). [15:51:11] PROBLEM - kafka181 APT on kafka181 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:59:14] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 119 packages available for upgrade (4 critical updates). [16:49:57] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.11, 9.43, 6.73 [16:51:55] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 5.19, 7.64, 6.38 [16:52:00] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.86, 19.53, 16.46 [16:52:22] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.11, 20.11, 16.78 [16:52:30] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.58, 19.55, 16.38 [16:53:19] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 21.25, 20.12, 16.48 [16:56:22] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 19.10, 20.30, 17.67 [16:56:30] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.59, 19.39, 17.08 [16:57:19] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 18.26, 19.95, 17.30 [17:05:44] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.28, 19.32, 18.75 [17:07:10] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.28, 20.71, 18.71 [17:09:04] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.72, 18.98, 18.28 [17:14:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761408830000&orgId=1&to=1761412460037 [17:29:52] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 13.06, 10.31, 7.18 [17:31:49] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 3.84, 7.76, 6.63 [17:34:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761410030000&orgId=1&to=1761413630000 [19:22:22] PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 176 bytes in 0.009 second response time [19:28:35] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 4.606 second response time [19:32:40] PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 176 bytes in 0.014 second response time [19:35:00] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 21.27, 12.79, 8.14 [19:36:37] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.373 second response time [19:36:57] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 6.50, 10.03, 7.67 [20:55:28] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/de4ed6017d3d65acd2a9e88639b5358fa2771a16 [20:55:29] 02ssl/03main 07WikiTideBot 03de4ed60 Bot: Auto-update domain lists [21:39:03] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 21.95, 12.88, 8.45 [21:41:00] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 5.43, 9.60, 7.77 [22:00:41] [02mw-config] 07pskyechology approved pull request #6150 13https://github.com/miraheze/mw-config/pull/6150#pullrequestreview-3380272281 [22:00:53] [02mw-config] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/c88a4351da97a81ebddf19edc03be8e8f50ad96b [22:00:53] 02mw-config/03main 07paladox 03c88a435 Increase max shell memory limit to 2GB (#6150)… [22:00:53] [02mw-config] 07pskyechology merged 07paladox's pull request #6150: Increase max shell memory limit to 2GB (03main...03paladox-patch-4) 13https://github.com/miraheze/mw-config/pull/6150 [22:00:55] [02mw-config] 07pskyechology 04deleted 03paladox-patch-4 at 03ae4d74f 13https://api.github.com/repos/miraheze/mw-config/commit/ae4d74f [22:01:50] !log [skye@test151] starting deploy of {'pull': 'cofnig', 'config': True} to test151 [22:01:51] !log [skye@test151] finished deploy of {'pull': 'cofnig', 'config': True} to test151 - SUCCESS in 0s [22:01:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:01:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:02:01] glorious typo [22:02:02] miraheze/mw-config - pskyechology the build passed. [22:02:11] !log [skye@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [22:02:12] !log [skye@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [22:02:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:02:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:03:34] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [22:03:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:03:58] !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [22:04:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:04:20] why does --cofnig succeed [22:04:23] it should produce an error imo [22:05:30] it just says failed to pull but continues anyway [22:05:39] bruh [22:05:46] @rhinosf1 see above [22:06:04] I didn't look at the code but I assume it tries to go to a folder by that name in mediawiki-staging [22:06:09] The fuck [22:06:22] if the pull failed the whole thing should fail [22:06:23] Someone file a bug [22:06:26] [1/7] ```$ mwdeploy --pull=cofnig --config --servers=all [22:06:26] [2/7] Failed to pull cofnig due to invalid name [22:06:27] [3/7] Execute: sudo -u www-data rsync --update -r --delete --exclude="." /srv/mediawiki-staging/config/ /srv/mediawiki/config/ [22:06:27] [4/7] Completed (0) in 0s! [22:06:27] [5/7] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:1053: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings [22:06:28] [6/7] warnings.warn( [22:06:28] [7/7] Start /srv/mediawiki/config/ deploys.``` [22:06:32] I'll get to it in like 2030 [22:06:32] it shouldn't deploy afterwards [22:06:44] shall i put it in the calendar [22:07:04] That should fail the pre stage and die before deploying [22:07:04] couldn't we just have it exit if it failed to pull [22:07:15] It should [22:07:40] I think this is from when @cosmicalpha rewrote bits and fucked with the exit code tracking [22:07:53] I've been meaning to unfuck it for months [22:08:27] It shouldn't deploy if any stage prior to the deploy exit with a non zero code [22:08:40] That's always been the design methodology [22:09:13] why do we need to track the exit code if we could just exit the entire thing normally [22:09:55] That's also a design methodology [22:10:03] It does things in stages [22:10:32] With the idea that eventually you could parallelise things and in theory you can only guarantee that Stage 1 will happen before Stage 2 [22:10:46] Not that any command in Stage 1 will happen in a certain order [22:10:51] ah [22:11:52] I wrote a draft to parallelise stuff years ago [22:12:01] It didn't work [22:12:10] I haven't ever got around to trying it again [22:41:28] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:41:29] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:41:30] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:41:30] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:41:31] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:41:33] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:41:34] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [22:41:35] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [22:41:37] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 96.46, 41.55, 17.63 [22:41:41] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:41:44] @paladox @rhinosf1 we're down [22:41:45] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [22:41:45] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:41:53] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [22:41:54] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:41:57] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [22:42:03] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:42:07] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [22:42:08] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.017 second response time [22:42:08] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:42:12] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:42:12] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:42:17] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [22:42:25] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:42:26] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:42:27] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [22:42:37] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [22:42:39] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [22:42:40] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.013 second response time [22:42:41] guess i'm not doing these renames now [22:42:41] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [22:42:47] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 82% [22:42:51] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [22:42:52] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [22:42:53] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [22:42:55] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:42:56] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:42:58] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [22:42:59] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:43:02] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:43:05] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:43:12] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:43:13] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [22:43:16] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:43:34] Oh what happened [22:43:40] check MM [22:43:42] !log [blankeclair@mwtask181] starting deploy of {'config': True} to all [22:43:43] !log [blankeclair@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [22:43:44] Is there any point in sending a manual report? Cloudflare wants me to send a string of text. [22:43:48] !log put c2 into maintenance mode [22:43:55] !log [blankeclair@mwtask181] starting deploy of {'config': True, 'force': True} to all [22:44:04] What's MM? [22:44:07] mattermost [22:44:07] I guess revert the config change [22:44:09] mattermost [22:44:14] Use —force [22:44:15] !log [blankeclair@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 19s [22:44:19] ? [22:44:34] [22:44:43] > [26/10/2025 09:44] I guess revert the config change [22:44:44] eh? [22:44:49] db161 is unresponsive [22:44:52] I think [22:44:57] I don't think that's caused by the change [22:45:03] assuming you mean the shell memory limit [22:45:05] https://grafana.wikitide.net/d/W9MIkA7iz/wikitide-cluster?orgId=1&from=now-1h&to=now&timezone=browser&var-job=node&var-node=db161.fsslc.wtnet&var-port=9100&refresh=auto [22:45:06] Logging in [22:45:08] she's not looking good [22:45:15] I'm on it [22:45:29] awww but i was assured we wont explode [22:45:33] Ah, depool db161 from mw* [22:45:43] Already done @paladox [22:45:48] You're about 10 steps behind [22:45:50] technology team you guys got this [22:45:50] I think this is related to the config change. [22:45:54] PROBLEM - db161 APT on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:45:56] I'm already typing the command to restart MySQL [22:46:03] If MF is listed in full process list [22:46:09] [26/10/2025 09:44] https://miraheze.org/ [22:46:11] cursed [22:46:11] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.298 second response time [22:46:11] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.300 second response time [22:46:11] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 3.269 second response time [22:46:13] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 1.566 second response time [22:46:14] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 2.895 second response time [22:46:17] wtf [22:46:23] PROBLEM - db161 MariaDB Connections on db161 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: MySQL server has gone away in /usr/lib/nagios/plugins/check_mysql_connections.php:74Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(74): mysqli->query('SHOW STATUS WHE...')#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 74Fatal error: Uncaught mysqli_sql_exception: MySQL server has g [22:46:23] n /usr/lib/nagios/plugins/check_mysql_connections.php:74Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(74): mysqli->query('SHOW STATUS WHE...')#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 74 [22:46:24] PROBLEM - db161 MariaDB on db161 is CRITICAL: Lost connection to server at 'handshake: reading initial communication packet', system error: 104 [22:46:24] !log restart MySQL on db161 [22:46:26] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 1.125 second response time [22:46:26] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.300 second response time [22:46:27] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.086 second response time [22:46:29] how did it just unbreak [22:46:30] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.388 second response time [22:46:35] i thought db171 was the one critical db lol [22:46:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:46:37] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.280 second response time [22:46:39] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.243 second response time [22:46:39] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.220 second response time [22:46:41] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.373 second response time [22:46:41] i guess it's back [22:46:43] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.064 second response time [22:46:46] rhinos restartet it [22:46:47] has g [22:46:51] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.243 second response time [22:46:51] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.237 second response time [22:46:52] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.225 second response time [22:46:55] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.069 second response time [22:46:56] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4327 bytes in 0.065 second response time [22:46:59] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.243 second response time [22:47:01] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.071 second response time [22:47:05] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.073 second response time [22:47:07] Thanks @rhinosf1 for dealing with it [22:47:12] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 0.070 second response time [22:47:13] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.218 second response time [22:47:16] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 0.066 second response time [22:47:16] !log [blankeclair@mwtask181] starting deploy of {'config': True, 'force': True} to all [22:47:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:47:21] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.086 second response time [22:47:21] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.070 second response time [22:47:22] !log put c2 out of maintence [22:47:26] aand we're back. [22:47:26] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 6.015 second response time [22:47:26] i can't spell, but okay [22:47:32] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.862 second response time [22:47:34] @paladox I doubt the max shell memory had anything to do it with it [22:47:37] Tbh [22:47:44] how would that affect the db [22:47:45] That shouldn't affect MySQL at all [22:47:46] why's mwdeploy not deploying [22:47:50] with some latency, but back regardless [22:47:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:47:58] https://tenor.com/view/thanks-thank-you-thank-you-so-much-thank-you-very-much-why-thank-you-gif-8412321576680194407 [22:47:59] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.980 second response time [22:48:02] Tech team [22:48:08] !log [blankeclair@mwtask181] starting deploy of {'config': True, 'force': True} to all [22:48:17] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [22:48:20] nevermind. [22:48:22] RECOVERY - db161 MariaDB on db161 is OK: Uptime: 111 Threads: 287 Questions: 26975 Slow queries: 9 Opens: 3599 Open tables: 3592 Queries per second avg: 243.018 [22:48:22] RECOVERY - db161 APT on db161 is OK: APT OK: 127 packages available for upgrade (0 critical updates). [22:48:23] RECOVERY - db161 MariaDB Connections on db161 is OK: OK connection usage: 27.1%Current connections: 271 [22:48:26] Rodejong said down again [22:48:47] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 35% [22:48:51] !log [blankeclair@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 43s [22:48:52] (rodejong lied) [22:48:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:48:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:48:59] A quarter of wikis are still down [22:49:01] i'm not surprised [22:49:05] c2 now unmainted [22:49:07] Also Ro should be ignored for anything tech [22:49:12] Now we're back [22:49:22] Thanks BlankEclair [22:49:26] yw ^_^ [22:49:27] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.855 second response time [22:49:34] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.195 second response time [22:49:39] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.054 second response time [22:49:41] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.050 second response time [22:49:45] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.202 second response time [22:49:45] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.170 second response time [22:49:45] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.057 second response time [22:49:49] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.059 second response time [22:49:54] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.196 second response time [22:50:03] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 0.060 second response time [22:56:17] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [23:06:12] [02ssl] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/99f0c947abc29e06aa9542d22c832caa301da814 [23:06:12] 02ssl/03main 07Skye 0399f0c94 Add redirect for maiyamawikiwiki… [23:09:30] PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 26899MiB (5% inode=99%); [23:09:53] PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 27009MiB (5% inode=99%); [23:13:53] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 71177MiB (15% inode=99%); [23:15:30] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 71048MiB (15% inode=99%); [23:25:37] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 2.30, 2.47, 10.94 [23:27:37] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 2.74, 2.59, 9.95