[01:03:28] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 13.38, 6.92, 3.02 [03:19:28] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 11.16, 9.95, 11.99 [03:25:28] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 13.03, 11.40, 11.92 [04:12:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:15:28] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 11.25, 10.99, 11.84 [06:17:28] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 13.76, 11.81, 12.02 [06:19:28] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 10.08, 10.79, 11.62 [06:25:28] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 12.32, 11.48, 11.68 [06:27:28] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 11.87, 11.78, 11.79 [06:33:28] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 12.48, 12.13, 11.94 [08:17:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:17:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:19:31] PROBLEM - swiftobject161 Disk Space on swiftobject161 is CRITICAL: DISK CRITICAL - free space: / 80124MiB (5% inode=85%); [12:20:05] doomsday is approaching for swiftobject161 [12:22:05] I've only had to deal with partitions after an install when my /boot ran out of space for kernel upgrades, I got rid of swap and enlarged the boot partition to fill up the space left behind between it and my main partition [12:22:28] even though I knew what to do, that was still pretty stressful [13:48:50] !log [paladox@mwtask181] starting deploy of {'world': True, 'l10n': True, 'versions': ['1.42', '1.43']} to all [13:48:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:49:18] !log [paladox@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [13:49:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:50:55] !log [paladox@mwtask181] starting deploy of {'world': True, 'l10n': True, 'versions': ['1.42', '1.43']} to all [13:50:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:59:09] [02RottenLinks] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/RottenLinks/commit/c64fa3f90ade [13:59:10] 02RottenLinks/03paladox-patch-1 07paladox 03c64fa3f Don't restrict rl_externallink index to 50 [13:59:13] [02RottenLinks] 07paladox opened pull request #86: Don't restrict rl_externallink index to 50 (03master...03paladox-patch-1) 13https://github.com/miraheze/RottenLinks/pull/86 [13:59:22] [02RottenLinks] 07coderabbitai[bot] commented on pull request #86: --- […] 13https://github.com/miraheze/RottenLinks/pull/86#issuecomment-2523313241 [14:03:59] miraheze/RottenLinks - paladox the build passed. [14:11:28] !log [paladox@mwtask181] finished deploy of {'world': True, 'l10n': True, 'versions': ['1.42', '1.43']} to all - SUCCESS in 1233s [14:11:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:12:31] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': ['1.42', '1.43'], 'upgrade_skins': 'Citizen'} to all [14:12:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:16:27] PROBLEM - mwtask181 jobrunner-high.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [14:16:32] !log [paladox@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [14:16:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:17:28] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 28.75, 24.68, 11.62 [14:17:28] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': ['1.42', '1.43'], 'upgrade_skins': 'Citizen'} to all [14:17:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:18:26] RECOVERY - mwtask181 jobrunner-high.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 0.030 second response time [14:20:58] PROBLEM - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10001 milliseconds [14:20:59] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:02] !log [paladox@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [14:21:03] PROBLEM - mwtask181 jobrunner.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [14:21:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:22:54] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.186 second response time [14:22:55] RECOVERY - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.012 second response time [14:22:59] RECOVERY - mwtask181 jobrunner.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.012 second response time [14:23:22] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 7.51, 22.45, 15.42 [14:25:19] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 1.89, 15.63, 13.75 [14:31:13] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': ['1.42', '1.43'], 'upgrade_skins': 'Citizen'} to all [14:31:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:31:53] !log [paladox@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [14:31:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:33:14] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': ['1.42', '1.43'], 'upgrade_skins': 'Citizen'} to all [14:33:15] !log [paladox@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [14:33:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:33:20] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'force_upgrade': True, 'versions': ['1.42', '1.43'], 'upgrade_skins': 'Citizen'} to all [14:33:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:33:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:36:59] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'force_upgrade': True, 'versions': ['1.42', '1.43'], 'upgrade_skins': 'Citizen'} to all - SUCCESS in 219s [14:37:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:17:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:49:52] [02RottenLinks] 07paladox merged pull request #86: Don't restrict rl_externallink index to 50 (03master...03paladox-patch-1) 13https://github.com/miraheze/RottenLinks/pull/86 [16:49:53] [02RottenLinks] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/RottenLinks/commit/3af3f5127bb65c249e6ee46e336e6ed225f817f6 [16:49:53] 02RottenLinks/03master 07paladox 033af3f51 Don't restrict rl_externallink index to 50 (#86) [16:52:18] [02RottenLinks] 07paladox 04deleted 03paladox-patch-1 at 03c64fa3f 13https://github.com/miraheze/RottenLinks/commit/c64fa3f [16:53:04] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': ['1.42', '1.43'], 'upgrade_extensions': 'RottenLinks'} to all [16:53:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:49] miraheze/RottenLinks - paladox the build passed. [16:56:22] !log [paladox@mwtask181] /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/rl.sql [16:56:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:10:49] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': ['1.42', '1.43'], 'upgrade_extensions': 'RottenLinks'} to all - SUCCESS in 1065s [17:10:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:26:49] !log [paladox@mwtask181] /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/el_from.sql [17:26:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:27:28] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 2.95, 8.43, 11.46 [17:31:28] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 1.43, 4.57, 9.17 [17:32:36] PROBLEM - lads.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lads.wiki' expires in 14 day(s) (Sat 21 Dec 2024 05:23:23 PM GMT +0000). [17:32:57] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/51b414eb7f8b72a9128a149ead6b926cfe1c3f78 [17:32:57] 02ssl/03master 07WikiTideSSLBot 0351b414e Bot: Update SSL cert for lads.wiki [18:02:12] RECOVERY - lads.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lads.wiki' will expire on Thu 06 Mar 2025 04:34:20 PM GMT +0000. [20:17:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [21:36:57] PROBLEM - allthetropes.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'allthetropes.org' expires in 14 day(s) (Sat 21 Dec 2024 09:24:19 PM GMT +0000). [21:37:12] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/901cef1657e6bce989773a391b6a3f1b8c399464 [21:37:12] 02ssl/03master 07WikiTideSSLBot 03901cef1 Bot: Update SSL cert for allthetropes.org [21:57:10] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 139.61 ms [21:59:10] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 139.61 ms [22:00:52] BlankEclair: pi [22:00:54] Oi [22:00:59] [02mw-config] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/15c3a86ae10d [22:00:59] 02mw-config/03paladox-patch-2 07paladox 0315c3a86 Set LoadMonitor [22:01:04] [02mw-config] 07paladox opened pull request #5759: Set LoadMonitor (03master...03paladox-patch-2) 13https://github.com/miraheze/mw-config/pull/5759 [22:01:41] BlankEclair: hello [22:01:50] have we figured out what the hell is happening to the job queue? [22:02:12] No [22:02:33] [02mw-config] 07github-actions[bot] pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/mw-config/commit/60f81712774cf6180906c46baea17f0ce8ed817f [22:02:34] 02mw-config/03paladox-patch-2 07github-actions 0360f8171 CI: lint code to MediaWiki standards… [22:03:02] miraheze/mw-config - paladox the build passed. [22:05:55] RECOVERY - allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Thu 06 Mar 2025 08:38:36 PM GMT +0000. [22:06:14] [02mw-config] 07paladox merged pull request #5759: Set LoadMonitor (03master...03paladox-patch-2) 13https://github.com/miraheze/mw-config/pull/5759 [22:06:14] [02mw-config] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/22aa68fe1d053ed038ee42129a90930e3973e741 [22:06:14] 02mw-config/03master 07paladox 0322aa68f Set LoadMonitor (#5759)… [22:06:15] [02mw-config] 07paladox 04deleted 03paladox-patch-2 at 0360f8171 13https://github.com/miraheze/mw-config/commit/60f8171 [22:06:28] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [22:06:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:53] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s [22:06:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:07:08] miraheze/mw-config - paladox the build passed. [22:10:20] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'speleo.wiki' expires in 14 day(s) (Sat 21 Dec 2024 09:44:03 PM GMT +0000). [22:10:31] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/5fd1cf4eae2ee1dfe830cb8859fa0abb7dfb44cb [22:10:31] 02ssl/03master 07WikiTideSSLBot 035fd1cf4 Bot: Update SSL cert for speleo.wiki [22:26:42] PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thelonsdalebattalion.co.uk' expires in 14 day(s) (Sat 21 Dec 2024 10:09:43 PM GMT +0000). [22:27:09] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/fed8cd47498b217738284501505e7c8152a96d17 [22:27:09] 02ssl/03master 07WikiTideSSLBot 03fed8cd4 Bot: Update SSL cert for thelonsdalebattalion.co.uk [22:31:23] PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'marinebiodiversitymatrix.org' expires in 14 day(s) (Sat 21 Dec 2024 10:16:01 PM GMT +0000). [22:31:47] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/92afd00f9270a7e0f4ac8680b764af6775e224ef [22:31:47] 02ssl/03master 07WikiTideSSLBot 0392afd00 Bot: Update SSL cert for marinebiodiversitymatrix.org [22:55:55] RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Thu 06 Mar 2025 09:28:33 PM GMT +0000. [23:00:47] RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Thu 06 Mar 2025 09:33:11 PM GMT +0000. [23:00:59] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/dc7bc8d053c30326b308496b4b7621bd8e312d68 [23:00:59] 02ssl/03master 07MacFan4000 03dc7bc8d add wiki.ma-sra.org and yunticu.s4aysnek.com [23:10:00] RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Thu 06 Mar 2025 09:11:56 PM GMT +0000. [23:27:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [23:31:01] [02python-functions] 07dependabot[bot] created 03dependabot/pip/dot-github/coverage-7.6.9 (+1 new commit) 13https://github.com/miraheze/python-functions/commit/7358ca11bef9 [23:31:02] 02python-functions/03dependabot/pip/dot-github/coverage-7.6.9 07dependabot[bot] 037358ca1 Bump coverage from 7.6.8 to 7.6.9 in /.github… [23:31:02] [02python-functions] 07dependabot[bot] opened pull request #107: Bump coverage from 7.6.8 to 7.6.9 in /.github (03master...03dependabot/pip/dot-github/coverage-7.6.9) 13https://github.com/miraheze/python-functions/pull/107 [23:31:03] [02python-functions] 07dependabot[bot] added the label 'dependencies' to pull request #107 (Bump coverage from 7.6.8 to 7.6.9 in /.github) 13https://github.com/miraheze/python-functions/pull/107 [23:31:05] [02python-functions] 07dependabot[bot] added the label 'python' to pull request #107 (Bump coverage from 7.6.8 to 7.6.9 in /.github) 13https://github.com/miraheze/python-functions/pull/107 [23:31:08] [02python-functions] 07coderabbitai[bot] commented on pull request #107: --- […] 13https://github.com/miraheze/python-functions/pull/107#issuecomment-2524619200 [23:36:08] [02python-functions] 07github-merge-queue[bot] created 03gh-readonly-queue/master/pr-107-314302e7e461b1b1695cea1b43478d5116e2b4d4 (+1 new commit) 13https://github.com/miraheze/python-functions/commit/b0939b1d4ed9 [23:36:08] 02python-functions/03gh-readonly-queue/master/pr-107-314302e7e461b1b1695cea1b43478d5116e2b4d4 07dependabot[bot] 03b0939b1 Bump coverage from 7.6.8 to 7.6.9 in /.github (#107)… [23:36:16] miraheze/python-functions - dependabot[bot] the build passed. [23:39:21] [02python-functions] 07github-merge-queue[bot] pushed 1 new commit to 03master 13https://github.com/miraheze/python-functions/commit/b0939b1d4ed900dec7e9b97ac73c2691aba5beb1 [23:39:21] 02python-functions/03master 07dependabot[bot] 03b0939b1 Bump coverage from 7.6.8 to 7.6.9 in /.github (#107)… [23:39:22] [02python-functions] 07github-merge-queue[bot] 04deleted 03gh-readonly-queue/master/pr-107-314302e7e461b1b1695cea1b43478d5116e2b4d4 at 03b0939b1 13https://github.com/miraheze/python-functions/commit/b0939b1 [23:39:22] [02python-functions] 07github-actions[bot] merged 07dependabot[bot]'s pull request #107: Bump coverage from 7.6.8 to 7.6.9 in /.github (03master...03dependabot/pip/dot-github/coverage-7.6.9) 13https://github.com/miraheze/python-functions/pull/107 [23:39:24] [02python-functions] 07github-actions[bot] 04deleted 03dependabot/pip/dot-github/coverage-7.6.9 at 037358ca1 13https://github.com/miraheze/python-functions/commit/7358ca1 [23:40:24] [02python-functions] 07dependabot[bot] 04force-pushed 03dependabot/pip/dot-github/langcodes-3.5.0 from 0351a6ee6 to 036852f05 13https://github.com/miraheze/python-functions/commits/dependabot/pip/dot-github/langcodes-3.5.0 [23:40:24] 02python-functions/03dependabot/pip/dot-github/langcodes-3.5.0 07dependabot[bot] 036852f05 Bump langcodes from 3.4.1 to 3.5.0 in /.github… [23:43:49] miraheze/python-functions - github-merge-queue[bot] the build passed. [23:44:18] miraheze/python-functions - dependabot[bot] the build has errored. [23:48:04] [02mw-config] 07BlankEclair opened pull request #5760: T12963: Raise DPL limits for gui7814sgtafanonwiki (07miraheze:03master...07BlankEclair:03T12963) 13https://github.com/miraheze/mw-config/pull/5760 [23:49:00] miraheze/mw-config - BlankEclair the build passed. [23:53:26] [02RequestSSL] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/RequestSSL/commit/06c324e2f69d [23:53:27] 02RequestSSL/03paladox-patch-1 07paladox 0306c324e Prevent setting $this->mDb twice… [23:53:30] [02RequestSSL] 07paladox opened pull request #60: Prevent setting $this->mDb twice (03master...03paladox-patch-1) 13https://github.com/miraheze/RequestSSL/pull/60 [23:53:35] [02RequestSSL] 07coderabbitai[bot] commented on pull request #60: --- […] 13https://github.com/miraheze/RequestSSL/pull/60#issuecomment-2524656735 [23:54:16] [02RequestSSL] 07paladox merged pull request #60: Prevent setting $this->mDb twice (03master...03paladox-patch-1) 13https://github.com/miraheze/RequestSSL/pull/60 [23:54:16] [02RequestSSL] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/RequestSSL/commit/59f8d2e0d1b36521627206dda3bb3a633fd4b356 [23:54:16] 02RequestSSL/03master 07paladox 0359f8d2e Prevent setting $this->mDb twice (#60)… [23:54:17] [02RequestSSL] 07paladox 04deleted 03paladox-patch-1 at 0306c324e 13https://github.com/miraheze/RequestSSL/commit/06c324e [23:56:03] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': ['1.42', '1.43'], 'upgrade_extensions': 'RequestSSL'} to all [23:56:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:59:15] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': ['1.42', '1.43'], 'upgrade_extensions': 'RequestSSL'} to all - SUCCESS in 192s [23:59:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log