[00:01:00] PROBLEM - civwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.802B7F5154150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1605:SSL alert number 40 [00:05:17] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 58841MiB (12% inode=99%); [00:05:57] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 25.51, 21.44, 17.59 [00:06:48] RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 58274MiB (12% inode=99%); [00:07:57] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 21.61, 21.06, 17.91 [00:13:57] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 14.35, 19.48, 18.58 [00:30:00] RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Sat 28 Jun 2025 11:58:25 PM GMT +0000. [00:33:18] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [00:33:19] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=philosophyballwiki 6001 --e 8000 (running for 03:23) [00:33:19] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 03:08:18) [00:33:20] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=tuscriaturaswiki 26001 --e 28000 (running for 02:13) [00:33:21] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=metawiki 32001 --e 34000 (running for 02:34) [00:39:49] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 33.19, 22.81, 15.63 [00:43:49] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 20.61, 23.61, 17.79 [00:45:49] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 9.47, 19.13, 16.91 [01:11:02] !log [blankeclair@mwtask171] runRefreshLinksChunked: [1100/1403] Handling unicodesubsetswiki (start id: 0) [01:11:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:19:03] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.30, 6.80, 6.33 [01:24:46] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.07, 7.09, 6.59 [01:26:40] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.78, 7.32, 6.73 [01:28:34] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.65, 6.65, 6.55 [01:33:22] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [01:33:23] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=plasticssongcontestwiki 1 --e 2000 (running for 14:59) [01:33:24] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 04:08:23) [01:33:25] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=upwellwiki 2001 (running for 01:45) [01:33:26] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=metawiki 42001 (running for 42:29) [01:43:49] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 29.04, 18.18, 11.95 [01:49:49] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 10.19, 19.15, 15.02 [01:56:19] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.07, 6.49, 6.18 [01:58:13] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.30, 6.00, 6.03 [02:03:57] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 19.90, 20.91, 15.00 [02:05:57] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 5.84, 15.26, 13.65 [02:26:32] !log [blankeclair@mwtask171] runRefreshLinksChunked: [1200/1403] Handling vulaahistorywiki (start id: 0) [02:26:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:33:10] PROBLEM - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:33:25] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [02:33:26] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=podpediawiki 10001 --e 12000 (running for 01:58) [02:33:27] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 05:08:26) [02:33:28] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=wackypediawiki 4001 --e 6000 (running for 01:01) [02:33:29] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=metawiki 42001 (running for 01:42:32) [02:35:10] RECOVERY - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.012 second response time [02:53:05] PROBLEM - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [02:54:13] !log [blankeclair@mwtask151] runRefreshLinksChunked: [600/1788] Handling pokemondraconiaadventuresfananimewikiwiki (start id: 0) [02:54:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:55:06] RECOVERY - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.980 second response time [02:58:45] !log [blankeclair@mwtask151] runRefreshLinksChunked: [608/1788] polandballwiki exited with 137 [02:58:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:01:49] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 22.68, 16.21, 10.92 [03:10:50] fucking balls [03:11:14] !log [blankeclair@mwtask151] runRefreshLinksChunked: [0/1180] Handling polandballwiki (start id: 0) [03:11:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:11:49] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 20.03, 19.89, 15.63 [03:12:52] !log [blankeclair@mwtask151] runRefreshLinksChunked: [0/1180] polandballwiki exited with 137 [03:12:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:12:56] fuck [03:13:00] to the back you go [03:13:20] !log [blankeclair@mwtask151] runRefreshLinksChunked: [0/1180] Handling polaricecrewwiki (start id: 0) [03:13:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:13:34] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 18.19, 14.08, 23.31 [03:19:34] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 12.19, 12.48, 19.61 [03:29:13] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 34.13, 24.06, 18.66 [03:33:28] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [03:33:29] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=potawiki 2001 --e 4000 (running for 00:13) [03:33:30] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 06:08:29) [03:33:31] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=wikionewiki 1 --e 2000 (running for 02:07) [03:33:32] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=metawiki 42001 (running for 02:42:35) [03:43:34] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 30.62, 20.21, 16.18 [03:45:34] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.96, 20.26, 16.66 [03:47:34] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 16.47, 19.38, 16.83 [03:50:58] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 17.91, 16.02, 23.53 [03:57:34] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 37.12, 24.16, 18.82 [04:00:58] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.54, 20.32, 21.75 [04:02:58] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 18.38, 19.58, 21.32 [04:04:58] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 13.35, 17.43, 20.33 [04:09:49] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 5.99, 14.44, 22.52 [04:09:57] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 36.67, 24.54, 15.78 [04:13:49] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 8.60, 11.41, 19.44 [04:33:31] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [04:33:32] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=ptd3wiki 1 --e 2000 (running for 13:22) [04:33:33] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 07:08:32) [04:33:34] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=windowsruwiki 1 --e 2000 (running for 01:09) [04:33:35] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=metawiki 42001 (running for 03:42:38) [04:34:58] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.26, 20.15, 15.74 [04:35:49] !log [blankeclair@mwtask171] runRefreshLinksChunked: [1300/1403] Handling wolfvisionwiki (start id: 0) [04:35:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:38:58] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 14.11, 19.26, 16.64 [04:43:49] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 37.75, 24.44, 16.50 [04:47:57] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 17.00, 16.13, 23.72 [04:53:57] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 9.29, 12.15, 19.58 [05:00:46] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 37.77, 25.77, 19.06 [05:05:49] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 6.59, 16.47, 23.14 [05:09:49] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 4.19, 9.63, 18.79 [05:33:35] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [05:33:36] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=ptsdwiki 1 --e 2000 (running for 00:22) [05:33:37] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 08:08:35) [05:33:38] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhinternetpediawiki 2001 --e 4000 (running for 02:01) [05:33:39] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=metawiki 42001 (running for 04:42:42) [05:34:25] Jesus, eight hours. [05:37:57] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 27.98, 20.88, 15.08 [05:41:57] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 15.52, 20.19, 16.36 [05:53:36] !log [blankeclair@mwtask151] runRefreshLinksChunked: [100/1180] Handling r7rwiki (start id: 0) [05:53:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:31:49] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 27.32, 18.91, 13.27 [06:33:39] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [06:33:40] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=rainversewiki 4001 --e 6000 (running for 02:35) [06:33:41] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 09:08:39) [06:33:42] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhtardiswiki 2001 --e 4000 (running for 02:30) [06:33:43] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=metawiki 42001 (running for 05:42:45) [06:33:54] no way it got to us :3 [06:37:49] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 7.26, 18.49, 15.88 [06:54:58] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 9.23, 13.57, 23.55 [06:58:58] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 29.65, 18.59, 23.00 [07:12:10] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 21.56, 17.12, 14.55 [07:14:05] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 16.84, 16.82, 14.74 [07:17:20] [Grafana] FIRING: The mediawiki job queue has more than 20000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:21:49] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 26.57, 22.26, 17.87 [07:23:49] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 19.67, 21.52, 18.16 [07:25:49] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 14.51, 18.08, 17.25 [07:31:57] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 29.18, 22.88, 15.95 [07:33:42] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [07:33:43] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=rainworldwiki 2001 --e 4000 (running for 13:08) [07:33:44] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 10:08:43) [07:33:45] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhwpwikiwiki 1 --e 2000 (running for 03:30) [07:33:46] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=metawiki 42001 (running for 06:42:49) [07:33:57] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 10.99, 18.57, 15.25 [07:36:37] !log [blankeclair@mwtask171] runRefreshLinksChunked: [1400/1403] Handling zuu6wiki (start id: 0) [07:36:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:36:55] yo wtf 171 almost done [07:52:20] [Grafana] FIRING: The mediawiki job queue has more than 20000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:02:20] [Grafana] FIRING: The mediawiki job queue has more than 20000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:03:34] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 8.45, 16.08, 23.74 [08:09:34] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 15.33, 13.49, 19.90 [08:27:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:27:34] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 24.86, 19.95, 18.09 [08:33:45] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [08:33:46] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=rainworldwiki 10001 --e 12000 (running for 02:39) [08:33:47] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 11:08:46) [08:33:48] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=creaturawiki 278001 --e 280000 (running for 00:23) [08:33:49] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=metawiki 42001 (running for 07:42:52) [08:35:34] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 16.80, 21.31, 20.13 [08:37:34] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 11.28, 18.05, 19.10 [08:52:26] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 29.54, 23.04, 20.03 [08:56:22] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 13.56, 21.56, 20.41 [08:58:19] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 10.58, 18.08, 19.28 [09:19:49] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 37.36, 24.16, 15.71 [09:19:55] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.31, 20.79, 18.95 [09:21:36] [02mw-config] 07Aeywoo opened pull request #5887: Add Extension:LinkTitles Configuration settings and remove simulatorwiki refrences (07miraheze:03master...07Aeywoo:03patch-5) 13https://github.com/miraheze/mw-config/pull/5887 [09:21:53] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 13.23, 17.47, 17.93 [09:22:24] miraheze/mw-config - Aeywoo the build has errored. [09:25:11] miraheze/mw-config - Aeywoo the build passed. [09:29:23] [02mw-config] 07Aeywoo retitled PR #5887: "Add Extension:LinkTitles Configuration settings and remove simulatorwiki refrences" ➜ "Add Extension:LinkTitles Configuration settings and remove simulatorwiki references" 13https://github.com/miraheze/mw-config/pull/5887 [09:33:48] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [09:33:49] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=rainworldwiki 16001 --e 18000 (running for 07:24) [09:33:49] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 6.58, 20.36, 21.80 [09:33:50] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 12:08:48) [09:33:51] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=creaturawiki 570001 --e 572000 (running for 00:27) [09:33:52] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=metzowiki 8001 --e 10000 (running for 02:38) [09:37:49] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 14.67, 15.95, 19.55 [09:50:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:51:42] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.89, 6.41, 5.71 [09:53:42] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 4.21, 5.68, 5.53 [09:58:17] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 25.97, 18.46, 14.76 [10:14:01] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 17.99, 23.81, 22.70 [10:17:57] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 15.73, 17.58, 20.29 [10:21:49] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 30.49, 21.98, 16.07 [10:24:20] It looks like this is pretty steadily decreasing per Grafana, looks like Transcode jobs are main culprit, so probably not much to worry about [10:24:29] [02mw-config] 07Aeywoo opened pull request #5888: Remove SemanticMediaWiki Conflict (07miraheze:03master...07Aeywoo:03patch-6) 13https://github.com/miraheze/mw-config/pull/5888 [10:25:24] miraheze/mw-config - Aeywoo the build passed. [10:29:58] I silenced it [10:29:59] It's known [10:30:05] And there ain't much we can do [10:33:51] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [10:33:52] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=rainworldwiki 22001 --e 24000 (running for 10:47) [10:33:53] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 13:08:52) [10:33:54] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=creaturawiki 830001 (running for 06:01) [10:33:55] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=micronationalistwiki 1 --e 2000 (running for 05:36) [10:45:49] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 5.87, 15.48, 22.62 [10:49:49] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 9.73, 11.53, 19.32 [11:07:34] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 39.05, 25.74, 17.13 [11:22:58] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 15.30, 16.72, 23.61 [11:30:38] !log [blankeclair@mwtask181] runRefreshLinksChunked: [1200/2761] Handling momowiki (start id: 0) [11:30:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:30:58] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 11.60, 13.76, 19.52 [11:33:55] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [11:33:56] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=rainworldwiki 28001 --e 30000 (running for 07:51) [11:33:57] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 14:08:55) [11:33:58] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=creaturawiki 830001 (running for 01:06:04) [11:33:59] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=mondodipipowiki 1 --e 2000 (running for 02:46) [11:42:35] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 27.79, 19.94, 16.35 [11:44:30] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 23.42, 21.84, 17.52 [11:46:25] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 17.84, 20.25, 17.44 [11:50:16] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 23.73, 22.53, 19.00 [11:54:06] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 11.48, 18.78, 18.42 [11:54:58] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.23, 21.15, 18.69 [11:59:34] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 19.53, 19.44, 23.99 [12:05:34] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 11.05, 14.39, 20.39 [12:09:49] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 24.03, 20.10, 17.50 [12:10:51] [02CreateWiki] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/10709070aab5970fcfb2770af7aa72bf2da946d3 [12:10:53] 02CreateWiki/03master 07translatewiki.net 031070907 Localisation updates from https://translatewiki.net. [12:10:55] [02MatomoAnalytics] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/MatomoAnalytics/commit/4e0abebd1c8fecf3cad1ba2b146e7347502ade82 [12:10:57] [02landing] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/landing/commit/046909c292ea073261152b2d69b3d6ef751aeaf0 [12:10:58] 02MatomoAnalytics/03master 07translatewiki.net 034e0abeb Localisation updates from https://translatewiki.net. [12:11:00] 02landing/03master 07translatewiki.net 03046909c Localisation updates from https://translatewiki.net. [12:11:57] miraheze/landing - translatewiki the build passed. [12:13:49] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 20.96, 22.84, 19.40 [12:15:49] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 12.50, 19.24, 18.51 [12:15:54] miraheze/MatomoAnalytics - translatewiki the build passed. [12:18:10] [02mediawiki-repos] 07Aeywoo opened pull request #70: Undeploy Skin:Eveskin per T13411 (07miraheze:03master...07Aeywoo:03patch-1) 13https://github.com/miraheze/mediawiki-repos/pull/70 [12:18:16] [02mediawiki-repos] 07coderabbitai[bot] commented on pull request #70: --- […] 13https://github.com/miraheze/mediawiki-repos/pull/70#issuecomment-2766051990 [12:19:19] [02mw-config] 07Aeywoo opened pull request #5889: Undeploy Skin:Eveskin per T13411 (07miraheze:03master...07Aeywoo:03patch-7) 13https://github.com/miraheze/mw-config/pull/5889 [12:19:24] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/c32a1a7dc61e [12:19:26] 02puppet/03paladox-patch-1 07paladox 03c32a1a7 changeprop: don't restart service instead reload it on config change [12:19:28] [02puppet] 07paladox opened pull request #4279: changeprop: don't restart service instead reload it on config change (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4279 [12:19:35] [02puppet] 07coderabbitai[bot] commented on pull request #4279: --- […] 13https://github.com/miraheze/puppet/pull/4279#issuecomment-2766055225 [12:20:34] miraheze/mw-config - Aeywoo the build passed. [12:21:34] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 23.30, 18.41, 17.89 [12:22:57] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/34500e9eaa3f9372cace525056fada4bcff70a28 [12:22:57] 02puppet/03paladox-patch-1 07paladox 0334500e9 Update init.pp [12:23:34] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 30.82, 22.00, 19.22 [12:27:28] miraheze/CreateWiki - translatewiki the build passed. [12:32:58] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 6.73, 11.69, 22.55 [12:33:58] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [12:33:59] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=rainworldwiki 34001 --e 36000 (running for 05:18) [12:34:00] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 15:08:58) [12:34:01] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=creaturawiki 830001 (running for 02:06:07) [12:34:02] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=mpwwiki 1 --e 2000 (running for 03:28) [12:36:58] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 8.74, 10.26, 19.45 [12:44:08] [02ssl] 07RhinosF1 closed pull request #815: T13039: acuralegendwiki.org: Redirect /index.php to /wiki (07miraheze:03master...07BlankEclair:03T13039) 13https://github.com/miraheze/ssl/pull/815 [12:44:21] [02puppet] 07RhinosF1 closed pull request #4135: Add the ability to add rewrite directives for custom domains (07miraheze:03master...07BlankEclair:03T13039) 13https://github.com/miraheze/puppet/pull/4135 [13:30:58] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 30.91, 22.07, 15.79 [13:34:01] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [13:34:01] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=rainworldwiki 42001 --e 44000 (running for 06:08) [13:34:02] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 16:09:01) [13:34:04] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=creaturawiki 830001 (running for 03:06:10) [13:34:04] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=nanowiki 2001 --e 4000 (running for 03:10) [13:36:39] [02puppet] 07paladox closed pull request #4279: changeprop: don't restart service instead reload it on config change (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4279 [13:36:42] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 0334500e9 13https://github.com/miraheze/puppet/commit/34500e9 [13:45:34] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 15.32, 15.93, 23.90 [13:47:08] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 17.99, 20.56, 17.77 [13:47:34] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 25.54, 18.89, 23.97 [13:49:03] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 7.46, 16.19, 16.52 [13:49:34] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 19.29, 18.78, 23.32 [13:49:57] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 33.34, 23.67, 16.67 [13:59:34] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 16.82, 16.74, 20.08 [14:00:48] PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49943MiB (10% inode=99%); [14:03:34] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 30.14, 23.40, 21.97 [14:10:58] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 9.11, 16.24, 22.73 [14:13:59] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 34.91, 22.74, 17.24 [14:14:58] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 11.62, 12.49, 19.61 [14:19:34] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 13.63, 20.76, 23.51 [14:20:42] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/f6d0eeb55872381d2c2a2cd4179bf9bc3ba3a162 [14:20:42] 02ssl/03master 07MacFan4000 03f6d0eeb add wiki.altamc.net [14:25:34] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 27.98, 22.36, 22.95 [14:25:57] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 17.79, 18.23, 23.75 [14:30:58] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.09, 20.29, 18.70 [14:31:57] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 8.81, 13.54, 20.00 [14:32:58] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.61, 21.40, 19.28 [14:34:04] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [14:34:05] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=rainworldwiki 52001 --e 54000 (running for 01:02) [14:34:06] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 17:09:05) [14:34:07] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=creaturawiki 830001 (running for 04:06:13) [14:34:08] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=necrodancerwiki 1 --e 2000 (running for 03:53) [14:41:57] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 32.59, 21.29, 19.62 [14:43:49] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 15.92, 16.39, 23.43 [14:47:18] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/b020a8d017d9 [14:47:18] 02puppet/03paladox-patch-1 07paladox 03b020a8d changeprop: increase LimitNOFILE to 10k in systemd [14:47:22] [02puppet] 07paladox opened pull request #4280: changeprop: increase LimitNOFILE to 10k in systemd (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4280 [14:47:31] [02puppet] 07coderabbitai[bot] commented on pull request #4280: --- […] 13https://github.com/miraheze/puppet/pull/4280#issuecomment-2766478705 [14:47:42] [02puppet] 07paladox merged pull request #4280: changeprop: increase LimitNOFILE to 10k in systemd (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4280 [14:47:42] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/47dd4bcaa55a9d40eb4426a5eb10ab24baac9bda [14:47:42] 02puppet/03master 07paladox 0347dd4bc changeprop: increase LimitNOFILE to 10k in systemd (#4280) [14:47:43] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03b020a8d 13https://github.com/miraheze/puppet/commit/b020a8d [14:47:51] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 27.90, 20.22, 23.09 [14:48:58] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 18.03, 21.73, 23.57 [14:49:50] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 22.87, 20.86, 23.00 [14:50:16] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 16.09, 19.25, 23.56 [14:54:55] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 9.33, 14.13, 19.68 [14:55:44] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 24.86, 23.48, 23.55 [14:57:10] !log [blankeclair@mwtask181] runRefreshLinksChunked: [1300/2761] Handling nesoya17maiwiki (start id: 0) [14:57:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:01:51] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 11.64, 16.16, 19.63 [15:02:03] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [15:02:29] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:00] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3972 bytes in 0.064 second response time [15:04:27] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.333 second response time [15:12:20] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 11.20, 17.72, 23.13 [15:12:47] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.94, 17.28, 16.16 [15:14:46] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 7.57, 14.02, 15.16 [15:16:10] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 5.88, 11.26, 19.29 [15:17:14] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/5cd3ddc565d7 [15:17:14] 02puppet/03paladox-patch-1 07paladox 035cd3ddc mediawiki_task: lower fpm_workers_multiplier to 1.5… [15:17:18] [02puppet] 07paladox opened pull request #4281: mediawiki_task: lower fpm_workers_multiplier to 1.5 (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4281 [15:17:21] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 26.14, 21.24, 19.50 [15:17:26] [02puppet] 07coderabbitai[bot] commented on pull request #4281: --- […] 13https://github.com/miraheze/puppet/pull/4281#issuecomment-2766565348 [15:18:13] [02puppet] 07paladox merged pull request #4281: mediawiki_task: lower fpm_workers_multiplier to 1.5 (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4281 [15:18:13] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/5dd9c0279bc48f089a77fcbe87356e2f5003e5ea [15:18:13] 02puppet/03master 07paladox 035dd9c02 mediawiki_task: lower fpm_workers_multiplier to 1.5 (#4281)… [15:18:14] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 035cd3ddc 13https://github.com/miraheze/puppet/commit/5cd3ddc [15:18:41] !log [blankeclair@mwtask151] runRefreshLinksChunked: [173/1180] reversecrappygameswiki exited with 1 [15:18:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:21:13] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 19.07, 22.21, 20.45 [15:22:14] PROBLEM - mwtask151 jobrunner.wikitide.net HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10002 milliseconds [15:22:17] PROBLEM - mwtask151 videoscaler.wikitide.net HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10004 milliseconds [15:22:22] [02mw-config] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/af8b36fb46fee76f0e6b264bb093392ccdd5c5b7 [15:22:22] 02mw-config/03master 07paladox 03af8b36f Revert "Increase wgFFmpegThreads to 8 (#5886)"… [15:22:28] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:29] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [15:22:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:23:09] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 10.46, 17.55, 18.94 [15:23:16] miraheze/mw-config - paladox the build passed. [15:23:30] !log [paladox@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [15:23:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:23:47] PROBLEM - mwtask181 jobrunner.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:23:50] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [15:23:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:24:13] RECOVERY - mwtask151 jobrunner.wikitide.net HTTPS on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.671 second response time [15:24:16] RECOVERY - mwtask151 videoscaler.wikitide.net HTTPS on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.014 second response time [15:24:22] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 31s [15:24:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:24:28] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.257 second response time [15:25:06] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [15:25:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:25:28] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 22s [15:25:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:25:36] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force_upgrade': True} to all [15:25:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:25:45] RECOVERY - mwtask181 jobrunner.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.010 second response time [15:25:57] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force_upgrade': True} to all - SUCCESS in 21s [15:25:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:27:08] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 25.22, 23.84, 21.36 [15:31:08] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 21.67, 23.77, 22.01 [15:34:06] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [15:34:08] mwtask151: not running [15:34:08] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 18:09:07) [15:34:10] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=creaturawiki 830001 (running for 05:06:16) [15:34:11] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=nicosnextbotswiki 2001 --e 4000 (running for 08:58) [15:35:08] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 6.99, 15.11, 18.92 [15:37:05] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/d08f7aa4d2af [15:37:05] 02puppet/03paladox-patch-1 07paladox 03d08f7aa eventgate: set UV_THREADPOOL_SIZE env in systemd [15:37:08] [02puppet] 07paladox opened pull request #4282: eventgate: set UV_THREADPOOL_SIZE env in systemd (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4282 [15:37:17] [02puppet] 07coderabbitai[bot] commented on pull request #4282: --- […] 13https://github.com/miraheze/puppet/pull/4282#issuecomment-2766621878 [15:39:01] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 18.30, 19.50, 23.79 [15:45:03] [02eventgate-deploy] 07paladox created 03fix (+1 new commit) 13https://github.com/miraheze/eventgate-deploy/commit/1b3d8c496619 [15:45:04] 02eventgate-deploy/03fix 07paladox 031b3d8c4 Fix submodule [15:45:12] [02eventgate-deploy] 07paladox opened pull request #1: Fix submodule (03master...03fix) 13https://github.com/miraheze/eventgate-deploy/pull/1 [15:45:19] [02eventgate-deploy] 07coderabbitai[bot] commented on pull request #1: --- […] 13https://github.com/miraheze/eventgate-deploy/pull/1#issuecomment-2766642655 [15:46:53] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 34.54, 24.85, 23.97 [15:47:08] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 21.56, 19.15, 18.09 [15:49:08] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 18.37, 18.56, 17.99 [15:53:08] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.23, 21.23, 19.20 [15:53:59] [02eventgate-deploy] 07paladox closed pull request #1: Fix submodule (03master...03fix) 13https://github.com/miraheze/eventgate-deploy/pull/1 [15:54:02] [02eventgate-deploy] 07paladox 04deleted 03fix at 031166db5 13https://github.com/miraheze/eventgate-deploy/commit/1166db5 [15:54:30] [02eventgate-deploy] 07paladox opened pull request #2: Fix submodule also update node_modules (03master...03fix) 13https://github.com/miraheze/eventgate-deploy/pull/2 [15:54:37] [02eventgate-deploy] 07coderabbitai[bot] commented on pull request #2: --- […] 13https://github.com/miraheze/eventgate-deploy/pull/2#issuecomment-2766666931 [15:55:08] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 16.56, 19.49, 18.82 [15:57:31] [02eventgate-deploy] 07paladox merged pull request #2: Fix submodule also update node_modules (03master...03fix) 13https://github.com/miraheze/eventgate-deploy/pull/2 [15:57:32] [02eventgate-deploy] 07paladox 04deleted 03fix at 03a0c0fc4 13https://github.com/miraheze/eventgate-deploy/commit/a0c0fc4 [15:58:29] [02puppet] 07paladox merged pull request #4282: eventgate: set UV_THREADPOOL_SIZE env in systemd (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4282 [15:58:30] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/c33c7d920f2737749b24dbc6f5c1c4b1e29246b3 [15:58:30] 02puppet/03master 07paladox 03c33c7d9 eventgate: set UV_THREADPOOL_SIZE env in systemd (#4282) [15:58:31] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03d08f7aa 13https://github.com/miraheze/puppet/commit/d08f7aa [15:58:41] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 15.32, 22.43, 23.95 [15:59:27] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.11, 16.71, 12.64 [16:00:58] PROBLEM - eventgate181 eventgate on eventgate181 is CRITICAL: connect to address 10.0.18.147 and port 8192: Connection refused [16:01:09] [02eventgate-deploy] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/eventgate-deploy/commit/f3bdf4dae34a2b9fc6bc3d788b2a34e872f14ab6 [16:01:09] 02eventgate-deploy/03master 07paladox 03f3bdf4d Fix src checkout commit [16:01:26] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 13.35, 15.37, 12.66 [16:01:44] PROBLEM - eventgate181 Puppet on eventgate181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_eventgate] [16:03:44] RECOVERY - eventgate181 Puppet on eventgate181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:04:35] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 12.79, 15.36, 20.28 [16:08:58] RECOVERY - eventgate181 eventgate on eventgate181 is OK: TCP OK - 0.000 second response time on 10.0.18.147 port 8192 [16:09:44] PROBLEM - eventgate181 Puppet on eventgate181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [16:15:08] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.82, 19.46, 16.39 [16:17:08] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 18.59, 19.10, 16.64 [16:23:14] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 21.02, 19.83, 18.47 [16:25:12] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 15.77, 18.34, 18.10 [16:34:09] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [16:34:10] mwtask151: not running [16:34:11] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 19:09:10) [16:34:12] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=creaturawiki 830001 (running for 06:06:18) [16:34:13] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=nicosnextbotswiki 6001 --e 8000 (running for 01:42) [16:37:10] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.56, 19.27, 14.78 [16:39:03] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.26, 7.00, 5.83 [16:41:03] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.85, 6.73, 5.86 [16:42:21] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php deleteBatch --wiki=jikfjwiki delete310325.txt --u='RhinosF1 (Miraheze)' --r='Request on discord' --i=1 (START) [16:42:22] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php deleteBatch --wiki=jikfjwiki delete310325.txt --u='RhinosF1 (Miraheze)' --r='Request on discord' --i=1 (END - exit=256) [16:42:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:42:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:43:03] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.50, 6.17, 5.75 [16:43:33] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php deleteBatch --wiki=jikfjwiki delete310325.txt --u='RhinosF1 (Miraheze)' --r='Request on discord' --i=1 (START) [16:43:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:45:07] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 29.34, 24.37, 18.69 [16:45:41] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php deleteBatch --wiki=jikfjwiki delete310325.txt --u='RhinosF1 (Miraheze)' --r='Request on discord' --i=1 (END - exit=0) [16:45:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:52:41] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 23.53, 20.97, 18.59 [16:56:37] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 24.46, 22.17, 19.58 [16:57:02] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 18.33, 22.46, 21.36 [16:59:01] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.13, 22.41, 21.42 [17:04:29] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 15.90, 21.32, 20.86 [17:07:32] [02mw-config] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/5f44550ad9fbfd2bb851e39c986d8f06e8a5ab24 [17:07:32] 02mw-config/03master 07paladox 035f44550 Reapply "Increase wgFFmpegThreads to 8 (#5886)"… [17:08:25] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 26.39, 23.16, 21.65 [17:08:31] miraheze/mw-config - paladox the build passed. [17:08:43] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/c6b5de5de518de2c016a0f2366c8a5579f5179bd [17:08:43] 02puppet/03master 07paladox 03c6b5de5 Revert "mediawiki_task: lower fpm_workers_multiplier to 1.5 (#4281)"… [17:08:57] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.71, 23.21, 22.98 [17:10:23] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 16.41, 20.61, 20.91 [17:12:56] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.49, 24.34, 23.42 [17:15:37] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 20.52, 18.84, 15.45 [17:16:18] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 7.78, 15.41, 18.92 [17:17:37] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 13.71, 16.32, 14.91 [17:23:11] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.27, 6.88, 6.23 [17:26:04]