[00:03:10] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49104MiB (10% inode=99%); [00:05:10] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 58926MiB (12% inode=99%); [00:06:29] RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 58897MiB (12% inode=99%); [00:11:53] !log [oa@mwtask181] starting deploy of {'world': True, 'versions': '1.43', 'upgrade_extensions': 'UserProfileV2'} to all [00:11:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:14:30] !log [oa@mwtask181] finished deploy of {'world': True, 'versions': '1.43', 'upgrade_extensions': 'UserProfileV2'} to all - SUCCESS in 157s [00:14:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:22:53] miraheze/mw-config - Aeywoo the build passed. [00:28:58] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [00:28:59] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki [00:29:00] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=noelsimssandboxwiki 1 --e 2000 [00:29:01] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=kylariswiki 12001 --e 14000 [00:29:02] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 4001 --e 6000 [01:29:01] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [01:29:02] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki [01:29:03] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=noelsimssandboxwiki 1 --e 2000 [01:29:04] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=kylariswiki 12001 --e 14000 [01:29:05] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 4001 --e 6000 [02:05:19] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/0c5bd2149c632cb9007e765fa9df0747ec458de4 [02:05:19] 02ssl/03master 07MacFan4000 030c5bd21 add wiki.thunderscientific.solutions [02:29:04] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [02:29:05] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki [02:29:06] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=noelsimssandboxwiki 1 --e 2000 [02:29:07] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=kylariswiki 12001 --e 14000 [02:29:08] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 4001 --e 6000 [02:58:25] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.92, 15.73, 8.97 [03:02:25] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.43, 18.36, 11.77 [03:29:07] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [03:29:08] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki [03:29:09] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=noelsimssandboxwiki 1 --e 2000 [03:29:10] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=kylariswiki 12001 --e 14000 [03:29:11] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 4001 --e 6000 [04:29:10] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [04:29:11] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki [04:29:12] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=noelsimssandboxwiki 1 --e 2000 [04:29:13] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=kylariswiki 12001 --e 14000 [04:29:14] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 4001 --e 6000 [04:54:30] Is it just me, or has the refreshLinks stuff not made any progress at all today? [05:00:35] i think it's slow [05:01:15] oh hmm [05:01:19] looks like nothing changed [05:01:28] uhhhh i'll get on it once i'm freed from blahaj [05:09:18] Might need to up the priority or something, not sure [05:09:52] my yubikey is all the way over *points to bedside table from the other end of the bed* there [05:10:01] and i'm all the way over *strokes blahaj* here [05:29:13] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [05:29:14] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki [05:29:15] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=noelsimssandboxwiki 1 --e 2000 [05:29:16] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=kylariswiki 12001 --e 14000 [05:29:17] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 4001 --e 6000 [05:37:30] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:38:12] PROBLEM - mwtask151 php-fpm on mwtask151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [05:38:32] PROBLEM - mwtask151 SSH on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:38:51] PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:38:53] PROBLEM - mwtask151 jobrunner.wikitide.net HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10002 milliseconds [05:38:56] PROBLEM - mwtask151 videoscaler.wikitide.net HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:39:19] PROBLEM - mwtask151 jobrunner-high.wikitide.net HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [05:39:26] PROBLEM - mwtask151 PowerDNS Recursor on mwtask151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:40:11] PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:40:21] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 91.65, 61.59, 28.34 [05:40:22] RECOVERY - mwtask151 php-fpm on mwtask151 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [05:40:28] RECOVERY - mwtask151 SSH on mwtask151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u5 (protocol 2.0) [05:40:49] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3972 bytes in 0.067 second response time [05:40:51] RECOVERY - mwtask151 jobrunner.wikitide.net HTTPS on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.011 second response time [05:40:53] RECOVERY - mwtask151 videoscaler.wikitide.net HTTPS on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.013 second response time [05:41:16] RECOVERY - mwtask151 jobrunner-high.wikitide.net HTTPS on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.012 second response time [05:41:25] RECOVERY - mwtask151 PowerDNS Recursor on mwtask151 is OK: DNS OK: 0.046 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205 [05:41:28] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.179 second response time [05:42:08] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures [05:46:05] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 5.78, 23.63, 21.52 [05:47:59] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 6.45, 18.22, 19.79 [06:29:16] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [06:29:17] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki [06:29:18] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=noelsimssandboxwiki 1 --e 2000 [06:29:19] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=kylariswiki 12001 --e 14000 [06:29:20] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 4001 --e 6000 [07:29:19] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [07:29:20] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki [07:29:21] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=noelsimssandboxwiki 1 --e 2000 [07:29:22] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=kylariswiki 12001 --e 14000 [07:29:23] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 4001 --e 6000 [08:29:22] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [08:29:23] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki [08:29:24] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=noelsimssandboxwiki 1 --e 2000 [08:29:25] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=kylariswiki 12001 --e 14000 [08:29:26] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 4001 --e 6000 [08:52:25] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.68, 17.56, 11.96 [09:00:38] PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 16273MiB (7% inode=95%); [09:10:25] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 12.80, 19.88, 19.43 [09:29:25] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [09:29:26] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki [09:29:27] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=noelsimssandboxwiki 1 --e 2000 [09:29:28] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=kylariswiki 12001 --e 14000 [09:29:29] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 4001 --e 6000 [10:29:28] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [10:29:29] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki [10:29:30] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=noelsimssandboxwiki 1 --e 2000 [10:29:31] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=kylariswiki 12001 --e 14000 [10:29:32] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 4001 --e 6000 [10:37:56] [1/2] Something is certainly happening here [10:37:57] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1354766385032462346/2025-03-27_21-37.png?ex=67e67c04&is=67e52a84&hm=639ac2115d295db61a3b4e5f94ddbdd2689c7382f04502c2446d3b4543ed5673& [10:38:27] [1/2] deadlock? [10:38:27] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1354766511276949576/2025-03-27_21-38.png?ex=67e67c22&is=67e52aa2&hm=2dace801cb68423b56983e0ea3316a7d4f7681fbc4991982f442e75d84bc4370& [10:39:39] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 1200 --e 1300 (START) [10:39:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:39:42] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 1200 --e 1300 (END - exit=0) [10:39:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:39:50] ugh [10:39:53] we restarting them scripts then [10:40:36] i may as well add a method for specifying the start parameter for a wiki [11:02:49] thanks script for not stopping [11:03:05] !log manually killed refreshLinks on mwtask181 because it was deadlocked [11:03:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:03:57] !log [blankeclair@mwtask181] runRefreshLinksChunked: [0/2761] Handling cravesagawiki (start id: 1200) [11:03:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:06:49] [1/3] mwtask151 still definitely running, dunno if deadlocked in the bad way though [11:06:50] [2/3] on id 10600 [11:06:50] [3/3] https://cdn.discordapp.com/attachments/808001911868489748/1354773652989546607/2025-03-27_22-06.png?ex=67e682c9&is=67e53149&hm=d56e5db3a3047c3a22a945bb29660197ad404b7b1fba84e2c279a53d02ed8113& [11:08:20] !log manually killed refreshLinks on mwtask161 because it was deadlocked [11:08:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:09:14] !log [blankeclair@mwtask161] runRefreshLinksChunked: [0/1417] Handling noelsimssandboxwiki (start id: 900) [11:09:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:10:19] hmm, p sure i passed the wrong start id for mwtask181 [11:10:21] oh well, already running [11:10:53] values for my ref: 12001 + 1500 [11:10:59] 17000, k cool [11:11:35] [02ssl] 07RhinosF1 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/0f9c5ddac50fecf099c9ec88104c7c89c5ee2187 [11:11:35] 02ssl/03master 07RhinosF1 030f9c5dd update… [11:11:53] !log [blankeclair@mwtask171] runRefreshLinksChunked: [0/1842] Handling kylariswiki (start id: 17000) [11:11:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:12:25] !log manually killed mwtask171, both for deadlocked and script fuckery [11:12:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:15:18] something's weird with mwtask171 bash i think [11:16:27] or hmm, maybe it's `cut`? [11:16:58] oh fuck me [11:17:03] my script is broken [11:18:22] !log manually killed refreshLinks on mwtask181 to load new script [11:18:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:18:58] !log [blankeclair@mwtask181] runRefreshLinksChunked: [0/2761] Handling cravesagawiki (start id: 5300) [11:19:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:19:43] !log manually killed refreshLinks on mwtask161 to load new script [11:19:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:20:23] !log [blankeclair@mwtask161] runRefreshLinksChunked: [0/1417] Handling noelsimssandboxwiki (start id: 12700) [11:20:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:20:38] PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: DISK CRITICAL - free space: / 9825MiB (4% inode=95%); [11:20:45] !log [blankeclair@mwtask171] runRefreshLinksChunked: [0/1842] Handling kylariswiki (start id: 17000) [11:20:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:22:08] time to check back on mwtask151 [11:22:24] bad deadlock imo [11:22:41] !log manually kill refreshLinks on mwtask151 due to bad deadlock [11:22:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:23:02] constantnoblewiki 10600 [11:23:35] !log [blankeclair@mwtask151] runRefreshLinksChunked: [0/2792] Handling constantnoblewiki (start id: 10600) [11:23:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:24:44] > mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki 10601 --e 12600 [11:24:46] > mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=noelsimssandboxwiki 12701 --e 14700 [11:24:47] > mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=laboramericawiki 1 --e 2000 [11:24:49] > mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 5301 --e 7300 [11:24:50] let's hope this changes [11:26:47] !log kill refreshLinks on mwtask151 to add --verbose to the script (still 100% cpu deadlocking!) [11:26:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:27:10] !log never mind, i was about to kill it, but it started running [11:27:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:29:31] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [11:29:32] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki 10601 --e 12600 [11:29:33] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=noelsimssandboxwiki 14701 --e 16700 [11:29:34] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=laboramericawiki 1 --e 2000 [11:29:35] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 5301 --e 7300 [11:34:57] PROBLEM - with.cpt-ra.bid - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [11:36:33] [02mw-config] 07RhinosF1 pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/fe47ca9d4831374dbde4ac7da1ec21014611a6cf [11:36:33] 02mw-config/03master 07Aeywoo 03fe47ca9 Uninstall RefreshSpecial (#5881)… [11:36:35] [02mw-config] 07RhinosF1 merged 07Aeywoo's pull request #5881: Uninstall RefreshSpecial (07miraheze:03master...07Aeywoo:03patch-4) 13https://github.com/miraheze/mw-config/pull/5881 [11:36:55] !log [rhinos@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [11:36:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:37:18] !log [rhinos@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [11:37:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:37:26] miraheze/mw-config - RhinosF1 the build passed. [11:56:28] PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 14944MiB (6% inode=95%); [12:04:43] RECOVERY - with.cpt-ra.bid - Cloudflare on sslhost is OK: OK - Certificate 'with.cpt-ra.bid' will expire on Sat 24 May 2025 07:05:41 PM GMT +0000. [12:11:29] [02CreateWiki] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/108b2e5929be037130d90219638995037cf7e7dd [12:11:29] 02CreateWiki/03master 07translatewiki.net 03108b2e5 Localisation updates from https://translatewiki.net. [12:11:30] [02landing] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/landing/commit/20ff2a297fd9b0fcdc34dd2a0a6f74422db842de [12:11:30] 02landing/03master 07translatewiki.net 0320ff2a2 Localisation updates from https://translatewiki.net. [12:11:32] [02ManageWiki] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/ManageWiki/commit/fdced75239fe41e4e06161bf596efd6ec31ceae9 [12:11:34] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/MirahezeMagic/commit/c51cbcef11417f277f4d357c82728a434fcb7356 [12:11:36] 02MirahezeMagic/03master 07translatewiki.net 03c51cbce Localisation updates from https://translatewiki.net. [12:11:38] [02WikiDiscover] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/WikiDiscover/commit/3a2928b6c0bdac47c915cdf9e3bccc72866c43b6 [12:11:40] 02ManageWiki/03master 07translatewiki.net 03fdced75 Localisation updates from https://translatewiki.net. [12:11:42] [02DiscordNotifications] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/DiscordNotifications/commit/a2b7c43937a154c47c28b6adda7974470e1597db [12:11:43] 02WikiDiscover/03master 07translatewiki.net 033a2928b Localisation updates from https://translatewiki.net. [12:11:45] 02DiscordNotifications/03master 07translatewiki.net 03a2b7c43 Localisation updates from https://translatewiki.net. [12:12:29] miraheze/landing - translatewiki the build passed. [12:14:47] miraheze/MirahezeMagic - translatewiki the build has errored. [12:17:19] miraheze/WikiDiscover - translatewiki the build passed. [12:18:09] miraheze/ManageWiki - translatewiki the build passed. [12:26:01] miraheze/CreateWiki - translatewiki the build passed. [12:29:34] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [12:29:35] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki 10601 --e 12600 [12:29:36] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=osseanfmwiki 1 --e 2000 [12:29:37] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=lethalwiki 6001 --e 8000 [12:29:38] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 5301 --e 7300 [12:31:29] miraheze/DiscordNotifications - translatewiki the build passed. [12:34:07] !log [blankeclair@mwtask161] runRefreshLinksChunked: [100/1417] Handling osvaldatorewiki (start id: 0) [12:34:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:34:14] oh neat [13:29:37] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [13:29:38] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki 10601 --e 12600 [13:29:39] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=ovenbreakwiki 2001 --e 4000 [13:29:40] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=lingofantasywiki 2001 [13:29:41] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 5301 --e 7300 [13:33:05] !log [www-data@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/CampaignEvents.php /srv/mediawiki/1.43/maintenance/run.php CampaignEvents:UpdateUTCTimestamps (START) [13:33:06] !log [www-data@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/CampaignEvents.php /srv/mediawiki/1.43/maintenance/run.php CampaignEvents:UpdateUTCTimestamps (END - exit=256) [13:33:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:33:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:33:27] !log [www-data@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CampaignEvents:UpdateUTCTimestamps --wiki=metawiki (END - exit=256) [13:33:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:38:20] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: rhinosf1, last run 5 minutes ago with 0 failures [13:40:21] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:45:17] [02puppet] 07RhinosF1 requested changes on pull request #4251: Secrets need to be different between beta and prod 13https://github.com/miraheze/puppet/pull/4251#pullrequestreview-2721576045 [13:46:59] PROBLEM - you.r-fit.cc - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [13:57:16] PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: DISK CRITICAL - free space: / 13273MiB (5% inode=95%); [14:01:06] miraheze/mw-config - RhinosF1 the build passed. [14:16:43] RECOVERY - you.r-fit.cc - Cloudflare on sslhost is OK: OK - Certificate 'you.r-fit.cc' will expire on Sat 24 May 2025 05:41:10 PM GMT +0000. [14:29:40] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [14:29:41] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki 10601 --e 12600 [14:29:42] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=ovenbreakwiki 4001 --e 6000 [14:29:43] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=looneypyramidswiki 2001 --e 4000 [14:29:44] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 5301 --e 7300 [14:36:08] !log [blankeclair@mwtask171] runRefreshLinksChunked: [100/1842] Handling lostzenwikiwiki (start id: 0) [14:36:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:37:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:29:43] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [15:29:44] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki 10601 --e 12600 [15:29:45] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=ovenbreakwiki 4001 --e 6000 [15:29:46] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=ltswiki 2001 --e 4000 [15:29:47] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 5301 --e 7300 [16:27:52] PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49934MiB (10% inode=99%); [16:28:29] !log [blankeclair@mwtask171] runRefreshLinksChunked: [200/1842] Handling meingartenwiki (start id: 0) [16:28:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:29:46] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [16:29:47] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki 10601 --e 12600 [16:29:48] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=ovenbreakwiki 4001 --e 6000 [16:29:49] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=meingartenwiki 1 --e 2000 [16:29:50] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 5301 --e 7300 [16:57:20] [02puppet] 07Universal-Omega commented on pull request #4251: We only need one version on here though. That can be done via heira on private puppet. 13https://github.com/miraheze/puppet/pull/4251#issuecomment-2758774576 [16:58:58] [02puppet] 07RhinosF1 commented on pull request #4251: Then the database password bit should be redone to avoid that and it documented. 13https://github.com/miraheze/puppet/pull/4251#issuecomment-2758780409 [17:08:02] [02WikiDiscover] 07Universal-Omega pushed 1 new commit to 03api-extend-ApiQueryGeneratorBase 13https://github.com/miraheze/WikiDiscover/commit/6e5397841a5e4ca57451e7226292cbb3f878b284 [17:08:03] 02WikiDiscover/03api-extend-ApiQueryGeneratorBase 07CosmicAlpha 036e53978 Remove editcount for now… [17:12:40] miraheze/WikiDiscover - Universal-Omega the build passed. [17:29:50] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [17:29:51] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki 12601 --e 14600 [17:29:52] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=phightingwiki 2001 --e 4000 [17:29:53] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=mgqwiki 16001 --e 18000 [17:29:54] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 5301 --e 7300 [17:44:14] PROBLEM - ourlifewiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ourlifewiki.org' expires in 14 day(s) (Fri 11 Apr 2025 05:41:16 PM GMT +0000). [17:44:28] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/9b4a7c8db48adf7db731653663de2442d6dbf597 [17:44:28] 02ssl/03master 07WikiTideSSLBot 039b4a7c8 Bot: Update SSL cert for ourlifewiki.org [17:54:39] PROBLEM - sptwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sptwiki.org' expires in 14 day(s) (Fri 11 Apr 2025 05:33:12 PM GMT +0000). [17:54:52] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/ee86850a7fd7f7d837275a6fa6d5d4d740469b38 [17:54:52] 02ssl/03master 07WikiTideSSLBot 03ee86850 Bot: Update SSL cert for sptwiki.org [18:02:46] !log [blankeclair@mwtask161] runRefreshLinksChunked: [200/1417] Handling pizzatabewiki (start id: 0) [18:02:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:03:52] PROBLEM - infectowiki.com - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [18:12:58] RECOVERY - ourlifewiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ourlifewiki.org' will expire on Wed 25 Jun 2025 04:45:50 PM GMT +0000. [18:24:27] RECOVERY - sptwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'sptwiki.org' will expire on Wed 25 Jun 2025 04:56:14 PM GMT +0000. [18:29:53] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [18:29:54] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki 12601 --e 14600 [18:29:55] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=pluspiwiki 1 --e 2000 [18:29:56] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=mockelectionswiki 1 --e 2000 [18:29:57] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 5301 --e 7300 [18:37:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [18:54:24] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php runJobs --wiki=loginwiki (START) [18:54:25] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php runJobs --wiki=loginwiki (END - exit=0) [18:54:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:55:59] !log [rhinos@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php runJobs (START) [18:56:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:04:43] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.95, 20.24, 14.80 [19:08:43] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.55, 19.78, 15.81 [19:29:56] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [19:29:57] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki 12601 --e 14600 [19:29:58] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=politycznabdwiki 2001 [19:29:59] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=mockelectionswiki 16001 --e 18000 [19:30:00] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 5301 --e 7300 [19:52:03] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:AssignImportedEdits --wiki=avidwiki --import-prefix=avid --from=N0ah (END - exit=0) [19:52:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:21] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=projectsomniawiki --username=Dandylion50 (END - exit=256) [19:55:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:47] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=projectsomniawiki Dandylion50 (END - exit=0) [19:55:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:45] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CentralAuth/maintenance/migrateAccount.php --wiki=projectsomniawiki --auto --username=Dandylion50 --attachmissing --attachbroken (END - exit=0) [19:57:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:43] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.61, 18.35, 13.28 [20:12:43] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 20.12, 19.24, 14.26 [20:16:43] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.79, 20.13, 15.67 [20:18:43] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.75, 19.76, 16.07 [20:20:14] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php exxtensions/CentralAuth/attachAccount --wiki=testwiki 1.txt (END - exit=256) [20:20:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:20:32] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php extensions/CentralAuth/attachAccount --wiki=testwiki 1.txt (END - exit=256) [20:20:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:01] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CentralAuth/maintenance/attachAccount.php --wiki=testwiki 1.txt (END - exit=256) [20:21:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:26] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CentralAuth/maintenance/attachAccount.php --wiki=testwiki --userlist=1.txt (END - exit=0) [20:21:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:29] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CentralAuth/maintenance/attachAccount.php --wiki=projectsomniawiki --userlist=2.txt (END - exit=0) [20:22:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:43] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.68, 23.29, 18.33 [20:29:58] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CentralAuth/maintenance/attachAccount.php --wiki=projectsomniawiki --userlist=2.txt (END - exit=0) [20:29:59] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [20:30:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:30:00] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki 12601 --e 14600 [20:30:01] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=precurewikiwiki 1 --e 2000 [20:30:02] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=mockelectionswiki 34001 --e 36000 [20:30:03] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 5301 --e 7300 [20:36:43] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 14.49, 22.27, 21.74 [20:38:43] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 8.59, 17.47, 20.04 [20:49:02] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=pickaxegamewiki (END - exit=0) [20:49:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:43] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.77, 19.06, 14.42 [21:14:27] !log [blankeclair@mwtask161] runRefreshLinksChunked: [300/1417] Handling proshogiwiki (start id: 0) [21:14:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:43] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.22, 20.56, 15.58 [21:16:43] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 18.79, 20.32, 16.12 [21:20:43] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.90, 21.00, 17.43 [21:28:43] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.41, 22.94, 19.70 [21:30:03] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [21:30:04] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki 12601 --e 14600 [21:30:05] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=puritywiki 1 --e 2000 [21:30:06] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=mockelectionswiki 50001 --e 52000 [21:30:07] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 5301 --e 7300 [21:30:43] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.99, 22.35, 19.89 [21:34:55] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [8 system event log (SEL) entries present] [21:40:43] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 11.60, 18.34, 19.62 [21:41:12] !log bouncing kafka to clear it's default.mediawiki.job.LocalGlobalUserPageCacheUpdateJob topic [21:41:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [22:00:18] !log [blankeclair@mwtask181] Starting import for kaiserreichwiki (XML: dump.xml; Images: None) (START) [22:00:19] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importDump --wiki=kaiserreichwiki --no-updates -- dump.xml (START) [22:00:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:00:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:24:11] !log [rhinos@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php runJobs (END - exit=0) [22:24:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:30:06] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [22:30:07] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki 12601 --e 14600 [22:30:08] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=r2wiki 48001 --e 50000 [22:30:09] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=mockupediawiki 1 --e 2000 [22:30:10] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 5301 --e 7300 [22:52:43] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importDump --wiki=kaiserreichwiki --no-updates -- dump.xml (END - exit=0) [22:52:44] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php rebuildall --wiki=kaiserreichwiki (START) [22:52:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:52:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:43] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php rebuildall --wiki=kaiserreichwiki (END - exit=0) [23:05:44] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initEditCount --wiki=kaiserreichwiki (START) [23:05:45] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initEditCount --wiki=kaiserreichwiki (END - exit=0) [23:05:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:46] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initSiteStats --wiki=kaiserreichwiki --update (START) [23:05:47] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initSiteStats --wiki=kaiserreichwiki --update (END - exit=0) [23:05:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:48] !log [blankeclair@mwtask181] Finished import for kaiserreichwiki (XML: dump.xml; Images: None) (END - exit=0) [23:05:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:08:24] !log [blankeclair@mwtask161] runRefreshLinksChunked: [400/1417] Handling remnantprosperawiki (start id: 0) [23:08:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:08:28] !log [blankeclair@mwtask161] runRefreshLinksChunked: [401/1417] renamewikitest2wiki exited with 1 [23:08:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:09] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [23:30:10] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=constantnoblewiki 12601 --e 14600 [23:30:11] mwtask161: not running [23:30:12] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=mogaiwiki 24001 --e 26000 [23:30:13] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=cravesagawiki 5301 --e 7300 [23:56:43] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.06, 20.27, 12.92 [23:58:43] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.47, 20.83, 14.04