[00:09:45] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 54715MiB (12% inode=99%); [00:13:01] RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 55092MiB (12% inode=99%); [00:40:20] !log [universalomega@mwtask181] starting deploy of {'force': True, 'versions': '1.43', 'upgrade_extensions': 'ManageWiki'} to all [00:40:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:45] !log [universalomega@mwtask181] finished deploy of {'force': True, 'versions': '1.43', 'upgrade_extensions': 'ManageWiki'} to all - SUCCESS in 25s [00:40:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:41:38] [02ManageWiki] 07Universal-Omega merged pull request #676: Remove non-core values from defaults (03main...03core-def) 13https://github.com/miraheze/ManageWiki/pull/676 [00:41:38] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/641768f73ebf25555a24dd4bfb190b293b7da05f [00:41:38] 02ManageWiki/03main 07CosmicAlpha 03641768f Remove non-core values from defaults (#676) [00:41:39] [02ManageWiki] 07Universal-Omega 04deleted 03core-def at 03bd5c46f 13https://api.github.com/repos/miraheze/ManageWiki/commit/bd5c46f [00:46:20] miraheze/ManageWiki - Universal-Omega the build passed. [00:58:15] [02ManageWiki] 07Universal-Omega created 03def-group-exists (+1 new commit) 13https://github.com/miraheze/ManageWiki/commit/1054410b0aee [00:58:15] 02ManageWiki/03def-group-exists 07CosmicAlpha 031054410 Check if DefaultPrivateGroup exists before attempting to remove it [00:58:20] [02ManageWiki] 07Universal-Omega opened pull request #677: Check if DefaultPrivateGroup exists before attempting to remove it (03main...03def-group-exists) 13https://github.com/miraheze/ManageWiki/pull/677 [00:58:26] [02ManageWiki] 07coderabbitai[bot] commented on pull request #677:

[…] 13https://github.com/miraheze/ManageWiki/pull/677#issuecomment-2888685354 [01:01:12] miraheze/ManageWiki - Universal-Omega the build has errored. [01:06:20] miraheze/ManageWiki - Universal-Omega the build passed. [01:27:20] [02ManageWiki] 07Universal-Omega created 03cross-req (+1 new commit) 13https://github.com/miraheze/ManageWiki/commit/7acda9420d40 [01:27:21] 02ManageWiki/03cross-req 07CosmicAlpha 037acda94 Make Requirements cross-database compatible [01:27:25] [02ManageWiki] 07Universal-Omega opened pull request #678: Make Requirements cross-database compatible (03main...03cross-req) 13https://github.com/miraheze/ManageWiki/pull/678 [01:27:31] [02ManageWiki] 07coderabbitai[bot] commented on pull request #678:

[…] 13https://github.com/miraheze/ManageWiki/pull/678#issuecomment-2888694875 [01:30:07] miraheze/ManageWiki - Universal-Omega the build has errored. [01:33:19] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cross-req 13https://github.com/miraheze/ManageWiki/commit/29d85a8cee47b5da3f9f720fe57beb1baf445a2b [01:33:19] 02ManageWiki/03cross-req 07CosmicAlpha 0329d85a8 Update [01:35:55] miraheze/ManageWiki - Universal-Omega the build has errored. [01:36:54] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cross-req 13https://github.com/miraheze/ManageWiki/commit/415cd8818f517d4cd572c6306e9d32834d9f1171 [01:36:54] 02ManageWiki/03cross-req 07CosmicAlpha 03415cd88 Add [01:40:19] miraheze/ManageWiki - Universal-Omega the build has errored. [02:09:17] !log [blankeclair@mwtask181] starting deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'MsCatSelect'} to all [02:09:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:14:07] PROBLEM - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:14:09] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:14:26] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:46] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 59.20, 34.59, 16.21 [02:15:05] PROBLEM - mwtask181 PowerDNS Recursor on mwtask181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:15:13] PROBLEM - mwtask181 jobrunner-high.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [02:15:51] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:16:07] RECOVERY - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 5.235 second response time [02:16:07] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3972 bytes in 0.766 second response time [02:16:08] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:16:22] PROBLEM - mwtask181 mathoid on mwtask181 is CRITICAL: connect to address 10.0.18.106 and port 10044: Connection refused [02:17:58] PROBLEM - mwtask181 jobrunner.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:19:06] PROBLEM - mwtask181 SSH on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:20:26] PROBLEM - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10004 milliseconds [02:20:28] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:21:09] RECOVERY - mwtask181 SSH on mwtask181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u6 (protocol 2.0) [02:21:09] !log [blankeclair@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [02:21:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:21:13] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [02:21:15] wha [02:21:17] RECOVERY - mwtask181 jobrunner-high.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 0.038 second response time [02:21:19] oh [02:21:19] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 18 minutes ago with 0 failures [02:21:27] RECOVERY - mwtask181 PowerDNS Recursor on mwtask181 is OK: DNS OK: 0.048 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205 [02:21:29] Killed apparently [02:22:02] RECOVERY - mwtask181 jobrunner.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 0.039 second response time [02:22:20] RECOVERY - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.019 second response time [02:22:22] RECOVERY - mwtask181 mathoid on mwtask181 is OK: TCP OK - 0.000 second response time on 10.0.18.106 port 10044 [02:22:25] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3972 bytes in 0.078 second response time [02:22:35] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.219 second response time [02:26:50] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 5.20, 21.94, 23.88 [02:30:38] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 3.13, 11.97, 19.39 [03:43:45] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49829MiB (10% inode=99%); [03:53:01] PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49898MiB (10% inode=99%); [04:50:40] [02mw-config] 07BlankEclair opened pull request #5963: T13716: Add utgwiki to localClientDatabases for itself (07miraheze:03main...07BlankEclair:03T13716) 13https://github.com/miraheze/mw-config/pull/5963 [04:51:10] [02mw-config] 07BlankEclair merged pull request #5963: T13716: Add utgwiki to localClientDatabases for itself (07miraheze:03main...07BlankEclair:03T13716) 13https://github.com/miraheze/mw-config/pull/5963 [04:51:11] [02mw-config] 07BlankEclair pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/6b89941ec5ea6b74f38b7f68aa23624c143d78ce [04:51:11] 02mw-config/03main 07Claire 036b89941 T13716: Add utgwiki to localClientDatabases for itself (#5963) [04:51:13] !log [blankeclair@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [04:51:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:51:28] miraheze/mw-config - BlankEclair the build passed. [04:51:48] !log [blankeclair@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 35s [04:51:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:51:59] miraheze/mw-config - BlankEclair the build passed. [05:03:04] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CentralAuth:CreateLocalAccount --wiki=vocaloidlyricswiki Kesera (END - exit=256) [05:03:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:03:18] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CentralAuth:createLocalAccount --wiki=vocaloidlyricswiki Kesera (END - exit=0) [05:03:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:10:47] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CentralAuth:attachAccount --wiki=vocaloidlyricswiki --userlist vocaloidlyricswiki_Keresa.txt (END - exit=0) [05:10:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:11:14] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CentralAuth:attachAccount --wiki=vocaloidlyricswiki --userlist vocaloidlyricswiki_Kesera.txt (END - exit=0) [05:11:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:24:03] !log MediaWiki\Extension\CentralAuth\User\CentralAuthUser::getInstance( User::newFromName( 'Kesera' ) )->attach( 'vocaloidlyricswiki', 'admin' ); [05:24:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:27:17] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 23.02, 14.46, 8.24 [06:29:17] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 7.81, 11.66, 7.98 [06:37:54] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.29, 17.15, 10.63 [06:39:54] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.79, 18.88, 12.06 [06:43:54] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 8.88, 15.91, 12.64 [08:53:54] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.27, 20.15, 14.86 [08:55:54] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 10.36, 17.21, 14.49 [11:41:54] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.63, 17.01, 9.90 [11:45:54] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.28, 20.86, 13.09 [11:47:54] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.90, 21.84, 14.40 [11:53:54] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 6.57, 16.33, 14.87 [11:56:46] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 77 packages available for upgrade (1 critical updates). [11:58:13] PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (1 critical updates). [11:58:34] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 50 packages available for upgrade (1 critical updates). [11:58:42] PROBLEM - db172 APT on db172 is CRITICAL: APT CRITICAL: 55 packages available for upgrade (1 critical updates). [11:59:00] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [11:59:05] PROBLEM - db171 APT on db171 is CRITICAL: APT CRITICAL: 55 packages available for upgrade (1 critical updates). [11:59:19] PROBLEM - bast181 APT on bast181 is CRITICAL: APT CRITICAL: 46 packages available for upgrade (1 critical updates). [11:59:42] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (1 critical updates). [11:59:54] PROBLEM - mw202 APT on mw202 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [11:59:54] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.22, 19.20, 16.14 [12:00:05] PROBLEM - db161 APT on db161 is CRITICAL: APT CRITICAL: 82 packages available for upgrade (1 critical updates). [12:00:30] PROBLEM - mem201 APT on mem201 is CRITICAL: APT CRITICAL: 75 packages available for upgrade (1 critical updates). [12:00:52] PROBLEM - rdb151 APT on rdb151 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (1 critical updates). [12:01:20] PROBLEM - mem161 APT on mem161 is CRITICAL: APT CRITICAL: 74 packages available for upgrade (1 critical updates). [12:01:20] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 54 packages available for upgrade (1 critical updates). [12:01:23] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 47 packages available for upgrade (1 critical updates). [12:01:24] PROBLEM - cp37 APT on cp37 is CRITICAL: APT CRITICAL: 47 packages available for upgrade (1 critical updates). [12:01:26] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [12:01:45] PROBLEM - bast161 APT on bast161 is CRITICAL: APT CRITICAL: 46 packages available for upgrade (1 critical updates). [12:02:06] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 49 packages available for upgrade (1 critical updates). [12:33:54] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.78, 22.91, 22.08 [12:35:54] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.78, 23.25, 22.31 [12:41:54] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 5.29, 15.93, 19.86 [17:19:45] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cross-req 13https://github.com/miraheze/ManageWiki/commit/8a7258bfcb06d03d9cc662b4f6897705b57d8961 [17:19:45] 02ManageWiki/03cross-req 07CosmicAlpha 038a7258b Update [17:22:42] miraheze/ManageWiki - Universal-Omega the build has errored. [17:23:27] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cross-req 13https://github.com/miraheze/ManageWiki/commit/f0083d243cf2bf5bf5905219085ba57e1d93d37e [17:23:27] 02ManageWiki/03cross-req 07CosmicAlpha 03f0083d2 Update [17:28:06] miraheze/ManageWiki - Universal-Omega the build passed. [17:29:36] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cross-req 13https://github.com/miraheze/ManageWiki/commit/939a0f687653c1a47013997e133dfddf9ac84faf [17:29:36] 02ManageWiki/03cross-req 07CosmicAlpha 03939a0f6 Update [17:34:32] miraheze/ManageWiki - Universal-Omega the build passed. [17:51:34] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cross-req 13https://github.com/miraheze/ManageWiki/commit/220d3927759ee8cb074ea7e4f88cf42e932f55ad [17:51:34] 02ManageWiki/03cross-req 07CosmicAlpha 03220d392 Update [17:52:58] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cross-req 13https://github.com/miraheze/ManageWiki/commit/34fbd46c6f1e670655549b60e97b7445cadd182a [17:52:58] 02ManageWiki/03cross-req 07CosmicAlpha 0334fbd46 Update [17:53:28] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cross-req 13https://github.com/miraheze/ManageWiki/commit/306fa181b1d4ddf06a2c10cb971825c1af55cf53 [17:53:28] 02ManageWiki/03cross-req 07CosmicAlpha 03306fa18 Remove [17:53:57] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cross-req 13https://github.com/miraheze/ManageWiki/commit/dd4b5b9a224161f579d4f102dcf99e4b63aff529 [17:53:57] 02ManageWiki/03cross-req 07CosmicAlpha 03dd4b5b9 Update [17:58:45] miraheze/ManageWiki - Universal-Omega the build passed. [19:35:54] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.94, 18.52, 11.64 [19:40:18] RECOVERY - cloud20 APT on cloud20 is OK: APT OK: 136 packages available for upgrade (0 critical updates). [19:45:47] !log [root@puppet181] Upgraded packages on changeprop201: net-tools [19:45:48] !log upgrade net-tools on cloud20 [19:45:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:45:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:45:53] !log running sudo upgrade-packages -y on puppet181 [19:45:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:46:08] !log [root@puppet181] Upgraded packages on cp38: net-tools [19:46:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:46:21] !log [root@puppet181] Upgraded packages on cloud15: net-tools [19:46:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:46:32] !log [root@puppet181] Upgraded packages on mattermost1: net-tools [19:46:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:46:39] RECOVERY - changeprop201 APT on changeprop201 is OK: APT OK: 77 packages available for upgrade (0 critical updates). [19:46:41] RECOVERY - mattermost1 APT on mattermost1 is OK: APT OK: 47 packages available for upgrade (0 critical updates). [19:46:47] !log [root@puppet181] Upgraded packages on cloud19: net-tools [19:46:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:47:01] !log [root@puppet181] Upgraded packages on cp36: net-tools [19:47:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:47:03] RECOVERY - cp38 APT on cp38 is OK: APT OK: 46 packages available for upgrade (0 critical updates). [19:47:05] RECOVERY - cloud19 APT on cloud19 is OK: APT OK: 136 packages available for upgrade (0 critical updates). [19:47:13] !log [root@puppet181] Upgraded packages on db172: net-tools [19:47:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:47:27] !log [root@puppet181] Upgraded packages on db171: net-tools [19:47:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:47:39] !log [root@puppet181] Upgraded packages on matomo151: net-tools [19:47:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:47:44] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [19:47:56] !log [root@puppet181] Upgraded packages on eventgate181: net-tools [19:47:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:09] !log [root@puppet181] Upgraded packages on cp37: net-tools [19:48:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:17] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 49 packages available for upgrade (0 critical updates). [19:48:23] !log [root@puppet181] Upgraded packages on db151: net-tools [19:48:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:38] !log [root@puppet181] Upgraded packages on cloud16: net-tools [19:48:40] RECOVERY - db172 APT on db172 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [19:48:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:46] RECOVERY - cp37 APT on cp37 is OK: APT OK: 46 packages available for upgrade (0 critical updates). [19:48:52] !log [root@puppet181] Upgraded packages on cloud17: net-tools [19:48:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:57] RECOVERY - cp36 APT on cp36 is OK: APT OK: 75 packages available for upgrade (0 critical updates). [19:49:00] RECOVERY - eventgate181 APT on eventgate181 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [19:49:05] RECOVERY - db171 APT on db171 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [19:49:06] !log [root@puppet181] Upgraded packages on db161: net-tools [19:49:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:21] !log [root@puppet181] Upgraded packages on cloud18: net-tools [19:49:22] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 134 packages available for upgrade (0 critical updates). [19:49:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:35] !log [root@puppet181] Upgraded packages on db182: net-tools [19:49:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:49] !log [root@puppet181] Upgraded packages on graylog161: net-tools [19:49:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:52] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 134 packages available for upgrade (0 critical updates). [19:50:03] RECOVERY - db151 APT on db151 is OK: APT OK: 82 packages available for upgrade (0 critical updates). [19:50:04] !log [root@puppet181] Upgraded packages on db181: net-tools [19:50:04] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 134 packages available for upgrade (0 critical updates). [19:50:05] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 48 packages available for upgrade (0 critical updates). [19:50:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:17] !log [root@puppet181] Upgraded packages on bast161: net-tools [19:50:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:32] !log [root@puppet181] Upgraded packages on ldap171: net-tools [19:50:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:35] RECOVERY - ldap171 APT on ldap171 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [19:50:46] !log [root@puppet181] Upgraded packages on kafka181: net-tools [19:50:48] RECOVERY - db182 APT on db182 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [19:50:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:53] RECOVERY - db161 APT on db161 is OK: APT OK: 81 packages available for upgrade (0 critical updates). [19:50:59] !log [root@puppet181] Upgraded packages on bots171: net-tools [19:51:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:15] !log [root@puppet181] Upgraded packages on bast181: net-tools [19:51:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:23] RECOVERY - bots171 APT on bots171 is OK: APT OK: 48 packages available for upgrade (0 critical updates). [19:51:28] !log [root@puppet181] Upgraded packages on mem151: net-tools [19:51:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:42] !log [root@puppet181] Upgraded packages on mem161: net-tools [19:51:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:45] RECOVERY - bast161 APT on bast161 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [19:51:49] RECOVERY - bast181 APT on bast181 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [19:51:49] RECOVERY - db181 APT on db181 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [19:51:52] RECOVERY - kafka181 APT on kafka181 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [19:51:55] !log [root@puppet181] Upgraded packages on mem201: net-tools [19:51:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:09] !log [root@puppet181] Upgraded packages on mem191: net-tools [19:52:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:24] !log [root@puppet181] Upgraded packages on mw152: net-tools [19:52:24] RECOVERY - mem191 APT on mem191 is OK: APT OK: 44 packages available for upgrade (0 critical updates). [19:52:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:30] RECOVERY - mem201 APT on mem201 is OK: APT OK: 74 packages available for upgrade (0 critical updates). [19:52:40] !log [root@puppet181] Upgraded packages on mw162: net-tools [19:52:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:56] !log [root@puppet181] Upgraded packages on mw161: net-tools [19:52:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:12] !log [root@puppet181] Upgraded packages on mw151: net-tools [19:53:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:17] RECOVERY - mem161 APT on mem161 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [19:53:22] RECOVERY - mw161 APT on mw161 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:53:26] RECOVERY - mem151 APT on mem151 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [19:53:27] !log [root@puppet181] Upgraded packages on mw153: net-tools [19:53:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:43] !log [root@puppet181] Upgraded packages on mw163: net-tools [19:53:45] RECOVERY - mw162 APT on mw162 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:53:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:59] !log [root@puppet181] Upgraded packages on mw171: net-tools [19:54:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:03] RECOVERY - mw152 APT on mw152 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:54:15] !log [root@puppet181] Upgraded packages on mw192: net-tools [19:54:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:31] !log [root@puppet181] Upgraded packages on mw172: net-tools [19:54:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:42] RECOVERY - mw151 APT on mw151 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:54:47] !log [root@puppet181] Upgraded packages on mw193: net-tools [19:54:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:53] RECOVERY - mw172 APT on mw172 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:54:54] RECOVERY - mw192 APT on mw192 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:55:02] !log [root@puppet181] Upgraded packages on mw173: net-tools [19:55:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:06] RECOVERY - mw171 APT on mw171 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:55:17] !log [root@puppet181] Upgraded packages on mw191: net-tools [19:55:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:23] RECOVERY - mw153 APT on mw153 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:55:32] !log [root@puppet181] Upgraded packages on mw201: net-tools [19:55:39] RECOVERY - mw163 APT on mw163 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:55:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:48] !log [root@puppet181] Upgraded packages on mw202: net-tools [19:55:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:51] RECOVERY - mw191 APT on mw191 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:55:54] RECOVERY - mw202 APT on mw202 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:55:58] RECOVERY - mw201 APT on mw201 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:56:05] !log [root@puppet181] Upgraded packages on mon181: net-tools [19:56:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:22] !log [root@puppet181] Upgraded packages on mw181: net-tools [19:56:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:31] RECOVERY - mw173 APT on mw173 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:56:38] !log [root@puppet181] Upgraded packages on mw183: net-tools [19:56:39] RECOVERY - mw193 APT on mw193 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:56:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:53] !log [root@puppet181] Upgraded packages on mw203: net-tools [19:56:54] RECOVERY - mon181 APT on mon181 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [19:56:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:01] RECOVERY - mw183 APT on mw183 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:57:06] RECOVERY - mw181 APT on mw181 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:57:11] !log [root@puppet181] Upgraded packages on mw182: net-tools [19:57:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:26] !log [root@puppet181] Upgraded packages on mwtask151: net-tools [19:57:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:42] !log [root@puppet181] Upgraded packages on mwtask161: net-tools [19:57:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:57] !log [root@puppet181] Upgraded packages on mwtask171: net-tools [19:58:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:02] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:58:11] !log [root@puppet181] Upgraded packages on phorge171: net-tools [19:58:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:24] !log [root@puppet181] Upgraded packages on prometheus151: net-tools [19:58:24] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 47 packages available for upgrade (0 critical updates). [19:58:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:38] !log [root@puppet181] Upgraded packages on os161: net-tools [19:58:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:42] RECOVERY - mw203 APT on mw203 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:58:46] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:58:52] !log [root@puppet181] Upgraded packages on reports171: net-tools [19:58:54] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:58:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:58] RECOVERY - mw182 APT on mw182 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [19:59:08] !log [root@puppet181] Upgraded packages on ns1: net-tools [19:59:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:23] !log [root@puppet181] Upgraded packages on os162: net-tools [19:59:30] RECOVERY - os161 APT on os161 is OK: APT OK: 46 packages available for upgrade (0 critical updates). [19:59:34] RECOVERY - os162 APT on os162 is OK: APT OK: 48 packages available for upgrade (0 critical updates). [19:59:37] !log [root@puppet181] Upgraded packages on rdb151: net-tools [19:59:39] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 49 packages available for upgrade (0 critical updates). [19:59:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:50] RECOVERY - reports171 APT on reports171 is OK: APT OK: 47 packages available for upgrade (0 critical updates). [19:59:52] !log [root@puppet181] Upgraded packages on os151: net-tools [19:59:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:06] !log [root@puppet181] Upgraded packages on puppet181: net-tools [20:00:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:17] RECOVERY - os151 APT on os151 is OK: APT OK: 48 packages available for upgrade (0 critical updates). [20:00:21] !log [root@puppet181] Upgraded packages on os191: net-tools [20:00:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:29] RECOVERY - ns1 APT on ns1 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [20:00:42] !log [root@puppet181] Upgraded packages on mwtask181: net-tools [20:00:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:47] RECOVERY - rdb151 APT on rdb151 is OK: APT OK: 75 packages available for upgrade (0 critical updates). [20:00:56] !log [root@puppet181] Upgraded packages on test151: net-tools [20:00:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:06] RECOVERY - os191 APT on os191 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [20:01:12] !log [root@puppet181] Upgraded packages on swiftproxy161: net-tools [20:01:13] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:01:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:20] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [20:01:30] !log [root@puppet181] Upgraded packages on swiftobject151: net-tools [20:01:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:40] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 74 packages available for upgrade (0 critical updates). [20:01:48] !log [root@puppet181] Upgraded packages on swiftproxy171: net-tools [20:01:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:04] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [20:02:04] !log [root@puppet181] Upgraded packages on swiftac171: net-tools [20:02:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:14] RECOVERY - test151 APT on test151 is OK: APT OK: 69 packages available for upgrade (0 critical updates). [20:02:21] !log [root@puppet181] Upgraded packages on swiftobject201: net-tools [20:02:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:39] !log [root@puppet181] Upgraded packages on swiftobject161: net-tools [20:02:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:42] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [20:02:56] !log [root@puppet181] Upgraded packages on swiftobject191: net-tools [20:02:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:12] !log [root@puppet181] Upgraded packages on swiftobject181: net-tools [20:03:16] RECOVERY - swiftobject201 APT on swiftobject201 is OK: APT OK: 75 packages available for upgrade (0 critical updates). [20:03:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:31] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 75 packages available for upgrade (0 critical updates). [20:03:31] !log [root@puppet181] Upgraded packages on swiftobject171: net-tools [20:03:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:42] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 79 packages available for upgrade (0 critical updates). [20:03:46] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 74 packages available for upgrade (0 critical updates). [20:03:49] !log [root@puppet181] Upgraded packages on ns2: net-tools [20:03:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:44] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 74 packages available for upgrade (0 critical updates). [20:04:48] RECOVERY - swiftobject191 APT on swiftobject191 is OK: APT OK: 75 packages available for upgrade (0 critical updates). [20:04:52] RECOVERY - ns2 APT on ns2 is OK: APT OK: 71 packages available for upgrade (0 critical updates). [20:13:01] PROBLEM - cp38 Disk Space on cp38 is CRITICAL: DISK CRITICAL - free space: / 27254MiB (5% inode=99%); [20:33:45] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27179MiB (5% inode=99%); [20:37:54] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 15.50, 20.52, 23.81 [20:39:09] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'speleo.wiki' expires in 1 day(s) (Tue 20 May 2025 08:21:38 PM GMT +0000). [20:43:54] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 9.71, 13.99, 19.86 [21:13:54] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.03, 19.56, 16.83 [21:15:54] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 13.67, 17.36, 16.38 [21:35:54] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.67, 20.61, 17.80 [21:37:54] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.02, 21.89, 18.60 [21:39:54] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.69, 22.68, 19.31 [21:41:54] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 23.38, 19.98 [22:05:54] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.64, 23.92, 23.49 [22:07:54] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.25, 24.86, 23.89 [22:09:54] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.61, 23.27, 23.43 [22:19:54] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 13.47, 15.78, 19.59 [22:36:57] [02mw-config] 07Universal-Omega created 03update-req (+1 new commit) 13https://github.com/miraheze/mw-config/commit/51ae098399fe [22:36:57] 02mw-config/03update-req 07CosmicAlpha 0351ae098 Update requires options [22:37:01] [02mw-config] 07Universal-Omega opened pull request #5964: Update requires options (03main...03update-req) 13https://github.com/miraheze/mw-config/pull/5964 [22:37:57] miraheze/mw-config - Universal-Omega the build passed. [22:37:58] [02mw-config] 07Universal-Omega pushed 1 new commit to 03update-req 13https://github.com/miraheze/mw-config/commit/eb489669990d16a49476f41a670e7757f113fb4b [22:37:58] 02mw-config/03update-req 07CosmicAlpha 03eb48966 Update [22:38:48] miraheze/mw-config - Universal-Omega the build passed. [22:39:53] [02mw-config] 07Universal-Omega pushed 1 new commit to 03update-req 13https://github.com/miraheze/mw-config/commit/70be60f38b75e64bac2a56bdc7272f1fd55a6d6d [22:39:54] 02mw-config/03update-req 07CosmicAlpha 0370be60f Update [22:40:43] miraheze/mw-config - Universal-Omega the build passed. [22:41:27] [02mw-config] 07Universal-Omega pushed 1 new commit to 03update-req 13https://github.com/miraheze/mw-config/commit/123afc31b7592a5e337c40d4fac900e625868902 [22:41:27] 02mw-config/03update-req 07CosmicAlpha 03123afc3 Update [22:42:22] miraheze/mw-config - Universal-Omega the build passed. [22:43:04] [02mw-config] 07Universal-Omega pushed 1 new commit to 03update-req 13https://github.com/miraheze/mw-config/commit/c9d148d60ae35d5c699ada798f38f25e78297fe6 [22:43:04] 02mw-config/03update-req 07CosmicAlpha 03c9d148d Update [22:43:53] miraheze/mw-config - Universal-Omega the build passed. [22:44:14] [02mw-config] 07Universal-Omega pushed 1 new commit to 03update-req 13https://github.com/miraheze/mw-config/commit/dc22104c14e4c2ab7223a3ff42e668ed75843362 [22:44:14] 02mw-config/03update-req 07CosmicAlpha 03dc22104 Update [22:45:11] miraheze/mw-config - Universal-Omega the build passed. [22:45:16] [02mw-config] 07Universal-Omega pushed 1 new commit to 03update-req 13https://github.com/miraheze/mw-config/commit/de5e2ea0fbcc63ffc76fffc94ef6267d0532ad3f [22:45:16] 02mw-config/03update-req 07CosmicAlpha 03de5e2ea Update [22:45:50] [02mw-config] 07Universal-Omega pushed 1 new commit to 03update-req 13https://github.com/miraheze/mw-config/commit/9ad1c466c19346f37c981e5cbd487824300fe26f [22:45:50] 02mw-config/03update-req 07CosmicAlpha 039ad1c46 Update [22:46:08]