[00:11:41] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 54019MiB (11% inode=99%); [00:16:02] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': ['1.42', '1.43'], 'upgrade_extensions': ['MirahezeMagic', 'RemovePII']} to all [00:16:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:17] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.42', '1.43', '1.44'], 'upgrade_extensions': ['MirahezeMagic', 'RemovePII']} to test151 [00:16:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:19:13] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.42', '1.43', '1.44'], 'upgrade_extensions': ['MirahezeMagic', 'RemovePII']} to test151 - SUCCESS in 176s [00:19:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:19:55] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': ['1.42', '1.43'], 'upgrade_extensions': ['MirahezeMagic', 'RemovePII']} to all - SUCCESS in 232s [00:19:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:11:41] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49836MiB (10% inode=99%); [03:23:52] PROBLEM - lostidols.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [03:33:34] PROBLEM - www.lostidols.wiki - Cloudflare on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [04:48:00] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [04:50:00] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.19 ms [12:10:28] [02MatomoAnalytics] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/MatomoAnalytics/commit/7b42f6570a5c2a7494f37a8067e865e796103351 [12:10:28] 02MatomoAnalytics/03master 07translatewiki.net 037b42f65 Localisation updates from https://translatewiki.net. [12:10:30] [02CreateWiki] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/58b3e73da1815347d7bfe8673cc424743f6ee6c8 [12:10:32] [02DataDump] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/DataDump/commit/e95a57fa4df7213723a5624daf65f0f06d06f0fb [12:10:34] 02DataDump/03master 07translatewiki.net 03e95a57f Localisation updates from https://translatewiki.net. [12:10:35] 02CreateWiki/03master 07translatewiki.net 0358b3e73 Localisation updates from https://translatewiki.net. [12:14:50] miraheze/DataDump - translatewiki the build passed. [12:15:06] miraheze/MatomoAnalytics - translatewiki the build passed. [12:21:22] miraheze/CreateWiki - translatewiki the build passed. [12:51:45] RECOVERY - db171 APT on db171 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [12:52:01] RECOVERY - mw153 APT on mw153 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [12:52:03] RECOVERY - mw183 APT on mw183 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [12:52:05] RECOVERY - mw184 APT on mw184 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [12:52:05] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 62 packages available for upgrade (0 critical updates). [12:52:10] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 62 packages available for upgrade (0 critical updates). [12:52:10] RECOVERY - test151 APT on test151 is OK: APT OK: 69 packages available for upgrade (0 critical updates). [12:52:12] RECOVERY - mw173 APT on mw173 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [12:52:21] RECOVERY - db181 APT on db181 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [12:52:23] RECOVERY - mw163 APT on mw163 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [12:52:23] RECOVERY - db182 APT on db182 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [12:52:25] RECOVERY - mw171 APT on mw171 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [12:52:31] RECOVERY - db172 APT on db172 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [12:52:33] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 62 packages available for upgrade (0 critical updates). [12:52:35] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [12:52:40] RECOVERY - mw152 APT on mw152 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [12:52:42] RECOVERY - db161 APT on db161 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [12:52:43] !log upgrade rsync on all servers [12:52:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:52:46] RECOVERY - mw174 APT on mw174 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [12:52:47] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [12:52:49] RECOVERY - graphite151 APT on graphite151 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [12:52:53] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [12:52:54] RECOVERY - mw151 APT on mw151 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [12:53:03] RECOVERY - mw182 APT on mw182 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [12:53:04] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 64 packages available for upgrade (0 critical updates). [12:53:04] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 63 packages available for upgrade (0 critical updates). [12:53:05] RECOVERY - mw164 APT on mw164 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [12:53:08] RECOVERY - mw162 APT on mw162 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [12:53:09] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [12:53:11] RECOVERY - mw172 APT on mw172 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [12:53:18] RECOVERY - db151 APT on db151 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [12:53:23] RECOVERY - mw181 APT on mw181 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [12:53:35] RECOVERY - mw154 APT on mw154 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [12:53:38] RECOVERY - mw161 APT on mw161 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [12:53:51] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [12:54:15] !log [paladox@mwtask181] [root@mwtask181:/srv/mediawiki/1.42/maintenance]# sudo -u www-data php refreshLinks.php --wiki kanrikyarawiki - T13084 [12:54:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:55:33] PROBLEM - edapedia.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.804B2E2685140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1605:SSL alert number 40 [13:06:42] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 141.44 ms [13:08:40] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 141.45 ms [13:25:09] RECOVERY - edapedia.net - LetsEncrypt on sslhost is OK: OK - Certificate 'edapedia.net' will expire on Mon 07 Apr 2025 01:52:00 AM GMT +0000. [19:29:41] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27219MiB (5% inode=99%); [22:30:44] PROBLEM - mem161 APT on mem161 is CRITICAL: APT CRITICAL: 59 packages available for upgrade (1 critical updates). [22:31:13] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 62 packages available for upgrade (1 critical updates). [22:31:14] PROBLEM - cp37 APT on cp37 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (1 critical updates). [22:31:16] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (1 critical updates). [22:31:35] PROBLEM - eventgate181 APT on eventgate181 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (1 critical updates). [22:32:07] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 64 packages available for upgrade (1 critical updates). [22:32:21] PROBLEM - db181 APT on db181 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (1 critical updates). [22:32:22] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 65 packages available for upgrade (1 critical updates). [22:32:43] PROBLEM - bast161 APT on bast161 is CRITICAL: APT CRITICAL: 59 packages available for upgrade (1 critical updates). [22:34:01] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 61 packages available for upgrade (1 critical updates). [22:34:23] PROBLEM - db182 APT on db182 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (1 critical updates). [22:34:25] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [22:34:41] PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates). [22:35:00] PROBLEM - mem151 APT on mem151 is CRITICAL: APT CRITICAL: 59 packages available for upgrade (1 critical updates). [22:35:12] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [22:36:10] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [22:36:16] PROBLEM - mw154 APT on mw154 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [22:37:09] PROBLEM - mw164 APT on mw164 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [22:38:06] PROBLEM - ldap171 APT on ldap171 is CRITICAL: APT CRITICAL: 59 packages available for upgrade (1 critical updates). [22:39:08] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [22:39:30] PROBLEM - jobchron171 APT on jobchron171 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (1 critical updates). [22:40:09] PROBLEM - mw184 APT on mw184 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [22:40:36] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [22:41:10] PROBLEM - phorge171 APT on phorge171 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (1 critical updates). [22:41:24] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 61 packages available for upgrade (1 critical updates). [22:41:38] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [22:42:47] PROBLEM - kafka181 APT on kafka181 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (1 critical updates). [22:44:12] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [22:44:36] PROBLEM - mw174 APT on mw174 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [22:44:50] PROBLEM - mattermost1 APT on mattermost1 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (1 critical updates). [22:45:48] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (1 critical updates). [22:46:10] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (1 critical updates). [22:46:30] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (1 critical updates). [22:46:50] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [22:47:43] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [22:48:29] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [22:48:47] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 46 packages available for upgrade (1 critical updates). [22:49:12] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [22:49:18] PROBLEM - db151 APT on db151 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (1 critical updates). [22:49:52] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [22:50:03] PROBLEM - mw183 APT on mw183 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [22:50:33] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (1 critical updates). [22:50:50] PROBLEM - cp36 APT on cp36 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (1 critical updates). [22:53:10] PROBLEM - changeprop151 APT on changeprop151 is CRITICAL: APT CRITICAL: 65 packages available for upgrade (1 critical updates). [22:54:05] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 59 packages available for upgrade (1 critical updates). [22:55:26] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [22:56:11] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (1 critical updates). [22:56:13] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 70 packages available for upgrade (1 critical updates). [22:56:21] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [22:56:56] PROBLEM - db172 APT on db172 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (1 critical updates). [22:57:05] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 65 packages available for upgrade (1 critical updates). [22:57:50] PROBLEM - bots171 APT on bots171 is CRITICAL: APT CRITICAL: 71 packages available for upgrade (1 critical updates). [22:57:55] PROBLEM - graphite151 APT on graphite151 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates). [22:58:12] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (1 critical updates). [22:58:20] PROBLEM - bast181 APT on bast181 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (1 critical updates). [22:58:21] PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (1 critical updates). [22:58:42] PROBLEM - db161 APT on db161 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (1 critical updates). [22:59:05] PROBLEM - db171 APT on db171 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (1 critical updates). [22:59:06] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 64 packages available for upgrade (1 critical updates). [22:59:24] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [22:59:48] PROBLEM - rdb151 APT on rdb151 is CRITICAL: APT CRITICAL: 61 packages available for upgrade (1 critical updates).