[00:00:35] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.10 ms [00:03:47] PROBLEM - changeprop151 Current Load on changeprop151 is CRITICAL: LOAD CRITICAL - total load average: 8.36, 20.48, 14.66 [00:05:44] PROBLEM - changeprop151 Current Load on changeprop151 is WARNING: LOAD WARNING - total load average: 3.65, 14.89, 13.30 [00:07:41] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 6.46, 13.42, 13.00 [00:16:22] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 50500MiB (11% inode=99%); [00:34:01] !log [void@puppet181] Upgraded packages git-man, and git on changeprop151 [00:34:15] RECOVERY - changeprop151 APT on changeprop151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:34:15] !log [void@puppet181] Upgraded packages git-man, and git on cp36 [00:34:28] !log [void@puppet181] Upgraded packages git-man, and git on jobchron171 [00:34:42] !log [void@puppet181] Upgraded packages git-man, and git on bots171 [00:34:56] !log [void@puppet181] Upgraded packages git-man, and git on db172 [00:35:11] !log [void@puppet181] Upgraded packages git-man, and git on eventgate181 [00:35:20] RECOVERY - jobchron171 APT on jobchron171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:35:25] !log [void@puppet181] Upgraded packages git-man, and git on graylog161 [00:35:35] RECOVERY - bots171 APT on bots171 is OK: APT OK: 70 packages available for upgrade (0 critical updates). [00:35:38] !log [void@puppet181] Upgraded packages git-man, and git on mem151 [00:35:45] RECOVERY - cp36 APT on cp36 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [00:35:51] !log [void@puppet181] Upgraded packages git-man, and git on graphite151 [00:35:56] RECOVERY - graphite151 APT on graphite151 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [00:35:57] RECOVERY - mem151 APT on mem151 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [00:36:05] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 62 packages available for upgrade (0 critical updates). [00:36:05] !log [void@puppet181] Upgraded packages git-man, and git on cp37 [00:36:19] !log [void@puppet181] Upgraded packages git-man, and git on db171 [00:36:33] !log [void@puppet181] Upgraded packages git-man, and git on db151 [00:36:35] RECOVERY - db171 APT on db171 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [00:36:48] !log [void@puppet181] Upgraded packages git-man, and git on matomo151 [00:36:53] RECOVERY - db172 APT on db172 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [00:36:53] RECOVERY - eventgate181 APT on eventgate181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:37:01] !log [void@puppet181] Upgraded packages git-man, and git on db181 [00:37:15] !log [void@puppet181] Upgraded packages git-man, and git on db161 [00:37:24] RECOVERY - db151 APT on db151 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [00:37:29] !log [void@puppet181] Upgraded packages git-man, and git on db182 [00:37:32] RECOVERY - db161 APT on db161 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [00:37:38] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 44 packages available for upgrade (0 critical updates). [00:37:49] !log [void@puppet181] Upgraded packages git-man, and git on bast181 [00:37:54] RECOVERY - cp37 APT on cp37 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [00:38:02] !log [void@puppet181] Upgraded packages git-man, and git on kafka181 [00:38:08] RECOVERY - kafka181 APT on kafka181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:38:17] !log [void@puppet181] Upgraded packages git-man, and git on ldap171 [00:38:22] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49813MiB (10% inode=99%); [00:38:27] RECOVERY - db182 APT on db182 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [00:38:33] !log [void@puppet181] Upgraded packages git-man, and git on bast161 [00:38:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:38:47] !log [void@puppet181] Upgraded packages git-man, and git on cloud18 [00:38:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:38:50] !log [void@bots171] restart logbot [00:38:51] RECOVERY - bast161 APT on bast161 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [00:38:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:38:53] RECOVERY - db181 APT on db181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:39:01] !log [void@puppet181] Upgraded packages git-man, and git on cloud16 [00:39:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:14] !log [void@puppet181] Upgraded packages git-man, and git on mattermost1 [00:39:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:29] !log [void@puppet181] Upgraded packages git-man, and git on cloud17 [00:39:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:32] RECOVERY - bast181 APT on bast181 is OK: APT OK: 59 packages available for upgrade (0 critical updates). [00:39:42] RECOVERY - ldap171 APT on ldap171 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [00:39:43] !log [void@puppet181] Upgraded packages git-man, and git on cloud15 [00:39:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:46] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 63 packages available for upgrade (0 critical updates). [00:39:57] !log [void@puppet181] Upgraded packages git-man, and git on mem161 [00:39:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:03] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 63 packages available for upgrade (0 critical updates). [00:40:12] !log [void@puppet181] Upgraded packages git-man, and git on mon181 [00:40:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:15] RECOVERY - mon181 APT on mon181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:40:22] RECOVERY - mem161 APT on mem161 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [00:40:28] !log [void@puppet181] Upgraded packages git-man, and git on mw152 [00:40:29] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 63 packages available for upgrade (0 critical updates). [00:40:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:33] RECOVERY - mw152 APT on mw152 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [00:40:45] !log [void@puppet181] Upgraded packages git-man, and git on mw151 [00:40:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:58] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 63 packages available for upgrade (0 critical updates). [00:41:01] !log [void@puppet181] Upgraded packages git-man, and git on mw153 [00:41:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:41:13] RECOVERY - mattermost1 APT on mattermost1 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [00:41:17] !log [void@puppet181] Upgraded packages git-man, and git on mw162 [00:41:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:41:23] RECOVERY - mw151 APT on mw151 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [00:41:33] !log [void@puppet181] Upgraded packages git-man, and git on mw171 [00:41:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:41:47] RECOVERY - mw162 APT on mw162 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [00:41:49] !log [void@puppet181] Upgraded packages git-man, and git on mw161 [00:41:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:42:05] !log [void@puppet181] Upgraded packages git-man, and git on mw172 [00:42:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:42:21] !log [void@puppet181] Upgraded packages git-man, and git on mw164 [00:42:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:42:35] RECOVERY - mw153 APT on mw153 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [00:42:38] !log [void@puppet181] Upgraded packages git-man, and git on mw163 [00:42:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:42:44] RECOVERY - mw161 APT on mw161 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [00:42:56] !log [void@puppet181] Upgraded packages git-man, and git on mw182 [00:42:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:43:06] RECOVERY - mw172 APT on mw172 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [00:43:10] RECOVERY - mw163 APT on mw163 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [00:43:12] !log [void@puppet181] Upgraded packages git-man, and git on ns1 [00:43:12] [02mw-config] 07OAuthority pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/6a6203f1cb69faf2af415a5095ac75bd961a2b9f [00:43:12] 02mw-config/03master 07Edward Chernenko 036a6203f Fix typo in variable name ($wgUploadStashMaxAge) (#5791) [00:43:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:43:19] RECOVERY - mw171 APT on mw171 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [00:43:29] !log [void@puppet181] Upgraded packages git-man, and git on mw173 [00:43:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:43:42] miraheze/mw-config - edwardspec the build passed. [00:43:45] !log [void@puppet181] Upgraded packages git-man, and git on mwtask161 [00:43:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:43:54] miraheze/mw-config - OAuthority the build passed. [00:43:57] RECOVERY - ns1 APT on ns1 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [00:43:57] RECOVERY - mw182 APT on mw182 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [00:44:02] !log [void@puppet181] Upgraded packages git-man, and git on mw174 [00:44:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:07] RECOVERY - mw173 APT on mw173 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [00:44:12] RECOVERY - mw164 APT on mw164 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [00:44:17] !log [void@puppet181] Upgraded packages git-man, and git on os151 [00:44:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:34] !log [void@puppet181] Upgraded packages git-man, and git on mw181 [00:44:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:50] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [00:44:51] !log [void@puppet181] Upgraded packages git-man, and git on mwtask171 [00:44:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:58] RECOVERY - os151 APT on os151 is OK: APT OK: 60 packages available for upgrade (0 critical updates). [00:45:02] [02mw-config] 07OAuthority merged 07edwardspec's pull request #5791: Fix typo in variable name ($wgUploadStashMaxAge) (07miraheze:03master...07edwardspec:03master) 13https://github.com/miraheze/mw-config/pull/5791 [00:45:08] !log [void@puppet181] Upgraded packages git-man, and git on mwtask151 [00:45:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:45:25] !log [void@puppet181] Upgraded packages git-man, and git on mw183 [00:45:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:45:29] RECOVERY - mw174 APT on mw174 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [00:45:42] !log [void@puppet181] Upgraded packages git-man, and git on mw154 [00:45:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:45:48] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [00:45:52] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [00:46:01] !log [void@puppet181] Upgraded packages git-man, and git on mw184 [00:46:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:46:20] !log [void@puppet181] Upgraded packages git-man, and git on mwtask181 [00:46:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:46:24] RECOVERY - mw181 APT on mw181 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [00:46:35] !log [void@puppet181] Upgraded packages git-man, and git on os161 [00:46:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:46:49] !log [void@puppet181] Upgraded packages git-man, and git on phorge171 [00:46:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:46:55] RECOVERY - mw154 APT on mw154 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [00:46:58] RECOVERY - mw184 APT on mw184 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [00:47:00] RECOVERY - mw183 APT on mw183 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [00:47:02] !log [void@puppet181] Upgraded packages git-man, and git on prometheus151 [00:47:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:47:09] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [00:47:15] !log [void@puppet181] Upgraded packages git-man, and git on rdb151 [00:47:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:47:34] !log [void@puppet181] Upgraded packages git-man, and git on ns2 [00:47:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:47:44] RECOVERY - os161 APT on os161 is OK: APT OK: 59 packages available for upgrade (0 critical updates). [00:47:49] !log [void@puppet181] Upgraded packages git-man, and git on os162 [00:47:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:47:52] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:48:04] !log [void@puppet181] Upgraded packages git-man, and git on reports171 [00:48:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:48:05] RECOVERY - ns2 APT on ns2 is OK: APT OK: 59 packages available for upgrade (0 critical updates). [00:48:10] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [00:48:18] RECOVERY - rdb151 APT on rdb151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:48:19] !log [void@puppet181] Upgraded packages git-man, and git on puppet181 [00:48:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:48:33] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 63 packages available for upgrade (0 critical updates). [00:48:37] !log [void@puppet181] Upgraded packages git-man, and git on test151 [00:48:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:48:48] RECOVERY - reports171 APT on reports171 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [00:48:53] !log [void@puppet181] Upgraded packages git-man, and git on swiftac171 [00:48:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:49:09] !log [void@puppet181] Upgraded packages git-man, and git on swiftproxy161 [00:49:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:49:26] !log [void@puppet181] Upgraded packages git-man, and git on swiftobject171 [00:49:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:49:36] RECOVERY - os162 APT on os162 is OK: APT OK: 60 packages available for upgrade (0 critical updates). [00:49:43] !log [void@puppet181] Upgraded packages git-man, and git on swiftproxy171 [00:49:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:49:45] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 64 packages available for upgrade (0 critical updates). [00:50:00] !log [void@puppet181] Upgraded packages git-man, and git on swiftobject151 [00:50:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:50:16] RECOVERY - test151 APT on test151 is OK: APT OK: 69 packages available for upgrade (0 critical updates). [00:50:17] !log [void@puppet181] Upgraded packages git-man, and git on swiftobject161 [00:50:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:50:27] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 62 packages available for upgrade (0 critical updates). [00:50:35] !log [void@puppet181] Upgraded packages git-man, and git on swiftobject181 [00:50:35] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 62 packages available for upgrade (0 critical updates). [00:50:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:50:57] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 64 packages available for upgrade (0 critical updates). [00:51:12] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [00:51:18] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 63 packages available for upgrade (0 critical updates). [00:51:52] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 62 packages available for upgrade (0 critical updates). [00:57:25] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.16 ms [00:59:27] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.18 ms [01:03:48] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [01:05:49] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.11 ms [01:30:21] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [01:34:30] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.08 ms [02:21:14] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [02:27:30] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.06 ms [02:47:06] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [02:57:30] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.10 ms [03:17:10] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.16 ms [03:36:09] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.11 ms [03:57:55] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [03:59:57] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.10 ms [04:52:45] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 70%, RTA = 31.64 ms [04:56:48] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.07 ms [06:20:08] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 31.16 ms [06:22:09] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.10 ms [06:30:46] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [06:32:47] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.10 ms [06:50:34] !log sudo -u www-data php /srv/mediawiki/1.42/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki outlasterwiki [06:50:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:50:58] !log sudo -u www-data php /srv/mediawiki/1.42/extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki outlasterwiki --skipParse [06:50:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:51:11] !log sudo -u www-data php /srv/mediawiki/1.42/extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki outlasterwiki --skipLinks --indexOnSkip (logged in wrong order) [06:51:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:58:01] !log same as below for jbcstudioswiki and animalroyalewiki [06:58:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:43:31] RECOVERY - wiki.cowgl.xyz - Cloudflare on sslhost is OK: OK - Certificate 'wiki.cowgl.xyz' will expire on Wed 26 Mar 2025 06:11:33 PM GMT +0000. [08:44:05] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 31.16 ms [08:46:06] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.07 ms [09:50:15] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [09:52:17] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.09 ms [10:19:03] [02mw-config] 07BlankEclair opened pull request #5792: T13129: Set $wgVectorNightMode for roguetown2ewiki (07miraheze:03master...07BlankEclair:03T13129) 13https://github.com/miraheze/mw-config/pull/5792 [10:19:55] miraheze/mw-config - BlankEclair the build passed. [10:23:13] [02mw-config] 07BlankEclair opened pull request #5793: T13130: Set some default options for ssbuniversewiki (07miraheze:03master...07BlankEclair:03T13130) 13https://github.com/miraheze/mw-config/pull/5793 [10:24:12] miraheze/mw-config - BlankEclair the build passed. [11:58:44] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 31.12 ms [12:02:52] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.10 ms [12:10:17] [02CreateWiki] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/346f2410d43fde01559648a81c080b475005949d [12:10:17] 02CreateWiki/03master 07translatewiki.net 03346f241 Localisation updates from https://translatewiki.net. [12:10:19] [02ImportDump] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/ImportDump/commit/46bcdf92509c0b62a141aed6268a6c9c68c3c629 [12:10:20] 02ImportDump/03master 07translatewiki.net 0346bcdf9 Localisation updates from https://translatewiki.net. [12:10:22] [02MatomoAnalytics] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/MatomoAnalytics/commit/80d894879ede1aad96630243011c545c99a1a064 [12:10:24] [02landing] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/landing/commit/1781e0a440c493b09b5d03b8c8f85ed6dc17910a [12:10:26] [02WikiDiscover] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/WikiDiscover/commit/d80b410dbda6c18977ce401a46d6037353aa5400 [12:10:28] 02WikiDiscover/03master 07translatewiki.net 03d80b410 Localisation updates from https://translatewiki.net. [12:10:29] 02MatomoAnalytics/03master 07translatewiki.net 0380d8948 Localisation updates from https://translatewiki.net. [12:10:31] 02landing/03master 07translatewiki.net 031781e0a Localisation updates from https://translatewiki.net. [12:10:33] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/MirahezeMagic/commit/dcec3eec4ab0debc9d4020a41b04e49f640552f1 [12:10:35] 02MirahezeMagic/03master 07translatewiki.net 03dcec3ee Localisation updates from https://translatewiki.net. [12:11:06] miraheze/landing - translatewiki the build passed. [12:15:16] miraheze/WikiDiscover - translatewiki the build passed. [12:15:22] miraheze/MatomoAnalytics - translatewiki the build passed. [12:21:06] miraheze/CreateWiki - translatewiki the build passed. [12:21:19] miraheze/ImportDump - translatewiki the build passed. [12:21:34] miraheze/MirahezeMagic - translatewiki the build passed. [12:23:46] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw151.wikitide.net port 443 after 0 ms: Couldn't connect to server [12:23:59] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: connect to address 10.0.15.114 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:25:46] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.218 second response time [12:26:00] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.742 second response time [13:42:22] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27179MiB (5% inode=99%); [14:38:12] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 31.06 ms [14:40:13] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.15 ms [14:42:49] PROBLEM - comprehensibleinputwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'comprehensibleinputwiki.org' expires in 14 day(s) (Tue 11 Feb 2025 02:39:50 PM GMT +0000). [14:43:01] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/2e7299fabdb637739b200dc4ae6aa6ca3e2a448f [14:43:01] 02ssl/03master 07WikiTideSSLBot 032e7299f Bot: Update SSL cert for comprehensibleinputwiki.org [15:03:43] PROBLEM - geeu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'geeu.org' expires in 14 day(s) (Tue 11 Feb 2025 02:36:30 PM GMT +0000). [15:03:57] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/a70837055249540124275154f7f68a4cc96469bc [15:03:57] 02ssl/03master 07WikiTideSSLBot 03a708370 Bot: Update SSL cert for geeu.org [15:11:38] RECOVERY - comprehensibleinputwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'comprehensibleinputwiki.org' will expire on Sun 27 Apr 2025 01:44:26 PM GMT +0000. [15:32:27] RECOVERY - geeu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'geeu.org' will expire on Sun 27 Apr 2025 02:05:19 PM GMT +0000. [17:08:41] [02mw-config] 07AgentIsai created 03143-upgrade from 03master (+0 new commit) 13https://github.com/miraheze/mw-config/compare/143-upgrade [17:09:54] [02mw-config] 07AgentIsai pushed 1 new commit to 03143-upgrade 13https://github.com/miraheze/mw-config/commit/c9876c2fc50e86ad8a3dd695286a81cd2c60748d [17:09:54] 02mw-config/03143-upgrade 07Agent Isai 03c9876c2 T12840: Upgrade to 1.43 [17:10:10] [02mw-config] 07AgentIsai opened pull request #5794: T12840: Upgrade to MediaWiki 1.43 (03master...03143-upgrade) 13https://github.com/miraheze/mw-config/pull/5794 [17:10:38] [02mw-config] 07github-actions[bot] pushed 1 new commit to 03143-upgrade 13https://github.com/miraheze/mw-config/commit/87280bc5f843df45e6d6a2e6c2636dbc6c395ad4 [17:10:39] 02mw-config/03143-upgrade 07github-actions 0387280bc CI: lint code to MediaWiki standards… [17:11:08] miraheze/mw-config - AgentIsai the build passed. [17:14:19] [02puppet] 07AgentIsai created 03143-upgrade from 03master (+0 new commit) 13https://github.com/miraheze/puppet/compare/143-upgrade [17:14:48] [02puppet] 07AgentIsai pushed 1 new commit to 03143-upgrade 13https://github.com/miraheze/puppet/commit/daa41a7f43419cd778000a38c95fe86fb8a8a1cf [17:14:50] 02puppet/03143-upgrade 07Agent Isai 03daa41a7 T12840: Upgrade to MediaWiki 1.43 [17:15:02] [02puppet] 07AgentIsai opened pull request #4149: T12840: Upgrade to MediaWiki 1.43 (03master...03143-upgrade) 13https://github.com/miraheze/puppet/pull/4149 [17:15:09] [02puppet] 07coderabbitai[bot] commented on pull request #4149: --- […] 13https://github.com/miraheze/puppet/pull/4149#issuecomment-2616409873 [17:16:57] miraheze/puppet - AgentIsai the build passed. [17:26:52] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/6da8a3d6cf2abfbd1f5bc3a6823aa44a7b0f7809 [17:26:53] 02ssl/03master 07MacFan4000 036da8a3d remove www.durawiki.com (expired) [17:28:57] [02dns] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/fcb820b52c8224c5f821b62581e14a9e2ad56b76 [17:28:57] 02dns/03master 07MacFan4000 03fcb820b Delete zones/durawiki.com [17:30:01] miraheze/dns - MacFan4000 the build passed. [17:55:35] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-recentchanges-rc_id-bigint.sql (START) [17:55:36] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-revision-cleanup.sql (START) [17:55:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:55:37] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-change_tag-ct_rc_id.sql (START) [17:55:38] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-searchindex-pk-titlelength.sql (START) [17:55:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:55:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:55:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:56:12] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-searchindex-pk-titlelength.sql (END - exit=2) [17:56:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:57:14] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-searchindex-pk-titlelength.sql (START) [17:57:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:59:37] [02puppet] 07AgentIsai merged pull request #4149: T12840: Upgrade to MediaWiki 1.43 (03master...03143-upgrade) 13https://github.com/miraheze/puppet/pull/4149 [17:59:37] [02puppet] 07AgentIsai pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/03a4b7e8d142cd1811949c3ece6d41ab73c99f3f [17:59:38] 02puppet/03master 07Agent Isai 0303a4b7e T12840: Upgrade to MediaWiki 1.43 (#4149) [17:59:40] [02puppet] 07AgentIsai 04deleted 03143-upgrade at 03daa41a7 13https://github.com/miraheze/puppet/commit/daa41a7 [18:00:14] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [18:00:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:00:36] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [18:00:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:01:11] miraheze/puppet - AgentIsai the build passed. [18:13:57] [02mw-config] 07AgentIsai merged pull request #5794: T12840: Upgrade to MediaWiki 1.43 (03master...03143-upgrade) 13https://github.com/miraheze/mw-config/pull/5794 [18:13:57] [02mw-config] 07AgentIsai pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/e1a46cb11db561202dcfa4475fba08633f965623 [18:13:57] 02mw-config/03master 07Agent Isai 03e1a46cb T12840: Upgrade to MediaWiki 1.43 (#5794)… [18:13:58] [02mw-config] 07AgentIsai 04deleted 03143-upgrade at 0387280bc 13https://github.com/miraheze/mw-config/commit/87280bc [18:14:50] miraheze/mw-config - AgentIsai the build passed. [18:25:30] PROBLEM - mw181 MediaWiki Rendering on mw181 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.055 second response time [18:25:34] PROBLEM - mw184 MediaWiki Rendering on mw184 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.044 second response time [18:26:18] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/cuci_wiki_map.sql (START) [18:26:19] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/cuci_user.sql (START) [18:26:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:20] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-pagelinks-drop-pl_title.sql (START) [18:26:21] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/patch-cu_changes-drop-cuc_actiontext.sql (START) [18:26:22] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/cuci_temp_edit.sql (START) [18:26:23] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/patch-cu_changes-drop-cuc_only_for_read_old.sql (START) [18:26:24] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/patch-cu_changes-drop-cuc_private.sql (START) [18:26:25] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-page-page_links_updated-noinfinite.sql (START) [18:26:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:27:04] PROBLEM - mw182 MediaWiki Rendering on mw182 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.055 second response time [18:27:59] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/maintenance/archives/patch-page-page_links_updated-noinfinite.sql (START) [18:28:00] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/maintenance/archives/patch-pagelinks-drop-pl_title.sql (START) [18:28:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:01] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/CheckUser/schema/mysql/cuci_user.sql (START) [18:28:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:02] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/CheckUser/schema/mysql/cuci_temp_edit.sql (START) [18:28:03] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/CheckUser/schema/mysql/patch-cu_changes-drop-cuc_private.sql (START) [18:28:04] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/CheckUser/schema/mysql/patch-cu_changes-drop-cuc_actiontext.sql (START) [18:28:05] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/CheckUser/schema/mysql/cuci_wiki_map.sql (START) [18:28:06] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/CheckUser/schema/mysql/patch-cu_changes-drop-cuc_only_for_read_old.sql (START) [18:28:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:29:25] !log [agent@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/fixAutoblockLogTitles.php (START) [18:29:38] !log [agent@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CheckUser/maintenance/deleteReadOldRowsInCuChanges.php (START) [18:29:57] !log [agent@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CheckUser/maintenance/populateCentralCheckUserIndexTables.php (START) [18:30:01] !log [agent@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CheckUser/maintenance/populateCentralCheckUserIndexTables.php (END - exit=2) [18:30:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:30:37] !log [agent@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CheckUser/maintenance/populateCentralCheckUserIndexTables.php (START) [18:34:53] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.202 second response time [18:35:15] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.212 second response time [18:35:25] RECOVERY - mw184 MediaWiki Rendering on mw184 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.219 second response time [18:40:07] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CentralAuth/maintenance/migrateGuSalt.php --wiki=metawiki (END - exit=0) [18:40:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:43:42] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/DiscussionTools.json /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/DiscussionTools/maintenance/FixTrailingWhitespaceIds.php (START) [18:43:43] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/DiscussionTools.json /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/DiscussionTools/maintenance/FixTrailingWhitespaceIds.php (END - exit=256) [18:43:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:43:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:43:52] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/discussiontools.json /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/DiscussionTools/maintenance/FixTrailingWhitespaceIds.php (START) [18:43:53] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/discussiontools.json /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/DiscussionTools/maintenance/FixTrailingWhitespaceIds.php (END - exit=256) [18:43:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:43:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:48:23] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/GlobalBlocking/maintenance/UpdateAutoBlockParentIdColumn.php --wiki=metawiki (END - exit=0) [18:48:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:48:35] [02puppet] 07AgentIsai pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/b1b8bd8a42dc9476285a203a329adefb9202c650 [18:48:35] 02puppet/03master 07Agent Isai 03b1b8bd8 mwscript: always return .php [18:50:09] miraheze/puppet - AgentIsai the build has errored. [18:54:48] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/abusefilter.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/AbuseFilter/db_patches/mysql/patch-drop-af_user.sql (START) [18:54:49] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/abusefilter.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/AbuseFilter/db_patches/mysql/patch-drop-afh_user.sql (START) [18:54:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:50] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/abusefilter.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/AbuseFilter/db_patches/mysql/patch-drop-af_user.sql (END - exit=256) [18:54:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:51] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/abusefilter.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/AbuseFilter/db_patches/mysql/patch-drop-afh_user.sql (END - exit=256) [18:54:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:14] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/AbuseFilter/db_patches/mysql/patch-drop-afh_user.sql (START) [18:56:15] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/AbuseFilter/db_patches/mysql/patch-drop-af_user.sql (START) [18:56:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:46] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/translate.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/Translate/sql/mysql/patch-revtag-int-to-bigint-unsigned.sql (START) [18:57:47] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/AbuseFilter/db_patches/mysql/patch-drop-afh_user.sql (START) [18:57:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:48] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/translate.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/Translate/sql/mysql/patch-translate_message_group_subscriptions-composite-primary-key.sql (START) [18:57:49] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/AbuseFilter/db_patches/mysql/patch-drop-af_user.sql (START) [18:57:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:50] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/translate.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/Translate/sql/mysql/patch-translate_reviews-unsigned.sql (START) [18:57:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:51] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/translate.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/Translate/sql/mysql/translate_message_group_subscriptions.sql (START) [18:57:52] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/newsletter.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/Newsletter/sql/mysql/patch-drop-unique-indices.sql (START) [18:57:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:54] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php --wiki=loginwiki --wikidb mhglobal /srv/mediawiki-staging/1.43/extensions/CentralAuth/schema/mysql/patch-rq_type.sql (END - exit=256) [18:58:55] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php --wiki=loginwiki --wikidb mhglobal /srv/mediawiki-staging/1.43/extensions/GlobalBlocking/sql/mysql/patch-global_block_whitelist-default-gbw_address.sql (END - exit=0) [18:58:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:56] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php --wiki=loginwiki --wikidb mhglobal /srv/mediawiki-staging/1.43/extensions/GlobalBlocking/sql/mysql/patch-globalblocks-add-gb_autoblock_parent_id.sql (END - exit=256) [18:58:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:57] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php --wiki=loginwiki --wikidb mhglobal /srv/mediawiki-staging/1.43/extensions/GlobalBlocking/sql/mysql/patch-globalblocks-add-gb_create_account.sql (END - exit=256) [18:58:58] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php --wiki=loginwiki --wikidb mhglobal /srv/mediawiki-staging/1.43/extensions/GlobalBlocking/sql/mysql/patch-globalblocks-add-gb_enable_autoblock.sql (END - exit=256) [18:58:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:59:54] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php --wiki=loginwiki --wikidb mhglobal /srv/mediawiki-staging/1.43/extensions/GlobalBlocking/sql/mysql/patch-globalblocks-drop-gb_by.sql (END - exit=0) [18:59:55] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php --wiki=loginwiki --wikidb mhglobal /srv/mediawiki-staging/1.43/extensions/GlobalBlocking/sql/mysql/patch-globalblocks-modify-gb_address-index.sql (END - exit=256) [19:00:03] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php --wiki=loginwiki --wikidb mhglobal /srv/mediawiki-staging/1.43/extensions/GlobalBlocking/sql/mysql/patch-globalblocks-modify-gb_autoblock_parent_id-default.sql (END - exit=0) [19:00:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:03:11] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/newsletter.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/Newsletter/sql/mysql/patch-drop-unique-indices.sql (END - exit=0) [19:03:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:03:42] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:04:23] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/translate.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/Translate/sql/mysql/patch-translate_message_group_subscriptions-composite-primary-key.sql (END - exit=256) [19:04:24] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/translate.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/Translate/sql/mysql/translate_message_group_subscriptions.sql (END - exit=0) [19:04:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:04:25] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/translate.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/Translate/sql/mysql/patch-revtag-int-to-bigint-unsigned.sql (END - exit=0) [19:04:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:04:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:04:29] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/translate.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/Translate/sql/mysql/patch-translate_reviews-unsigned.sql (END - exit=0) [19:04:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:05:41] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.14 ms [19:10:19] [02mw-config] 07AgentIsai pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/5303623f2a9af52cffdd954da68b9a0d9be09dc7 [19:10:19] 02mw-config/03master 07Agent Isai 035303623 Update sitenotice now that maintenance is complete [19:11:13] miraheze/mw-config - AgentIsai the build passed. [19:12:12] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/titlekey.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/TitleKey/db_patches/abstractSchemaChanges/patch-modify-tk_key.sql (START) [19:12:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:12:19] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/titlekey.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/TitleKey/db_patches/abstractSchemaChanges/patch-modify-tk_key.sql (END - exit=2) [19:12:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:12:25] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/titlekey.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/TitleKey/db_patches/abstractSchemaChanges/patch-modify-tk_key.sql (START) [19:12:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:14:53] [02MirahezeMagic] 07AgentIsai merged 07paladox's pull request #545: Create MigrateSearchIndexSql.php (03master...03paladox-patch-2) 13https://github.com/miraheze/MirahezeMagic/pull/545 [19:14:53] [02MirahezeMagic] 07AgentIsai pushed 1 new commit to 03master 13https://github.com/miraheze/MirahezeMagic/commit/df58cab84cfc66cadb2f56cd45bbb57f81b0049e [19:14:53] 02MirahezeMagic/03master 07paladox 03df58cab Create MigrateSearchIndexSql.php (#545)… [19:14:53] [02MirahezeMagic] 07AgentIsai 04deleted 03paladox-patch-2 at 039be72ce 13https://github.com/miraheze/MirahezeMagic/commit/9be72ce [19:15:11] !log [agent@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_extensions': 'MirahezeMagic'} to all [19:15:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:16:46] !log [agent@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 95s [19:16:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:17:22] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/MirahezeMagic/maintenance/MigrateSearchIndexSql.php (START) [19:17:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:17:58] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/titlekey.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/TitleKey/db_patches/abstractSchemaChanges/patch-modify-tk_key.sql (END - exit=0) [19:18:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:13] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:18:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:43] PROBLEM - mw183 SSH on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:18:43] !log [agent@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw183.wikitide.net [19:18:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:51] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:19:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:20:12] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 20s [19:20:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:20:39] RECOVERY - mw183 SSH on mw183 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u4 (protocol 2.0) [19:25:43] miraheze/MirahezeMagic - AgentIsai the build passed. [19:28:50] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (1 critical updates). [19:29:36] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 64 packages available for upgrade (1 critical updates). [19:29:37] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php --wiki=bluearchivewiki (END - exit=2) [19:29:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:30:54] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [19:31:03] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/maintenance/archives/patch-pagelinks-drop-pl_title.sql (END - exit=2) [19:31:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:33:29] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 65 packages available for upgrade (1 critical updates). [19:33:38] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [19:34:49] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [19:34:50] PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates). [19:36:31] PROBLEM - mw164 APT on mw164 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [19:36:34] PROBLEM - mw154 APT on mw154 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [19:36:47] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [19:38:58] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/updateSpecialPages.php (START) [19:39:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:39:37] PROBLEM - mw184 APT on mw184 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [19:40:00] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [19:40:31] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/semanticmediawiki.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/SemanticMediaWiki/maintenance/setupStore.php (START) [19:40:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:40] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [19:40:41] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [19:43:56] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [19:44:09] PROBLEM - mw174 APT on mw174 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [19:45:11] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/semanticmediawiki.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/SemanticMediaWiki/maintenance/setupStore.php (END - exit=0) [19:45:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:46:32] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (1 critical updates). [19:46:46] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [19:47:58] RECOVERY - mw173 APT on mw173 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:48:02] RECOVERY - mw161 APT on mw161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:48:08] RECOVERY - mw174 APT on mw174 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:48:24] RECOVERY - mw164 APT on mw164 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:48:33] RECOVERY - mw154 APT on mw154 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:48:39] RECOVERY - mw162 APT on mw162 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:48:43] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:48:45] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:48:45] RECOVERY - mw171 APT on mw171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:48:46] RECOVERY - mw151 APT on mw151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:48:51] RECOVERY - mw182 APT on mw182 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:49:09] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (1 critical updates). [19:49:37] RECOVERY - mw163 APT on mw163 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:49:40] RECOVERY - mw184 APT on mw184 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:50:10] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/cuci_temp_edit.sql (START) [19:50:11] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/cuci_wiki_map.sql (START) [19:50:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:12] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/cuci_user.sql (START) [19:50:13] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/patch-cu_changes-drop-cuc_private.sql (START) [19:50:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:14] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-revision-cleanup.sql (START) [19:50:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:15] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/patch-cu_changes-drop-cuc_actiontext.sql (START) [19:50:16] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-pagelinks-drop-pl_title.sql (START) [19:50:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:17] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-page-page_links_updated-noinfinite.sql (START) [19:50:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:18] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-change_tag-ct_rc_id.sql (START) [19:50:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:19] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/patch-cu_changes-drop-cuc_only_for_read_old.sql (START) [19:50:20] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CheckUser/maintenance/deleteReadOldRowsInCuChanges.php (START) [19:50:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:21] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-recentchanges-rc_id-bigint.sql (START) [19:50:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:22] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/fixAutoblockLogTitles.php (START) [19:50:23] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-searchindex-pk-titlelength.sql (START) [19:50:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:24] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CheckUser/maintenance/populateCentralCheckUserIndexTables.php (START) [19:50:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:21] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-page-page_links_updated-noinfinite.sql (END - exit=256) [19:53:22] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-revision-cleanup.sql (END - exit=256) [19:53:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:23] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-change_tag-ct_rc_id.sql (END - exit=256) [19:53:24] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/patch-cu_changes-drop-cuc_private.sql (END - exit=256) [19:53:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:25] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-recentchanges-rc_id-bigint.sql (END - exit=256) [19:53:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:26] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/patch-cu_changes-drop-cuc_actiontext.sql (END - exit=256) [19:53:27] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/cuci_temp_edit.sql (END - exit=256) [19:53:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:28] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-searchindex-pk-titlelength.sql (END - exit=256) [19:53:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:29] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-pagelinks-drop-pl_title.sql (END - exit=256) [19:53:30] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/fixAutoblockLogTitles.php (END - exit=0) [19:53:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:31] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/patch-cu_changes-drop-cuc_only_for_read_old.sql (END - exit=256) [19:53:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:32] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/cuci_user.sql (END - exit=256) [19:53:33] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CheckUser/maintenance/deleteReadOldRowsInCuChanges.php (END - exit=0) [19:53:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:54] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 21.62, 18.93, 14.73 [19:54:02] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CheckUser/maintenance/populateCentralCheckUserIndexTables.php (END - exit=0) [19:54:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:53] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 13.28, 16.70, 14.43 [19:56:15] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 70 packages available for upgrade (1 critical updates). [19:56:17] PROBLEM - bots171 APT on bots171 is CRITICAL: APT CRITICAL: 71 packages available for upgrade (1 critical updates). [19:57:34] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 65 packages available for upgrade (1 critical updates). [20:04:44] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-recentchanges-rc_id-bigint.sql (END - exit=65280) [20:04:45] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-change_tag-ct_rc_id.sql (END - exit=65280) [20:04:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:39] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-revision-cleanup.sql (END - exit=65280) [20:05:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:07:49] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.39, 18.13, 15.43 [20:13:47] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 24.91, 22.27, 18.12 [20:15:05] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki-staging/1.43/maintenance/archives/patch-searchindex-pk-titlelength.sql (END - exit=65280) [20:15:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:17:46] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 15.28, 20.31, 18.41 [20:51:27] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 15.06, 13.06, 6.18 [20:53:27] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 3.29, 9.18, 5.59 [21:12:00] !log [agent@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/fixAutoblockLogTitles.php (END - exit=0) [21:12:01] !log [agent@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CheckUser/maintenance/deleteReadOldRowsInCuChanges.php (END - exit=0) [21:12:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:24:18] PROBLEM - mw184 HTTPS on mw184 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:24:29] PROBLEM - mw174 HTTPS on mw174 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:24:31] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [21:24:35] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [21:24:43] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki [21:24:44] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [21:24:45] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [21:24:46] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [21:24:47] PROBLEM - mw164 MediaWiki Rendering on mw164 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:24:50] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [21:24:51] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:24:54] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:24:57] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:25:08] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/0273c72996208faf53d83b99448e237ad7b0a9f6 [21:25:09] 02puppet/03master 07paladox 030273c72 mattermost: upgrade to 10.4.2 [21:25:09] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:25:09] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:25:09] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [21:25:10] PROBLEM - mw184 MediaWiki Rendering on mw184 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [21:25:11] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:25:15] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:25:16] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:25:19] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:25:19] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:25:20] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [21:25:21] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [21:25:22] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:25:33] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:25:39] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki [21:25:41] PROBLEM - mw174 MediaWiki Rendering on mw174 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:25:41] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:25:49] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:25:50] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [21:25:56] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:26:01] PROBLEM - mw154 MediaWiki Rendering on mw154 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:26:02] PROBLEM - mw154 HTTPS on mw154 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [21:26:04] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [21:26:05] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [21:26:06] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 436.87, 200.97, 80.72 [21:26:07] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:26:12] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:26:25] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:26:30] PROBLEM - mw164 HTTPS on mw164 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [21:27:20] PROBLEM - db171 Puppet on db171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:27:27] PROBLEM - db171 APT on db171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:28:42] PROBLEM - db171 MariaDB Connections on db171 is UNKNOWN: [21:29:08] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 9.730 second response time [21:29:16] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.058 second response time [21:29:21] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.053 second response time [21:29:23] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.055 second response time [21:29:25] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.059 second response time [21:29:46] PROBLEM - db171 MariaDB on db171 is UNKNOWN: [21:29:46] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.051 second response time [21:30:09] RECOVERY - mw154 HTTPS on mw154 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.709 second response time [21:31:04] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 2.852 second response time [21:31:17] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4108 bytes in 7.867 second response time [21:32:04] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.065 second response time [21:32:17] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.823 second response time [21:32:21] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 9.329 second response time [21:32:34] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 1.188 second response time [21:32:48] RECOVERY - mw164 HTTPS on mw164 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 3.610 second response time [21:33:15] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 4.031 second response time [21:34:45] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4107 bytes in 0.171 second response time [21:34:54] RECOVERY - mw184 HTTPS on mw184 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 2.127 second response time [21:34:56] RECOVERY - mw174 HTTPS on mw174 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 4.994 second response time [21:35:39] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [21:36:43] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [21:41:14] PROBLEM - db171 MariaDB on db171 is CRITICAL: Received error packet before completion of TLS handshake. The authenticity of the following error cannot be verified: 1040 - Too many connections [21:46:09] PROBLEM - db171 MariaDB on db171 is UNKNOWN: [21:48:06] PROBLEM - db171 MariaDB on db171 is CRITICAL: Can't connect to server on 'db171.wikitide.net' (115) [21:52:22] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/CheckUser/schema/mysql/patch-cu_changes-drop-cuc_actiontext.sql (END - exit=65280) [21:52:23] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/CheckUser/schema/mysql/cuci_temp_edit.sql (END - exit=65280) [21:52:24] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/CheckUser/schema/mysql/patch-cu_changes-drop-cuc_only_for_read_old.sql (END - exit=65280) [21:52:25] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/CheckUser/schema/mysql/cuci_user.sql (END - exit=65280) [21:52:26] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/CheckUser/schema/mysql/cuci_wiki_map.sql (END - exit=65280) [21:52:27] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/CheckUser/schema/mysql/patch-cu_changes-drop-cuc_private.sql (END - exit=65280) [21:53:57] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/maintenance/archives/patch-page-page_links_updated-noinfinite.sql (END - exit=65280) [21:57:27] oh man [22:24:19] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.89, 0.24, 0.08 [22:25:01] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [22:25:08] RECOVERY - db171 MariaDB on db171 is OK: Uptime: 84 Threads: 15 Questions: 69215 Slow queries: 0 Opens: 1447 Open tables: 1275 Queries per second avg: 823.988 [22:25:42] RECOVERY - db171 MariaDB Connections on db171 is OK: OK connection usage: 1.9%Current connections: 19 [22:25:43] RECOVERY - db171 APT on db171 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [22:27:51] RECOVERY - mw154 MediaWiki Rendering on mw154 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.232 second response time [22:28:02] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.186 second response time [22:28:12] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.162 second response time [22:28:16] RECOVERY - mw174 MediaWiki Rendering on mw174 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.166 second response time [22:28:24] RECOVERY - mw164 MediaWiki Rendering on mw164 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.162 second response time [22:28:30] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.178 second response time [22:28:40] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.186 second response time [22:28:44] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.211 second response time [22:28:45] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.170 second response time [22:28:45] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.184 second response time [22:28:46] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.175 second response time [22:28:50] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.191 second response time [22:28:51] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.316 second response time [22:28:58] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.156 second response time [22:29:03] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.171 second response time [22:29:07] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.232 second response time [22:29:10] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.180 second response time [22:29:10] RECOVERY - mw184 MediaWiki Rendering on mw184 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.178 second response time [22:29:12] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.183 second response time [22:29:16] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.217 second response time [22:32:58] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki-staging/1.43/extensions/CheckUser/schema/mysql/cuci_wiki_map.sql (END - exit=256) [22:33:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:45:20] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/AbuseFilter/db_patches/mysql/patch-drop-af_user.sql (END - exit=256) [22:45:21] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/AbuseFilter/db_patches/mysql/patch-drop-af_user.sql (END - exit=0) [22:45:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:45:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:59:34] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/AbuseFilter/db_patches/mysql/patch-drop-afh_user.sql (END - exit=256) [22:59:35] !log [agent@mwtask151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/sql.php /srv/mediawiki/1.43/extensions/AbuseFilter/db_patches/mysql/patch-drop-afh_user.sql (END - exit=0) [22:59:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:59:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:16:34] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/489b5461ffd4ed39584948e43420565754a7e357 [23:16:34] 02ssl/03master 07MacFan4000 03489b546 add wiki.eggnos.studio [23:18:43] !log [agent@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CheckUser/maintenance/populateCentralCheckUserIndexTables.php (END - exit=0) [23:18:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:17] [02mw-config] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/b2c3b244d8431841c83147465eede737ae19ba89 [23:22:17] 02mw-config/03master 07MacFan4000 03b2c3b24 use updated footer button [23:22:41] !log [macfan@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [23:22:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:23:01] !log [macfan@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 20s [23:23:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:23:17] miraheze/mw-config - MacFan4000 the build passed.