[00:00:54] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 0% [00:02:21] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 5191MiB (5% inode=98%); [00:02:30] PROBLEM - cp36 Disk Space on cp36 is CRITICAL: DISK CRITICAL - free space: / 5100MiB (5% inode=98%); [00:04:18] PROBLEM - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query leyhp.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:06:21] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21329MiB (24% inode=98%); [00:06:30] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21299MiB (24% inode=98%); [00:06:54] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [00:08:54] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 50% [00:10:13] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 45% [00:33:24] RECOVERY - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is OK: SSL OK - wiki.tmyt105.leyhp.com reverse DNS resolves to cp37.wikitide.net - CNAME OK [00:52:14] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [01:00:54] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [01:02:54] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 58% [01:11:07] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c1311dc77be4...3787e4457d4f [01:11:09] [02miraheze/puppet] 07Universal-Omega 033787e44 - cvtbot: use #miraheze-cvt-feed for broadcastchannel also [01:30:54] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 39% [01:31:27] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [01:40:34] PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: PROCS CRITICAL: 1 process with args 'cvtbot' [01:43:41] [02.github] 07lens0021 commented on pull request 03#32: change to re-usable test - 13https://github.com/miraheze/.github/pull/32#issuecomment-1926072398 [01:56:15] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 47% [01:59:27] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [02:03:48] [02miraheze/RequestSSL] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0 [+0/-0/±1] 13https://github.com/miraheze/RequestSSL/commit/df44123f5fc2 [02:03:50] [02miraheze/RequestSSL] 07dependabot[bot] 03df44123 - Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 [02:03:51] [02RequestSSL] 07dependabot[bot] labeled pull request 03#31: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/RequestSSL/pull/31 [02:03:53] [02RequestSSL] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0 - 13https://github.com/miraheze/RequestSSL [02:03:56] [02RequestSSL] 07dependabot[bot] labeled pull request 03#31: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/RequestSSL/pull/31 [02:04:34] RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot' [02:08:34] PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: PROCS CRITICAL: 3 processes with args 'cvtbot' [02:10:34] RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot' [02:11:00] [02RequestSSL] 07dependabot[bot] opened pull request 03#31: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/RequestSSL/pull/31 [02:18:34] PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: PROCS CRITICAL: 3 processes with args 'cvtbot' [02:50:34] RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot' [02:54:34] PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: PROCS CRITICAL: 3 processes with args 'cvtbot' [03:00:26] RECOVERY - db181 Backups SQL on db181 is OK: FILE_AGE OK: /var/log/sql-backup.log is 24 seconds old and 0 bytes [03:00:40] RECOVERY - db161 Backups SQL on db161 is OK: FILE_AGE OK: /var/log/sql-backup.log is 39 seconds old and 0 bytes [03:01:57] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/086b76c17334...4eda30b58682 [03:01:58] [02miraheze/mw-config] 07AgentIsai 034eda30b - Add c1 back to active clusters [03:02:08] PROBLEM - db151 Backups SQL on db151 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864127 seconds old and 92 bytes [03:03:10] miraheze/mw-config - AgentIsai the build passed. [03:06:44] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4eda30b58682...dc27175376d1 [03:06:45] [02miraheze/mw-config] 07AgentIsai 03dc27175 - Shuffle around dbclusters [03:07:05] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [03:07:06] !log [agent@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [03:07:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:07:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:07:14] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [03:07:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:07:23] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 9s [03:07:59] miraheze/mw-config - AgentIsai the build passed. [03:09:15] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 788 bytes in 0.217 second response time [03:09:47] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 788 bytes in 0.130 second response time [03:09:54] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 788 bytes in 0.133 second response time [03:10:04] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 788 bytes in 0.129 second response time [03:10:07] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 788 bytes in 0.138 second response time [03:10:22] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 788 bytes in 0.106 second response time [03:10:49] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 788 bytes in 0.116 second response time [03:11:02] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 788 bytes in 0.152 second response time [03:11:13] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 788 bytes in 0.143 second response time [03:13:25] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [03:23:03] !log [@test151] starting deploy of {'config': True} to test151 [03:23:04] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [03:26:05] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 8191 bytes in 0.172 second response time [03:27:22] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [03:41:04] Errors on db171? [03:45:44] Yes, we are aware and actively working to fix it [03:46:26] Thanks~ [03:55:22] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:56:50] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=metawiki (END - exit=65280) [03:58:52] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=metawiki --no-log11 (END - exit=65280) [04:05:15] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.234 second response time [04:05:15] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (START) [04:06:02] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (END - exit=2) [04:07:48] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.221 second response time [04:07:54] PROBLEM - mw151 MediaWiki Rendering on mw151 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 6693 bytes in 0.053 second response time [04:08:04] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.253 second response time [04:08:07] PROBLEM - mw161 MediaWiki Rendering on mw161 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 6693 bytes in 0.053 second response time [04:08:22] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.230 second response time [04:08:49] PROBLEM - mw181 MediaWiki Rendering on mw181 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 6693 bytes in 0.026 second response time [04:09:03] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.255 second response time [04:09:13] PROBLEM - mw162 MediaWiki Rendering on mw162 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 6693 bytes in 0.050 second response time [04:09:25] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 45% [04:10:19] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (START) [04:11:53] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (END - exit=2) [04:12:13] !log [agent@mwtask181] starting deploy of {'files': '../mediawiki/cache/active.json'} to all [04:12:14] !log [agent@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw151.wikitide.net [04:12:18] !log [agent@mwtask181] starting deploy of {'force': True, 'files': '../mediawiki/cache/active.json'} to all [04:12:24] !log [agent@mwtask181] finished deploy of {'force': True, 'files': '../mediawiki/cache/active.json'} to all - SUCCESS in 6s [04:12:54] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 41% [04:13:06] !log [agent@mwtask181] starting deploy of {'force': True, 'files': '../mediawiki/cache/metawiki.json'} to all [04:13:12] !log [agent@mwtask181] finished deploy of {'force': True, 'files': '../mediawiki/cache/metawiki.json'} to all - SUCCESS in 6s [04:13:19] !log [agent@mwtask181] starting deploy of {'force': True, 'files': '../mediawiki/cache/config-metawiki.json'} to all [04:13:25] !log [agent@mwtask181] finished deploy of {'force': True, 'files': '../mediawiki/cache/config-metawiki.json'} to all - SUCCESS in 6s [04:15:14] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=metawiki (END - exit=0) [04:15:16] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=metawiki (END - exit=0) [04:15:35] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (START) [04:25:57] !log [agent@mwtask181] starting deploy of {'force': True, 'folders': '../mediawiki/cache/'} to all [04:27:39] !log [agent@mwtask181] finished deploy of {'force': True, 'folders': '../mediawiki/cache/'} to all - SUCCESS in 101s [04:27:53] !log [agent@mwtask181] starting deploy of {'force': True, 'folders': '../mediawiki/cache/'} to [mw162, mw181] [04:27:54] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.243 second response time [04:28:07] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.246 second response time [04:28:37] !log [agent@mwtask181] finished deploy of {'force': True, 'folders': '../mediawiki/cache/'} to [mw162, mw181] - SUCCESS in 43s [04:28:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:28:49] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.199 second response time [04:29:14] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.273 second response time [04:29:25] RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 11% [04:38:34] RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot' [04:50:34] PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: PROCS CRITICAL: 3 processes with args 'cvtbot' [05:02:54] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 39% [05:06:32] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (END - exit=0) [05:06:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:10:54] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 40% [05:12:34] RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot' [05:16:34] PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: PROCS CRITICAL: 3 processes with args 'cvtbot' [05:20:20] [02miraheze/RemovePII] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/commit/ce613fb9e872 [05:20:23] [02miraheze/RemovePII] 07dependabot[bot] 03ce613fb - Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 [05:20:24] [02RemovePII] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0 - 13https://github.com/miraheze/RemovePII [05:20:26] [02RemovePII] 07dependabot[bot] opened pull request 03#79: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/RemovePII/pull/79 [05:20:28] [02RemovePII] 07dependabot[bot] labeled pull request 03#79: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/RemovePII/pull/79 [05:20:29] [02RemovePII] 07dependabot[bot] labeled pull request 03#79: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/RemovePII/pull/79 [05:21:54] PROBLEM - es.countryhumans.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'es.countryhumans.polandball.wiki' expires in 15 day(s) (Wed 21 Feb 2024 05:16:33 AM GMT +0000). [05:22:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7de81e6316a6...e7aca8fb191f [05:22:13] [02miraheze/ssl] 07MirahezeSSLBot 03e7aca8f - Bot: Update SSL cert for es.countryhumans.polandball.wiki [05:27:01] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=1d6chanwiki --no-updates --username-prefix=1d6chan 1d4chan.org-20230927-history.xml (START) [05:27:02] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=1d6chanwiki --no-updates --username-prefix=1d6chan 1d4chan.org-20230927-history.xml (END - exit=256) [05:27:03] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=1d6chanwiki (START) [05:27:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:27:04] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=1d6chanwiki (END - exit=0) [05:27:05] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=1d6chanwiki --active --update (END - exit=0) [05:27:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:27:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:27:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:27:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:27:26] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=1d6chanwiki --no-updates --username-prefix=1d6chan 1d4chan.org-20230927-history.xml (START) [05:27:27] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=1d6chanwiki --no-updates --username-prefix=1d6chan 1d4chan.org-20230927-history.xml (END - exit=256) [05:27:28] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=1d6chanwiki (START) [05:27:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:27:29] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=1d6chanwiki (END - exit=0) [05:27:30] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=1d6chanwiki --active --update (END - exit=2) [05:27:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:27:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:27:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:27:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:27:43] miraheze/RemovePII - dependabot[bot] the build passed. [05:28:10] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=1d6chanwiki --no-updates --username-prefix=1d6chan 1d4chan.org-20230927-history.xml (START) [05:28:11] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=1d6chanwiki --no-updates --username-prefix=1d6chan 1d4chan.org-20230927-history.xml (END - exit=256) [05:28:12] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=1d6chanwiki (START) [05:28:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:28:13] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=1d6chanwiki (END - exit=0) [05:28:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:28:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:28:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:29:01] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=1d6chanwiki --no-updates --username-prefix=1d6chan /home/agent/1d6chan_xml/1d4chan.org-20230927-history.xml (START) [05:29:02] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=1d6chanwiki --no-updates --username-prefix=1d6chan /home/agent/1d6chan_xml/1d4chan.org-20230927-history.xml (END - exit=256) [05:29:03] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=1d6chanwiki (START) [05:29:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:29:04] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=1d6chanwiki (END - exit=0) [05:29:05] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=1d6chanwiki --active --update (END - exit=0) [05:29:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:29:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:29:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:29:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:30:26] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=1d6chanwiki --no-updates --username-prefix=1d6chan /home/agent/1d6chan_xml/1d4chan.org-20230927-history.xml (START) [05:30:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:13] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=lhmnwiki --no-updates /home/agent/ImportDump/lhmnwiki-20240205021828.xml (START) [05:33:14] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=lhmnwiki --no-updates /home/agent/ImportDump/lhmnwiki-20240205021828.xml (END - exit=256) [05:33:15] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=lhmnwiki (START) [05:33:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:16] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=lhmnwiki (END - exit=0) [05:33:17] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=lhmnwiki --active --update (END - exit=0) [05:33:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:28] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=8086pcwiki --no-updates --username-prefix=dev /home/agent/ImportDump/8086pcwiki-20240204114316.xml (START) [05:33:29] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=8086pcwiki --no-updates --username-prefix=dev /home/agent/ImportDump/8086pcwiki-20240204114316.xml (END - exit=256) [05:33:30] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=8086pcwiki (START) [05:33:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:35] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=8086pcwiki (END - exit=0) [05:33:36] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=8086pcwiki --active --update (END - exit=0) [05:33:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:40] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=mothergooselibrarywiki --no-updates /home/agent/ImportDump/mothergooselibrarywiki-20240205020256.xml (START) [05:33:41] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=mothergooselibrarywiki --no-updates /home/agent/ImportDump/mothergooselibrarywiki-20240205020256.xml (END - exit=256) [05:33:42] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=mothergooselibrarywiki (START) [05:33:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:43] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=mothergooselibrarywiki (END - exit=0) [05:33:44] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=mothergooselibrarywiki --active --update (END - exit=0) [05:33:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:34:23] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=lhmnwiki --no-updates /home/agent/ImportDump/lhmnwiki-20240205021828.xml (START) [05:34:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:34:32] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=8086pcwiki --no-updates --username-prefix=dev /home/agent/ImportDump/8086pcwiki-20240204114316.xml (START) [05:34:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:34:38] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=mothergooselibrarywiki --no-updates /home/agent/ImportDump/mothergooselibrarywiki-20240205020256.xml (START) [05:34:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:35:00] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=mothergooselibrarywiki --no-updates /home/agent/ImportDump/mothergooselibrarywiki-20240205020256.xml (END - exit=0) [05:35:01] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=mothergooselibrarywiki (START) [05:35:03] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=8086pcwiki --no-updates --username-prefix=dev /home/agent/ImportDump/8086pcwiki-20240204114316.xml (END - exit=0) [05:35:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:35:04] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=8086pcwiki (START) [05:35:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:35:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:35:08] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=mothergooselibrarywiki (END - exit=0) [05:35:09] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=mothergooselibrarywiki --active --update (END - exit=0) [05:35:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:35:11] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=8086pcwiki (END - exit=0) [05:35:12] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=8086pcwiki --active --update (END - exit=0) [05:35:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:35:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:35:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:35:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:38:12] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=lhmnwiki --no-updates /home/agent/ImportDump/lhmnwiki-20240205021828.xml (END - exit=256) [05:38:13] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=lhmnwiki (START) [05:38:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:38:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:39:04] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=lhmnwiki (END - exit=0) [05:39:05] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=lhmnwiki --active --update (END - exit=0) [05:39:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:39:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:40:54] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 39% [05:44:34] RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot' [05:48:34] PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: PROCS CRITICAL: 3 processes with args 'cvtbot' [05:50:47] RECOVERY - es.countryhumans.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'es.countryhumans.polandball.wiki' will expire on Sun 05 May 2024 04:22:05 AM GMT +0000. [06:04:34] RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot' [06:06:38] PROBLEM - ru.countryhumans.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ru.countryhumans.polandball.wiki' expires in 15 day(s) (Wed 21 Feb 2024 05:42:49 AM GMT +0000). [06:06:49] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e7aca8fb191f...21f5cc11384c [06:06:52] [02miraheze/ssl] 07MirahezeSSLBot 0321f5cc1 - Bot: Update SSL cert for ru.countryhumans.polandball.wiki [06:07:37] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=mothergooselibrarywiki --no-updates --username-prefix=mothergoose /home/reception/ImportDump/mothergooselibrarywiki-20240205020256.xml (START) [06:07:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:07:52] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=lhmnwiki --no-updates --username-prefix=lhmn /home/reception/ImportDump/lhmnwiki-20240205021828.xml (START) [06:07:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:07:59] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=mothergooselibrarywiki --no-updates --username-prefix=mothergoose /home/reception/ImportDump/mothergooselibrarywiki-20240205020256.xml (END - exit=0) [06:08:00] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=mothergooselibrarywiki (START) [06:08:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:08:03] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=mothergooselibrarywiki (END - exit=0) [06:08:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:08:04] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=mothergooselibrarywiki --active --update (END - exit=0) [06:08:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:08:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:08:34] PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: PROCS CRITICAL: 3 processes with args 'cvtbot' [06:10:59] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=lhmnwiki --no-updates --username-prefix=lhmn /home/reception/ImportDump/lhmnwiki-20240205021828.xml (END - exit=256) [06:11:00] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=lhmnwiki (START) [06:11:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:11:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:11:27] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=lhmnwiki (END - exit=0) [06:11:28] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=lhmnwiki --active --update (END - exit=0) [06:11:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:11:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:15:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/21f5cc11384c...90103fefe023 [06:15:44] [02miraheze/ssl] 07MirahezeSSLBot 0390103fe - Bot: Add SSL cert for atlas.starworld.zone [06:33:08] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/730b9200ef77...1bc5cc0016a3 [06:33:09] [02miraheze/MirahezeMagic] 07Universal-Omega 031bc5cc0 - Enable removeMemcachedKey [06:35:16] RECOVERY - ru.countryhumans.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ru.countryhumans.polandball.wiki' will expire on Sun 05 May 2024 05:06:43 AM GMT +0000. [06:39:51] miraheze/MirahezeMagic - Universal-Omega the build passed. [06:42:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03$wgMirahezeMagicMemcachedServers [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/ca836e440902 [06:42:44] [02miraheze/mw-config] 07Universal-Omega 03ca836e4 - Set $wgMirahezeMagicMemcachedServers [06:42:46] [02mw-config] 07Universal-Omega created branch 03$wgMirahezeMagicMemcachedServers - 13https://github.com/miraheze/mw-config [06:42:47] [02mw-config] 07Universal-Omega opened pull request 03#5460: Set $wgMirahezeMagicMemcachedServers - 13https://github.com/miraheze/mw-config/pull/5460 [06:43:51] miraheze/mw-config - Universal-Omega the build passed. [06:53:52] !log [@test151] starting deploy of {'folders': '1.40/extensions/MirahezeMagic'} to test151 [06:53:53] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [06:53:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:53:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:54:00] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [06:54:01] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [06:54:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:54:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:54:10] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [06:54:11] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [06:54:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:54:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:57:01] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[MediaWiki-REL1_40 MirahezeMagic Sync],Exec[MediaWiki-REL1_41 MirahezeMagic Sync],Exec[MediaWiki-master MirahezeMagic Sync] [06:57:58] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php --wiki=redlandwiki (END - exit=2) [06:58:00] !log [reception@mwtask181] starting deploy of {'l10n': True, 'versions': ['1.40', '1.41']} to all [06:58:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:58:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:27] !log [@mwtask181] starting deploy of {'folders': '1.40/extensions/MirahezeMagic'} to all [07:02:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:34] !log [@mwtask181] finished deploy of {'folders': '1.40/extensions/MirahezeMagic'} to all - SUCCESS in 7s [07:02:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:46] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [07:02:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:54] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 7s [07:02:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:36] !log [reception@mwtask181] finished deploy of {'l10n': True, 'versions': ['1.40', '1.41']} to all - SUCCESS in 576s [07:07:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:15] !log [reception@mwtask181] starting deploy of {'versions': '1.40', 'upgrade_extensions': 'MirahezeMagic'} to all [07:08:16] !log [reception@mwtask181] finished deploy of {'versions': '1.40', 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 1s [07:08:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:40] [02mw-config] 07Reception123 closed pull request 03#5460: Set $wgMirahezeMagicMemcachedServers - 13https://github.com/miraheze/mw-config/pull/5460 [07:08:41] [02miraheze/mw-config] 07Reception123 deleted branch 03$wgMirahezeMagicMemcachedServers [07:08:44] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/dc27175376d1...0579fb0f4a48 [07:08:46] [02miraheze/mw-config] 07Universal-Omega 030579fb0 - Set $wgMirahezeMagicMemcachedServers (#5460) [07:08:47] [02mw-config] 07Reception123 deleted branch 03$wgMirahezeMagicMemcachedServers - 13https://github.com/miraheze/mw-config [07:09:38] miraheze/mw-config - Reception123 the build passed. [07:22:55] !log [@test151] starting deploy of {'config': True} to test151 [07:22:56] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [07:22:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:23:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:32:42] !log [@mwtask181] starting deploy of {'config': True} to all [07:32:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:32:50] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 7s [07:32:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:46:45] RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot' [07:52:09] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=1d6chanwiki --no-updates --username-prefix=1d6chan /home/agent/1d6chan_xml/1d4chan.org-20230927-history.xml (END - exit=35072) [07:52:10] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=1d6chanwiki (START) [07:52:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:52:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:56:01] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:57:36] PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: PROCS CRITICAL: 3 processes with args 'cvtbot' [08:01:30] RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot' [08:05:22] PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: PROCS CRITICAL: 3 processes with args 'cvtbot' [08:12:22] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [08:14:10] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 39% [08:25:55] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=1d6chanwiki (END - exit=2) [08:25:56] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=1d6chanwiki --active --update (END - exit=0) [08:25:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:25:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:38:45] RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot' [08:42:45] PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: PROCS CRITICAL: 3 processes with args 'cvtbot' [08:43:36] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=1d6chanwiki (START) [08:43:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:48:45] RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot' [08:52:45] PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: PROCS CRITICAL: 3 processes with args 'cvtbot' [08:58:45] RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot' [09:02:45] PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: PROCS CRITICAL: 3 processes with args 'cvtbot' [09:05:48] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [09:12:06] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [09:14:32] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=1d6chanwiki (END - exit=0) [09:14:33] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=1d6chanwiki --active --update (END - exit=0) [09:14:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:14:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:35:41] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [09:55:31] PROBLEM - arrieretoile.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for arrieretoile.fr could not be found [10:13:45] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [12:13:49] [02miraheze/ErrorPages] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ErrorPages/compare/b52e74e01278...b5710b2ce36e [12:13:50] [02miraheze/ErrorPages] 07translatewiki 03b5710b2 - Localisation updates from https://translatewiki.net. [12:13:53] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ImportDump/compare/2c420419ffcf...609bb01a4d3c [12:13:54] [02miraheze/ImportDump] 07translatewiki 03609bb01 - Localisation updates from https://translatewiki.net. [12:13:57] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/CreateWiki/compare/2a5dbeb7c403...4e9a22658f89 [12:13:58] [02miraheze/CreateWiki] 07translatewiki 034e9a226 - Localisation updates from https://translatewiki.net. [12:13:59] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/ManageWiki/compare/62ca5d306e31...6ac1b49cafd8 [12:14:01] [02miraheze/ManageWiki] 07translatewiki 036ac1b49 - Localisation updates from https://translatewiki.net. [12:14:47] miraheze/ErrorPages - translatewiki the build passed. [12:15:08] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [12:18:13] miraheze/ManageWiki - translatewiki the build passed. [12:19:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/90103fefe023...1485437e435a [12:19:37] [02miraheze/ssl] 07MirahezeSSLBot 031485437 - Bot: Add SSL cert for tdr.wiki [12:20:38] miraheze/ImportDump - translatewiki the build passed. [12:21:47] miraheze/CreateWiki - translatewiki the build passed. [12:26:01] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ErrorPages Sync] [12:36:51] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [12:45:00] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [12:55:43] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:02:33] RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot' [13:05:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-14 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/66db483c9418 [13:05:24] [02miraheze/puppet] 07paladox 0366db483 - cvtbot: fix WorkingDirectory and Environment [13:05:26] [02puppet] 07paladox created branch 03paladox-patch-14 - 13https://github.com/miraheze/puppet [13:05:27] [02puppet] 07paladox opened pull request 03#3739: cvtbot: fix WorkingDirectory and Environment - 13https://github.com/miraheze/puppet/pull/3739 [13:06:26] [02puppet] 07paladox closed pull request 03#3739: cvtbot: fix WorkingDirectory and Environment - 13https://github.com/miraheze/puppet/pull/3739 [13:06:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3787e4457d4f...1b9ca1896d2b [13:06:29] [02miraheze/puppet] 07paladox 031b9ca18 - cvtbot: fix WorkingDirectory and Environment (#3739) [13:06:29] PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: PROCS CRITICAL: 1 process with args 'cvtbot' [13:06:32] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-14 [13:06:33] [02puppet] 07paladox deleted branch 03paladox-patch-14 - 13https://github.com/miraheze/puppet [13:32:56] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [13:38:11] [02miraheze/ImportDump] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/a31f00d8c5fe [13:38:12] [02miraheze/ImportDump] 07dependabot[bot] 03a31f00d - Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 [13:38:13] [02ImportDump] 07dependabot[bot] opened pull request 03#63: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/ImportDump/pull/63 [13:38:16] [02ImportDump] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0 - 13https://github.com/miraheze/ImportDump [13:38:19] [02ImportDump] 07dependabot[bot] labeled pull request 03#63: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/ImportDump/pull/63 [13:38:21] [02ImportDump] 07dependabot[bot] labeled pull request 03#63: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/ImportDump/pull/63 [13:39:04] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 40% [13:41:15] miraheze/ImportDump - dependabot[bot] the build has errored. [13:46:35] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [13:52:56] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:05:12] !log [paladox@mwtask181] test [14:05:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:06:40] !log [paladox@mwtask181] starting deploy of {'world': True, 'l10n': True, 'versions': ['1.40', '1.41']} to all [14:06:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:08:48] !log [paladox@mwtask181] finished deploy of {'world': True, 'l10n': True, 'versions': ['1.40', '1.41']} to all - SUCCESS in 128s [14:08:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:23:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-14 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/8a78e582acaf [14:23:51] [02miraheze/puppet] 07paladox 038a78e58 - Fix rebuildVersionCache not supporting 1.41 [14:23:53] [02puppet] 07paladox created branch 03paladox-patch-14 - 13https://github.com/miraheze/puppet [14:23:54] [02puppet] 07paladox opened pull request 03#3740: Fix rebuildVersionCache not supporting 1.41 - 13https://github.com/miraheze/puppet/pull/3740 [14:25:25] miraheze/puppet - paladox the build passed. [14:26:17] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 40% [14:30:35] [02puppet] 07paladox synchronize pull request 03#3740: Fix rebuildVersionCache not supporting 1.41 - 13https://github.com/miraheze/puppet/pull/3740 [14:30:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-14 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8a78e582acaf...7e194ebc9a14 [14:30:39] [02miraheze/puppet] 07paladox 037e194eb - Update mwdeploy.py [14:31:33] !log [paladox@mwtask181] starting deploy of {'world': True, 'l10n': True, 'versions': ['1.40', '1.41']} to all [14:31:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:32:24] miraheze/puppet - paladox the build passed. [14:33:28] !log [paladox@mwtask181] finished deploy of {'world': True, 'l10n': True, 'versions': ['1.40', '1.41']} to all - SUCCESS in 115s [14:33:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:34:16] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [14:46:04] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [15:04:28] [02miraheze/GlobalNewFiles] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0 [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/commit/245785480e8b [15:04:30] [02miraheze/GlobalNewFiles] 07dependabot[bot] 032457854 - Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 [15:04:31] [02GlobalNewFiles] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0 - 13https://github.com/miraheze/GlobalNewFiles [15:04:32] [02GlobalNewFiles] 07dependabot[bot] labeled pull request 03#51: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/GlobalNewFiles/pull/51 [15:04:34] [02GlobalNewFiles] 07dependabot[bot] opened pull request 03#51: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/GlobalNewFiles/pull/51 [15:04:35] [02GlobalNewFiles] 07dependabot[bot] labeled pull request 03#51: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/GlobalNewFiles/pull/51 [15:08:31] miraheze/GlobalNewFiles - dependabot[bot] the build passed. [15:09:48] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0579fb0f4a48...9a7bcf2789fe [15:09:50] [02miraheze/mw-config] 07AgentIsai 039a7bcf2 - Add db162 to Database.php [15:10:17] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 39% [15:10:40] miraheze/mw-config - AgentIsai the build has errored. [15:17:47] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [15:23:03] !log [@test151] starting deploy of {'config': True} to test151 [15:23:04] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [15:23:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:23:05] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9a7bcf2789fe...1c9f3867ffdf [15:23:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:23:07] [02miraheze/mw-config] 07AgentIsai 031c9f386 - Place metawiki and loginwiki on readonly [15:23:58] miraheze/mw-config - AgentIsai the build has errored. [15:24:13] PROBLEM - test151 MediaWiki Rendering on test151 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [15:26:14] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [15:26:20] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [15:27:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-14-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/cb7a5ce4804f [15:27:57] [02miraheze/puppet] 07paladox 03cb7a5ce - Add swift::expirer [15:28:00] [02puppet] 07paladox created branch 03paladox-patch-14-1 - 13https://github.com/miraheze/puppet [15:28:03] [02puppet] 07paladox opened pull request 03#3741: Add swift::expirer - 13https://github.com/miraheze/puppet/pull/3741 [15:28:09] [02puppet] 07paladox edited pull request 03#3741: Add swift:: expirer - 13https://github.com/miraheze/puppet/pull/3741 [15:28:13] [02puppet] 07paladox edited pull request 03#3741: Add swift::expirer - 13https://github.com/miraheze/puppet/pull/3741 [15:28:40] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1c9f3867ffdf...e2ca832e9165 [15:28:41] [02miraheze/mw-config] 07AgentIsai 03e2ca832 - Fix syntax [15:28:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-14-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cb7a5ce4804f...dddf0dd44030 [15:28:46] [02miraheze/puppet] 07paladox 03dddf0dd - Update common.yaml [15:28:47] [02puppet] 07paladox synchronize pull request 03#3741: Add swift::expirer - 13https://github.com/miraheze/puppet/pull/3741 [15:29:33] miraheze/mw-config - AgentIsai the build has errored. [15:30:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-14-1 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/dddf0dd44030...526c270f83a0 [15:30:27] [02miraheze/puppet] 07paladox 03526c270 - Create expirer.pp [15:30:29] [02puppet] 07paladox synchronize pull request 03#3741: Add swift::expirer - 13https://github.com/miraheze/puppet/pull/3741 [15:31:16] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [15:31:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:31:24] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 8s [15:31:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:31:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-14-1 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/526c270f83a0...98438327b3b8 [15:31:29] [02miraheze/puppet] 07paladox 039843832 - Create object-expirer.conf.erb [15:31:32] [02puppet] 07paladox synchronize pull request 03#3741: Add swift::expirer - 13https://github.com/miraheze/puppet/pull/3741 [15:32:24] [02puppet] 07paladox synchronize pull request 03#3741: Add swift::expirer - 13https://github.com/miraheze/puppet/pull/3741 [15:32:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-14-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/98438327b3b8...5f8d56e3b6bb [15:32:28] [02miraheze/puppet] 07paladox 035f8d56e - Update expirer.pp [15:32:57] [02puppet] 07paladox edited pull request 03#3741: Add swift::expirer - 13https://github.com/miraheze/puppet/pull/3741 [15:33:16] [02puppet] 07paladox closed pull request 03#3741: Add swift::expirer - 13https://github.com/miraheze/puppet/pull/3741 [15:33:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±2] 13https://github.com/miraheze/puppet/compare/1b9ca1896d2b...457d57deaf9e [15:33:20] [02miraheze/puppet] 07paladox 03457d57d - Add swift::expirer (#3741) [15:33:23] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-14-1 [15:33:24] [02puppet] 07paladox deleted branch 03paladox-patch-14-1 - 13https://github.com/miraheze/puppet [15:44:27] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 7 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw182 [15:44:56] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 7 backends are down. mw151 mw161 mw162 mw171 mw172 mw181 mw182 [15:46:27] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 18 backends are healthy [15:46:56] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 18 backends are healthy [15:53:21] PROBLEM - db162 Disk Space on db162 is CRITICAL: DISK CRITICAL - free space: / 0MiB (0% inode=97%); [15:53:32] !log [@test151] starting deploy of {'config': True} to test151 [15:53:33] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [15:54:13] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 8191 bytes in 0.263 second response time [15:54:20] RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.074 second response time [15:56:17] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 41% [15:57:04] PROBLEM - wiki.vot.gg - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.vot.gg All nameservers failed to answer the query. [15:57:21] RECOVERY - db162 Disk Space on db162 is OK: DISK OK - free space: / 20264MiB (47% inode=97%); [16:04:17] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 39% [16:17:15] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [16:19:21] PROBLEM - db162 Disk Space on db162 is CRITICAL: DISK CRITICAL - free space: / 611MiB (1% inode=97%); [16:21:10] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e2ca832e9165...8bf944ebcba1 [16:21:11] [02miraheze/mw-config] 07AgentIsai 038bf944e - Change echo cluster [16:22:11] miraheze/mw-config - AgentIsai the build has errored. [16:22:21] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8bf944ebcba1...d591532f593f [16:22:23] [02miraheze/mw-config] 07AgentIsai 03d591532 - Fix comment [16:22:36] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d591532f593f...2b96ce8ef8ae [16:22:37] [02miraheze/mw-config] 07AgentIsai 032b96ce8 - Unset readonly [16:23:12] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [16:23:15] miraheze/mw-config - AgentIsai the build passed. [16:23:19] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 7s [16:23:35] miraheze/mw-config - AgentIsai the build passed. [16:23:39] !log [@test151] starting deploy of {'config': True} to test151 [16:23:40] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [16:23:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:26:06] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.qadrishattari.xyz All nameservers failed to answer the query. [16:27:04] PROBLEM - wiki.vot.gg - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.vot.gg could not be found [16:27:32] [02miraheze/MatomoAnalytics] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0 [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/commit/69d7d6e74e81 [16:27:34] [02miraheze/MatomoAnalytics] 07dependabot[bot] 0369d7d6e - Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 [16:27:36] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#115: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/115 [16:27:38] [02MatomoAnalytics] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0 - 13https://github.com/miraheze/MatomoAnalytics [16:27:41] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#115: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/115 [16:27:44] [02MatomoAnalytics] 07dependabot[bot] opened pull request 03#115: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/115 [16:32:13] miraheze/MatomoAnalytics - dependabot[bot] the build passed. [16:35:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/457d57deaf9e...42b0c2a41ef1 [16:35:57] [02miraheze/puppet] 07paladox 0342b0c2a - matomo: upgrade to 5.0.2 [16:54:13] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [16:56:05] RECOVERY - wiki.qadrishattari.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.qadrishattari.xyz reverse DNS resolves to cp37.wikitide.net - CNAME OK [17:03:20] [02puppet] 07Universal-Omega commented on pull request 03#3739: cvtbot: fix WorkingDirectory and Environment - 13https://github.com/miraheze/puppet/pull/3739#issuecomment-1927485392 [17:12:17] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03revert-3739-paladox-patch-14 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/91d36cba829a [17:12:19] [02miraheze/puppet] 07Universal-Omega 0391d36cb - Revert "cvtbot: fix WorkingDirectory and Environment (#3739)" [17:12:22] [02puppet] 07Universal-Omega created branch 03revert-3739-paladox-patch-14 - 13https://github.com/miraheze/puppet [17:12:23] [02puppet] 07Universal-Omega opened pull request 03#3742: Revert "cvtbot: fix WorkingDirectory and Environment (#3739)" - 13https://github.com/miraheze/puppet/pull/3742 [17:18:28] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-execute [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/d5d3e10565b5 [17:18:30] [02miraheze/puppet] 07Universal-Omega 03d5d3e10 - CVTBot: make sure irc has execute permissions [17:18:31] [02puppet] 07Universal-Omega created branch 03cvtbot-execute - 13https://github.com/miraheze/puppet [17:18:33] [02puppet] 07Universal-Omega opened pull request 03#3743: CVTBot: make sure irc has execute permissions - 13https://github.com/miraheze/puppet/pull/3743 [17:19:29] [02puppet] 07Universal-Omega closed pull request 03#3742: Revert "cvtbot: fix WorkingDirectory and Environment (#3739)" - 13https://github.com/miraheze/puppet/pull/3742 [17:19:30] [02miraheze/puppet] 07Universal-Omega deleted branch 03revert-3739-paladox-patch-14 [17:19:32] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/42b0c2a41ef1...38c46ea0330c [17:19:33] [02miraheze/puppet] 07Universal-Omega 0338c46ea - Revert "cvtbot: fix WorkingDirectory and Environment (#3739)" (#3742) [17:19:35] [02puppet] 07Universal-Omega deleted branch 03revert-3739-paladox-patch-14 - 13https://github.com/miraheze/puppet [17:19:43] [02puppet] 07Universal-Omega closed pull request 03#3743: CVTBot: make sure irc has execute permissions - 13https://github.com/miraheze/puppet/pull/3743 [17:19:45] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/38c46ea0330c...92971134e79c [17:19:46] [02miraheze/puppet] 07Universal-Omega 039297113 - CVTBot: make sure irc has execute permissions (#3743) [17:19:49] [02miraheze/puppet] 07Universal-Omega deleted branch 03cvtbot-execute [17:19:51] [02puppet] 07Universal-Omega deleted branch 03cvtbot-execute - 13https://github.com/miraheze/puppet [17:20:58] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:21:59] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/92971134e79c...0a2280f596e8 [17:22:01] [02miraheze/puppet] 07Universal-Omega 030a2280f - Fix [17:22:47] RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot' [17:27:45] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0a2280f596e8...99b017fbf493 [17:27:47] [02miraheze/puppet] 07Universal-Omega 0399b017f - cvtbot: set controlchannel to #miraheze-cvt [17:30:43] !log [agent@db161] SQL backing up on db161 [17:30:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:31:36] !log [agent@db151] Backing up SQL on db151 [17:31:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:04] !log [agent@db171] Backing up SQL on db171 [17:32:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:10] RECOVERY - db151 Backups SQL on db151 is OK: FILE_AGE OK: /var/log/sql-backup.log is 28 seconds old and 0 bytes [17:32:39] !log [agent@db181] Backing up SQL on db181 [17:32:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:42:48] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki/1.41/extensions/AbuseFilter/db_patches/mysql/patch-add-af_actor.sql (START) [17:42:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:00] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki/1.41/extensions/AbuseFilter/db_patches/mysql/patch-add-afh_actor.sql (START) [17:43:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:48:45] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/extensions/CheckUser/schema/mysql/cu_useragent_clienthints.sql (START) [17:48:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:48:55] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/extensions/CheckUser/schema/mysql/cu_useragent_clienthints_map.sql (START) [17:48:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:49:07] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/extensions/AntiSpoof/sql/mysql/patch-change-spoofuser-binary.sql (START) [17:49:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:49:26] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [17:49:44] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [17:55:27] [02miraheze/ManageWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/69e41abf4507 [17:55:30] [02miraheze/ManageWiki] 07dependabot[bot] 0369e41ab - Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 [17:55:31] [02ManageWiki] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0 - 13https://github.com/miraheze/ManageWiki [17:55:32] [02ManageWiki] 07dependabot[bot] labeled pull request 03#447: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/ManageWiki/pull/447 [17:55:34] [02ManageWiki] 07dependabot[bot] labeled pull request 03#447: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/ManageWiki/pull/447 [17:55:36] [02ManageWiki] 07dependabot[bot] opened pull request 03#447: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/ManageWiki/pull/447 [17:58:06] miraheze/ManageWiki - dependabot[bot] the build has errored. [18:06:23] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03rebuildVersionCache [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/d90d68533994 [18:06:26] [02miraheze/MirahezeMagic] 07Universal-Omega 03d90d685 - rebuildVersionCache: don't use GitInfoCacheDirectory [18:06:27] [02MirahezeMagic] 07Universal-Omega created branch 03rebuildVersionCache - 13https://github.com/miraheze/MirahezeMagic [18:06:28] [02MirahezeMagic] 07Universal-Omega opened pull request 03#459: rebuildVersionCache: don't use GitInfoCacheDirectory - 13https://github.com/miraheze/MirahezeMagic/pull/459 [18:12:42] miraheze/MirahezeMagic - Universal-Omega the build passed. [18:13:15] PROBLEM - phorge171 Puppet on phorge171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- renaming phab to issue-tracker, last run 5 minutes ago with 0 failures [18:18:31] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:18:36] [02MirahezeMagic] 07paladox closed pull request 03#459: rebuildVersionCache: don't use GitInfoCacheDirectory - 13https://github.com/miraheze/MirahezeMagic/pull/459 [18:18:37] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/1bc5cc0016a3...9df082380774 [18:18:38] [02miraheze/MirahezeMagic] 07Universal-Omega 039df0823 - rebuildVersionCache: don't use GitInfoCacheDirectory (#459) [18:18:40] [02miraheze/MirahezeMagic] 07paladox deleted branch 03rebuildVersionCache [18:18:41] [02MirahezeMagic] 07paladox deleted branch 03rebuildVersionCache - 13https://github.com/miraheze/MirahezeMagic [18:19:07] !log [paladox@mwtask181] starting deploy of {'pull': 'world', 'world': True, 'versions': ['1.40', '1.41']} to all [18:19:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:19:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2b96ce8ef8ae...4b6f5ac95c5a [18:19:11] [02miraheze/mw-config] 07Universal-Omega 034b6f5ac - Add phorge to disallowedSubdomains [18:19:11] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [18:20:04] miraheze/mw-config - Universal-Omega the build passed. [18:21:14] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/agent/pagetriage.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php extension /srv/mediawiki/1.41/extensions/PageTriage/sql/mysql/patch_ptrp_tags_updated_nullable.sql (START) [18:21:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:21:36] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/maintenance/archives/patch-externallinks-el_to_default.sql (START) [18:21:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:21:46] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/maintenance/archives/patch-filearchive-fa_size_to_bigint.sql (START) [18:21:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:22:09] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/maintenance/archives/patch-image-img_size_to_bigint.sql (START) [18:22:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:22:19] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/maintenance/archives/patch-oldimage-oi_size_to_bigint.sql (START) [18:22:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:22:31] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/maintenance/archives/patch-pagelinks-target_id.sql (START) [18:22:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:22:39] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03phorge [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/dc86f7e2c74d [18:22:39] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/maintenance/archives/patch-uploadstash-us_size_to_bigint.sql (START) [18:22:40] !log [paladox@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [18:22:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:22:42] [02miraheze/ssl] 07Universal-Omega 03dc86f7e - Add phorge redirects [18:22:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:22:45] [02ssl] 07Universal-Omega created branch 03phorge - 13https://github.com/miraheze/ssl [18:22:46] [02ssl] 07Universal-Omega opened pull request 03#773: Add phorge redirects - 13https://github.com/miraheze/ssl/pull/773 [18:22:52] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/maintenance/archives/patch-user-user_is_temp.sql (START) [18:22:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:01] !log [@mwtask181] starting deploy of {'config': True} to all [18:23:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:11] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [18:23:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:19] !log [@test151] starting deploy of {'config': True} to test151 [18:23:20] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [18:23:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:30] !log [@test151] starting deploy of {'folders': '1.40/extensions/MirahezeMagic'} to test151 [18:23:31] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [18:23:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:38] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [18:23:39] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [18:23:41] !log [@mwtask181] starting deploy of {'folders': '1.40/extensions/MirahezeMagic'} to all [18:23:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:47] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [18:23:48] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [18:23:49] !log [@mwtask181] finished deploy of {'folders': '1.40/extensions/MirahezeMagic'} to all - SUCCESS in 8s [18:23:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:01] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/agent/pagetriage.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php extension /srv/mediawiki/1.41/extensions/PageTriage/sql/mysql/patch_ptrp_tags_updated_nullable.sql (END - exit=256) [18:24:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:08] !log [paladox@test151] starting deploy of {'pull': 'config', 'config': True, 'force': True} to test151 [18:24:09] !log [paladox@test151] finished deploy of {'pull': 'config', 'config': True, 'force': True} to test151 - SUCCESS in 0s [18:24:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:18] !log [paladox@test151] starting deploy of {'pull': 'config', 'config': True, 'force': True} to test151 [18:24:19] !log [paladox@test151] finished deploy of {'pull': 'config', 'config': True, 'force': True} to test151 - SUCCESS in 0s [18:24:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:21] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [18:24:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:30] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 10s [18:24:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:38] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.40, 22.45, 17.39 [18:26:14] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync],Exec[MediaWiki-REL1_40 MirahezeMagic Sync],Exec[MediaWiki-REL1_41 MirahezeMagic Sync],Exec[MediaWiki-master MirahezeMagic Sync] [18:26:32] !log [paladox@test151] starting deploy of {'pull': 'config', 'config': True, 'force': True} to test151 [18:26:33] !log [paladox@test151] finished deploy of {'pull': 'config', 'config': True, 'force': True} to test151 - SUCCESS in 0s [18:26:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:52] miraheze/MirahezeMagic - paladox the build passed. [18:27:48] [02puppet] 07Universal-Omega closed pull request 03#3737: Rename phabricator.miraheze.org to issue-tracker.miraheze.org - 13https://github.com/miraheze/puppet/pull/3737 [18:27:49] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-1/±6] 13https://github.com/miraheze/puppet/compare/99b017fbf493...68bfb0ed6e8a [18:27:51] [02miraheze/puppet] 07Universal-Omega 0368bfb0e - Rename phabricator.miraheze.org to issue-tracker.miraheze.org (#3737) [18:27:52] [02miraheze/puppet] 07Universal-Omega deleted branch 03issue-tracker [18:27:53] [02puppet] 07Universal-Omega deleted branch 03issue-tracker - 13https://github.com/miraheze/puppet [18:29:03] !log [paladox@test151] starting deploy of {'config': True, 'world': True, 'l10n': True, 'versions': ['1.40', '1.41', '1.42']} to test151 [18:29:04] !log [paladox@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [18:29:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:29:07] !log [paladox@test151] starting deploy of {'config': True, 'world': True, 'l10n': True, 'force': True, 'versions': ['1.40', '1.41', '1.42']} to test151 [18:29:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:29:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:31:03] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03upgrade-141 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/4913cac486e0 [18:31:06] [02miraheze/mw-config] 07AgentIsai 034913cac - Upgrade all wikis to MediaWiki 1.41 [18:31:08] [02mw-config] 07AgentIsai created branch 03upgrade-141 - 13https://github.com/miraheze/mw-config [18:31:21] [02mw-config] 07AgentIsai opened pull request 03#5461: Upgrade all wikis to MediaWiki 1.41 - 13https://github.com/miraheze/mw-config/pull/5461 [18:31:31] [02ssl] 07Universal-Omega closed pull request 03#773: Add phorge redirects - 13https://github.com/miraheze/ssl/pull/773 [18:31:34] [02miraheze/ssl] 07Universal-Omega deleted branch 03phorge [18:31:36] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1485437e435a...99eb637f3857 [18:31:39] [02miraheze/ssl] 07Universal-Omega 0399eb637 - Add phorge redirects (#773) [18:31:40] [02ssl] 07Universal-Omega deleted branch 03phorge - 13https://github.com/miraheze/ssl [18:32:21] miraheze/mw-config - AgentIsai the build passed. [18:33:32] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:35:34] PROBLEM - phorge171 phabricator.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2919 bytes in 0.031 second response time [18:35:38] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.39, 23.86, 20.68 [18:36:13] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.598 second response time [18:37:01] !log UPDATE oauth_registered_consumer SET oarc_name = 'Miraheze Issue Tracker', oarc_description = 'Miraheze Issue Tracker Production', oarc_callback_url = 'https://issue-tracker.miraheze.org/auth/login/mediawiki:mediawiki.meta/' WHERE oarc_consumer_key = '...'; [18:37:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:38:14] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:39:38] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.32, 23.85, 21.39 [18:41:38] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.92, 23.81, 21.68 [18:43:28] !log [paladox@test151] finished deploy of {'config': True, 'world': True, 'l10n': True, 'force': True, 'versions': ['1.40', '1.41', '1.42']} to test151 - SUCCESS in 861s [18:43:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:43:40] !log [paladox@test151] starting deploy of {'config': True, 'world': True, 'l10n': True, 'force': True, 'versions': ['1.40', '1.41', '1.42']} to test151 [18:43:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:44:01] !log [paladox@mwtask181] starting deploy of {'world': True, 'l10n': True, 'versions': ['1.40', '1.41']} to all [18:44:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:45:38] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.26, 23.91, 22.20 [18:45:45] !log [paladox@test151] finished deploy of {'config': True, 'world': True, 'l10n': True, 'force': True, 'versions': ['1.40', '1.41', '1.42']} to test151 - SUCCESS in 125s [18:45:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:47:20] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [18:47:54] PROBLEM - phorge.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for phorge.miraheze.org could not be found [18:48:17] !log [paladox@mwtask181] finished deploy of {'world': True, 'l10n': True, 'versions': ['1.40', '1.41']} to all - SUCCESS in 255s [18:48:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:48:24] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/68bfb0ed6e8a...a585b59d4ac3 [18:48:26] [02miraheze/puppet] 07Universal-Omega 03a585b59 - Remove absented site [18:49:33] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.09, 23.93, 22.63 [18:50:53] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [18:51:28] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 24.10, 22.84 [18:53:22] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.54, 23.85, 22.90 [18:56:10] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 8947MiB (10% inode=98%); [18:56:46] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [18:57:28] PROBLEM - cp36 Current Load on cp36 is CRITICAL: LOAD CRITICAL - total load average: 12.11, 5.86, 2.84 [18:57:44] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 8255MiB (9% inode=98%); [18:57:47] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [18:58:07] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 6 backends are down. mw151 mw161 mw162 mw171 mw172 mw182 [18:58:17] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [18:58:24] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 6 backends are down. mw151 mw152 mw171 mw172 mw181 mw182 [18:58:30] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 3 backends are down. mw171 mw172 mw182 [18:58:36] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.30, 15.12, 8.12 [18:58:37] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: CRITICAL - NGINX Error Rate is 81% [18:58:54] PROBLEM - cp37 Current Load on cp37 is CRITICAL: LOAD CRITICAL - total load average: 8.82, 5.96, 2.95 [18:58:59] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/extensions/CheckUser/schema/mysql/cu_useragent_clienthints.sql (END - exit=0) [18:59:01] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/extensions/CheckUser/schema/mysql/cu_useragent_clienthints_map.sql (END - exit=0) [18:59:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:59:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:59:26] RECOVERY - cp36 Current Load on cp36 is OK: LOAD OK - total load average: 5.10, 5.72, 3.19 [18:59:46] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 3.538 second response time [18:59:58] [02mw-config] 07Universal-Omega reviewed pull request 03#5461 commit - 13https://github.com/miraheze/mw-config/pull/5461#discussion_r1478754518 [19:00:00] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw161 [19:00:16] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 5 backends are down. mw151 mw152 mw161 mw181 mw182 [19:00:36] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 20.17, 16.78, 9.57 [19:00:36] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 1% [19:00:43] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3572 bytes in 1.567 second response time [19:00:53] PROBLEM - cp37 Current Load on cp37 is WARNING: LOAD WARNING - total load average: 7.50, 6.47, 3.49 [19:01:57] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 18 backends are healthy [19:02:07] RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3594 bytes in 1.100 second response time [19:02:08] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 18 backends are healthy [19:02:15] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 18 backends are healthy [19:02:17] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 18 backends are healthy [19:02:26] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 18 backends are healthy [19:02:52] RECOVERY - cp37 Current Load on cp37 is OK: LOAD OK - total load average: 1.96, 4.76, 3.22 [19:04:10] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 40% [19:07:55] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/extensions/AntiSpoof/sql/mysql/patch-change-spoofuser-binary.sql (END - exit=0) [19:07:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:18] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki/1.41/extensions/AbuseFilter/db_patches/mysql/patch-add-af_actor.sql (END - exit=0) [19:08:19] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki/1.41/extensions/AbuseFilter/db_patches/mysql/patch-add-afh_actor.sql (END - exit=0) [19:08:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:11:48] PROBLEM - cp36 Disk Space on cp36 is CRITICAL: DISK CRITICAL - free space: / 5158MiB (5% inode=98%); [19:12:08] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 39% [19:13:07] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.90, 18.34, 20.30 [19:13:45] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 6717MiB (7% inode=98%); [19:17:32] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03upgrade-141 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4913cac486e0...9e9ca35eff13 [19:17:33] [02miraheze/mw-config] 07AgentIsai 039e9ca35 - Revert alpha to 1.42 [19:17:34] [02mw-config] 07AgentIsai synchronize pull request 03#5461: Upgrade all wikis to MediaWiki 1.41 - 13https://github.com/miraheze/mw-config/pull/5461 [19:20:37] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [19:35:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:36:22] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 35% [19:36:26] Agent: seems a huge refreshLinks backlog [19:36:46] Agent: maybe totaltenniswiki [19:37:24] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/maintenance/archives/patch-user-user_is_temp.sql (END - exit=0) [19:37:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:44] [02mw-config] 07AgentIsai closed pull request 03#5461: Upgrade all wikis to MediaWiki 1.41 - 13https://github.com/miraheze/mw-config/pull/5461 [19:37:46] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4b6f5ac95c5a...f8559d91bc5f [19:37:49] [02miraheze/mw-config] 07AgentIsai 03f8559d9 - Upgrade all wikis to MediaWiki 1.41 (#5461) [19:37:50] [02miraheze/mw-config] 07AgentIsai deleted branch 03upgrade-141 [19:37:51] [02mw-config] 07AgentIsai deleted branch 03upgrade-141 - 13https://github.com/miraheze/mw-config [19:38:05] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [19:38:08] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/maintenance/archives/patch-filearchive-fa_size_to_bigint.sql (END - exit=0) [19:38:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:38:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:38:12] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/maintenance/archives/patch-oldimage-oi_size_to_bigint.sql (END - exit=0) [19:38:13] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/maintenance/archives/patch-uploadstash-us_size_to_bigint.sql (END - exit=0) [19:38:14] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 8s [19:38:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:38:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:38:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:38:45] miraheze/mw-config - AgentIsai the build passed. [19:38:56] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/maintenance/archives/patch-image-img_size_to_bigint.sql (END - exit=0) [19:38:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:39:09] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/maintenance/archives/patch-externallinks-el_to_default.sql (END - exit=0) [19:39:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:44:26] [02miraheze/RottenLinks] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/commit/46a4b91c1d0a [19:44:27] [02miraheze/RottenLinks] 07dependabot[bot] 0346a4b91 - Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 [19:44:30] [02RottenLinks] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0 - 13https://github.com/miraheze/RottenLinks [19:44:32] [02RottenLinks] 07dependabot[bot] opened pull request 03#77: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/RottenLinks/pull/77 [19:44:34] [02RottenLinks] 07dependabot[bot] labeled pull request 03#77: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/RottenLinks/pull/77 [19:45:10] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php /srv/mediawiki-staging/1.41/maintenance/archives/patch-pagelinks-target_id.sql (END - exit=65280) [19:45:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:45:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:48:28] miraheze/RottenLinks - dependabot[bot] the build passed. [19:53:04] !log [@test151] starting deploy of {'config': True} to test151 [19:53:05] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [19:53:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:43] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=metawiki --wikidb mhglobal /srv/mediawiki/1.41/extensions/OATHAuth/sql/mysql/tables-generated.sql (END - exit=256) [19:54:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:00] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/OATHAuth/maintenance/UpdateForMultipleDevicesSupport.php --wiki=metawiki (END - exit=0) [19:55:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:20] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/OATHAuth/maintenance/UpdateForMultipleDevicesSupport.php --wiki=loginwiki (END - exit=0) [19:55:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:11] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/AbuseFilter/maintenance/MigrateActorsAF.php (START) [19:56:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:21] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/fixInconsistentRedirects.php (START) [19:56:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:16] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php /srv/mediawiki/1.41/extensions/LoginNotify/sql/mysql/tables-generated.sql (START) [20:05:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:15] !log [paladox@mwtask181] starting deploy of {'folders': '1.41/vendor'} to all [20:10:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:24] !log [paladox@mwtask181] finished deploy of {'folders': '1.41/vendor'} to all - SUCCESS in 8s [20:10:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:12] RECOVERY - vmcodex.net - reverse DNS on sslhost is OK: SSL OK - vmcodex.net reverse DNS resolves to cp37.wikitide.net - CNAME FLAT [20:14:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a585b59d4ac3...9ba27b17a2b4 [20:14:57] [02miraheze/puppet] 07paladox 039ba27b1 - Switch default branch to 1.41 [20:24:22] [02miraheze/mediawiki-repos] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/1a393f6f4a75...c81b29421df1 [20:24:25] [02miraheze/mediawiki-repos] 07paladox 03c81b294 - Use master branch for Anisa [20:25:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [20:25:37] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address miraheze.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:26:12] PROBLEM - wiki.vot.gg - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:37] PROBLEM - cp36 Nginx Backend for puppet181 on cp36 is CRITICAL: connect to address localhost and port 8204: Connection refused [20:26:37] PROBLEM - cp36 Nginx Backend for mw172 on cp36 is CRITICAL: connect to address localhost and port 8118: Connection refused [20:26:40] PROBLEM - cp36 Nginx Backend for phorge171 on cp36 is CRITICAL: connect to address localhost and port 8202: Connection refused [20:26:45] PROBLEM - cp36 Nginx Backend for mw161 on cp36 is CRITICAL: connect to address localhost and port 8115: Connection refused [20:26:52] PROBLEM - cp36 Nginx Backend for mw181 on cp36 is CRITICAL: connect to address localhost and port 8119: Connection refused [20:26:54] PROBLEM - wiki.zahira.me - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:56] PROBLEM - cp36 Nginx Backend for swiftproxy161 on cp36 is CRITICAL: connect to address localhost and port 8206: Connection refused [20:26:57] PROBLEM - wiki.fbpml.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:59] PROBLEM - cp36 Nginx Backend for mw152 on cp36 is CRITICAL: connect to address localhost and port 8114: Connection refused [20:27:01] PROBLEM - petrawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:02] PROBLEM - monarchists.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:03] PROBLEM - wiki.tmyt105.leyhp.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:09] PROBLEM - cp36 Nginx Backend for test151 on cp36 is CRITICAL: connect to address localhost and port 8181: Connection refused [20:27:10] PROBLEM - cp36 NTP time on cp36 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o