[00:00:40] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:00:48] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7dcfb0a1155e...fdbc4d0f11c1 [00:00:51] [02miraheze/puppet] 07AgentIsai 03fdbc4d0 - Add mwtask171 to mwdeploy [00:01:21] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:13] miraheze/puppet - AgentIsai the build passed. [00:02:37] !log [agent@mwtask181] starting deploy of {'config': True} to all [00:02:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:40] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:02:46] !log [agent@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mwtask171.wikitide.net [00:02:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:55] !log [agent@mwtask181] starting deploy of {'config': True, 'force': True} to mwtask171 [00:02:56] !log [agent@mwtask181] finished deploy of {'config': True, 'force': True} to mwtask171 - FAIL: [65280] [00:02:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:21] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:03:26] !log [agent@mwtask181] starting deploy of {'force': True, 'folders': '1.41'} to mwtask171 [00:03:27] !log [agent@mwtask181] finished deploy of {'force': True, 'folders': '1.41'} to mwtask171 - FAIL: [65280] [00:03:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:04:41] PROBLEM - cp28 Disk Space on cp28 is WARNING: DISK WARNING - free space: / 29086MiB (6% inode=99%); [00:16:10] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:59:09] Anyone with server access/knowledge know if my theory on [[phab:T11912]] of missing software is possible? I may be totally wrong so wondering if anyone knew if the software is def added [00:59:09] https://meta.miraheze.org/wiki/phab:T11912 [00:59:11]

[01:03:16] All of the required packages are installed [01:03:37] gucci, I’ll leave another comment later [01:03:50] Seems it’s probably the settings theb [01:03:56] which confuses me more [01:04:10] cause like how has no one reported that [01:04:29] if the settings needed to work are missing we’d have noticed [01:05:03] I mean i can check mw-config tmr [01:15:27] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:23:58] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fdbc4d0f11c1...017db093eaf0 [01:24:01] [02miraheze/puppet] 07AgentIsai 03017db09 - Also include deploykey for staging [01:53:34] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/017db093eaf0...629a1fabbb0c [01:53:37] [02miraheze/puppet] 07AgentIsai 03629a1fa - Ensure www-data exists in canary hosts [01:54:28] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/629a1fabbb0c...5ae946ee72bc [01:54:31] [02miraheze/puppet] 07AgentIsai 035ae946e - mediawiki staging: do not include rsync [01:56:27] !log [agent@mwtask181] starting deploy of {'config': True, 'force': True} to mwtask171 [01:56:28] !log [agent@mwtask181] finished deploy of {'config': True, 'force': True} to mwtask171 - SUCCESS in 1s [01:56:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:56:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:56:34] !log [agent@mwtask181] starting deploy of {'force': True, 'folders': '1.41'} to mwtask171 [01:56:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:56:55] !log [agent@mwtask181] finished deploy of {'force': True, 'folders': '1.41'} to mwtask171 - SUCCESS in 20s [01:56:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:57:24] !log [agent@mwtask181] starting deploy of {'upgrade_vendor': True, 'landing': True, 'errorpages': True, 'force': True, 'versions': '1.41'} to mwtask171 [01:57:26] !log [agent@mwtask181] finished deploy of {'upgrade_vendor': True, 'landing': True, 'errorpages': True, 'force': True, 'versions': '1.41'} to mwtask171 - SUCCESS in 1s [01:57:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:57:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:57:50] !log [agent@mwtask181] starting deploy of {'force': True, 'folders': '../mediawiki/cache'} to mwtask171 [01:57:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:57:57] !log [agent@mwtask181] finished deploy of {'force': True, 'folders': '../mediawiki/cache'} to mwtask171 - FAIL: [2] [01:58:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:58:05] !log [agent@mwtask181] starting deploy of {'l10n': True, 'force': True, 'versions': '1.41'} to mwtask171 [01:58:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:58:20] !log [agent@mwtask181] finished deploy of {'l10n': True, 'force': True, 'versions': '1.41'} to mwtask171 - SUCCESS in 14s [01:58:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:58:44] RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.066 second response time [01:58:56] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.661 second response time [01:59:59] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=metawiki (START) [02:00:00] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=metawiki (END - exit=0) [02:00:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:05] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=polandballwiki (START) [02:00:06] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=polandballwiki (END - exit=0) [02:00:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:11] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=rosettacodewiki (START) [02:00:12] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=rosettacodewiki (END - exit=0) [02:00:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:22:49] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/e2f80b676765...dd2f4c00b8bc [02:22:52] [02miraheze/ImportDump] 07Universal-Omega 03dd2f4c0 - ImportDumpJob: remove paths from runChild [02:26:50] !log [universalomega@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_pack': 'miraheze'} to all [02:26:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:28:30] !log [universalomega@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_pack': 'miraheze'} to all - SUCCESS in 100s [02:28:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:28:48] !log [universalomega@mwtask181] starting deploy of {'l10n': True, 'versions': '1.41'} to all [02:28:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:31:17] miraheze/ImportDump - Universal-Omega the build passed. [02:34:49] !log [universalomega@mwtask181] finished deploy of {'l10n': True, 'versions': '1.41'} to all - SUCCESS in 361s [02:34:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:36:02] [02mediawiki-archived] 07Universal-Omega created branch 03master - 13https://github.com/miraheze/mediawiki-archived [02:36:04] [02miraheze/mediawiki-archived] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/mediawiki-archived/commit/f7847031d368 [02:36:06] [02miraheze/mediawiki-archived] 07Universal-Omega 03f784703 - Initial commit [02:49:43] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/deleteWiki.php --wiki=sagan4alphawiki --delete --deletewiki sagan4alpha (END - exit=0) [02:49:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:28] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/deleteWiki.php --wiki=sagan4alphawiki --delete --deletewiki sagan4alphawiki (END - exit=256) [02:52:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:46] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/deleteWiki.php --wiki=testwiki --delete --deletewiki sagan4alphawiki (END - exit=256) [02:52:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:30:45] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/deleteWiki.php --wiki=sagan4alphawiki --delete --deletewiki sagan4alphawiki (END - exit=0) [03:30:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:32:27] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 156.76 ms [03:36:29] RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 138.76 ms [03:42:46] [02miraheze/CreateWiki] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/d0b616256723...8e0333af7250 [03:42:48] [02miraheze/CreateWiki] 07AgentIsai 038e0333a - Return FatalError if trying to delet a wiki that does not exist [03:48:05] miraheze/CreateWiki - AgentIsai the build has errored. [03:55:58] [02miraheze/CreateWiki] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/886b42ca4a73 [03:56:00] [02miraheze/CreateWiki] 07AgentIsai 03886b42c - Expect fatal error [03:56:01] [02CreateWiki] 07AgentIsai created branch 03AgentIsai-patch-1 - 13https://github.com/miraheze/CreateWiki [03:56:07] [02CreateWiki] 07AgentIsai opened pull request 03#480: CI: Expect fatal error on certain conditions - 13https://github.com/miraheze/CreateWiki/pull/480 [03:56:11] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 158.12 ms [04:00:15] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 179.43 ms [04:01:42] miraheze/CreateWiki - AgentIsai the build has errored. [04:03:38] [02miraheze/MirahezeMagic] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/cecc2ebff9d7 [04:03:41] [02miraheze/MirahezeMagic] 07AgentIsai 03cecc2eb - Test using LoggerFactory for logging [04:03:44] [02MirahezeMagic] 07AgentIsai created branch 03AgentIsai-patch-1 - 13https://github.com/miraheze/MirahezeMagic [04:03:46] [02MirahezeMagic] 07AgentIsai opened pull request 03#473: Test using LoggerFactory for logging - 13https://github.com/miraheze/MirahezeMagic/pull/473 [04:05:35] [02miraheze/MirahezeMagic] 07github-actions[bot] pushed 031 commit to 03AgentIsai-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/cecc2ebff9d7...db3f0cfe3bb2 [04:05:36] [02miraheze/MirahezeMagic] 07github-actions 03db3f0cf - CI: lint code to MediaWiki standards [04:05:38] [02MirahezeMagic] 07github-actions[bot] synchronize pull request 03#473: Test using LoggerFactory for logging - 13https://github.com/miraheze/MirahezeMagic/pull/473 [04:07:13] miraheze/MirahezeMagic - AgentIsai the build has errored. [04:09:34] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=sagan4alphawiki (END - exit=0) [04:09:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:10:08] !log [agent@mwtask181] starting deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all [04:10:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:10:17] !log [agent@mwtask181] finished deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 9s [04:10:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:11:05] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=sagan4alphawiki --no-updates sagan4alphawiki_xml_38716137bb8e7ad2cc28.xml (START) [04:11:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:13:07] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=sagan4alphawiki --no-updates sagan4alphawiki_xml_38716137bb8e7ad2cc28.xml (END - exit=2) [04:13:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:18:59] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/deleteWiki.php --wiki=testwiki --delete --deletewiki sagan4alphawiki (END - exit=0) [04:19:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:23:38] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=sagan4alphawiki --no-updates sagan4alphawiki_xml_38716137bb8e7ad2cc28.xml (START) [04:23:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:24:50] [02MirahezeMagic] 07AgentIsai synchronize pull request 03#473: Test using LoggerFactory for logging - 13https://github.com/miraheze/MirahezeMagic/pull/473 [04:24:53] [02miraheze/MirahezeMagic] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/db3f0cfe3bb2...19c59e275eed [04:24:56] [02miraheze/MirahezeMagic] 07AgentIsai 0319c59e2 - Fix escape [04:26:15] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/deleteWiki.php --wiki=testwiki --delete --deletewiki sagan4wiki (END - exit=0) [04:26:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:29:05] miraheze/MirahezeMagic - AgentIsai the build has errored. [04:33:01] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#473: Test using LoggerFactory for logging - 13https://github.com/miraheze/MirahezeMagic/pull/473#issuecomment-1970377960 [04:34:43] [02MirahezeMagic] 07AgentIsai synchronize pull request 03#473: Test using LoggerFactory for logging - 13https://github.com/miraheze/MirahezeMagic/pull/473 [04:34:44] [02miraheze/MirahezeMagic] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/19c59e275eed...15363e949eec [04:34:45] [02miraheze/MirahezeMagic] 07AgentIsai 0315363e9 - Fix escaping [04:36:46] [02CreateWiki] 07Universal-Omega commented on pull request 03#480: CI: Expect fatal error on certain conditions - 13https://github.com/miraheze/CreateWiki/pull/480#issuecomment-1970380661 [04:38:52] [02CreateWiki] 07Universal-Omega commented on pull request 03#480: CI: Expect fatal error on certain conditions - 13https://github.com/miraheze/CreateWiki/pull/480#issuecomment-1970382298 [04:39:02] miraheze/MirahezeMagic - AgentIsai the build has errored. [04:39:51] [02CreateWiki] 07Universal-Omega commented on pull request 03#480: CI: Expect fatal error on certain conditions - 13https://github.com/miraheze/CreateWiki/pull/480#issuecomment-1970383075 [04:40:43] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=sagan4wiki --no-updates sagan4wiki_xml_253a52472386da99ec62.xml (START) [04:40:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:44:06] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03revert-8e0333a [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/24a03630184f [04:44:09] [02miraheze/CreateWiki] 07Universal-Omega 0324a0363 - Revert "Return FatalError if trying to delet a wiki that does not exist" [04:44:12] [02CreateWiki] 07Universal-Omega created branch 03revert-8e0333a - 13https://github.com/miraheze/CreateWiki [04:44:13] [02CreateWiki] 07Universal-Omega opened pull request 03#481: Revert "Return FatalError if trying to delet a wiki that does not exist" - 13https://github.com/miraheze/CreateWiki/pull/481 [04:45:37] [02CreateWiki] 07Universal-Omega commented on pull request 03#481: Revert "Return FatalError if trying to delet a wiki that does not exist" - 13https://github.com/miraheze/CreateWiki/pull/481#issuecomment-1970387626 [04:46:05] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=sagan4wiki --no-updates sagan4wiki_xml_253a52472386da99ec62.xml (END - exit=0) [04:46:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:46:54] [02CreateWiki] 07Universal-Omega closed pull request 03#481: Revert "Return FatalError if trying to delet a wiki that does not exist" - 13https://github.com/miraheze/CreateWiki/pull/481 [04:46:56] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03revert-8e0333a [04:46:59] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/8e0333af7250...aa45e329c2c1 [04:47:03] [02miraheze/CreateWiki] 07Universal-Omega 03aa45e32 - Revert "Return FatalError if trying to delet a wiki that does not exist" [04:47:04] [02CreateWiki] 07Universal-Omega deleted branch 03revert-8e0333a - 13https://github.com/miraheze/CreateWiki [04:47:55] [02CreateWiki] 07Universal-Omega closed pull request 03#480: CI: Expect fatal error on certain conditions - 13https://github.com/miraheze/CreateWiki/pull/480 [04:47:58] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03AgentIsai-patch-1 [04:48:01] [02CreateWiki] 07Universal-Omega deleted branch 03AgentIsai-patch-1 - 13https://github.com/miraheze/CreateWiki [04:50:00] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4wiki . --search-recursively (START) [04:50:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:50:07] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4wiki . --search-recursively (END - exit=0) [04:50:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:50:39] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03deletewikis-delete-output [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/7f8e2788dc57 [04:50:40] [02miraheze/CreateWiki] 07Universal-Omega 037f8e278 - deleteWiki: give fatalError if WikiManager::delete has output [04:50:41] [02CreateWiki] 07Universal-Omega created branch 03deletewikis-delete-output - 13https://github.com/miraheze/CreateWiki [04:50:45] [02CreateWiki] 07Universal-Omega opened pull request 03#482: deleteWiki: give fatalError if WikiManager::delete has output - 13https://github.com/miraheze/CreateWiki/pull/482 [04:51:16] [02CreateWiki] 07Universal-Omega commented on pull request 03#480: CI: Expect fatal error on certain conditions - 13https://github.com/miraheze/CreateWiki/pull/480#issuecomment-1970392320 [04:51:22] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/restoreManageWikiBackups.php --wiki=sagan4wiki --filename sagan4wiki_managewiki_backup_36c8d960247d7dddc15a.json (END - exit=256) [04:51:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:51:31] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/restoreManageWikiBackup.php --wiki=sagan4wiki --filename sagan4wiki_managewiki_backup_36c8d960247d7dddc15a.json (END - exit=0) [04:51:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:51:57] miraheze/CreateWiki - Universal-Omega the build passed. [04:52:03] [02CreateWiki] 07Universal-Omega synchronize pull request 03#482: deleteWiki: give fatalError if WikiManager::delete has output - 13https://github.com/miraheze/CreateWiki/pull/482 [04:52:05] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03deletewikis-delete-output [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/7f8e2788dc57...d8226fc3ead1 [04:52:06] [02miraheze/CreateWiki] 07Universal-Omega 03d8226fc - Fix [04:52:25] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=sagan4wiki (START) [04:52:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:53:03] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=sagan4wiki (END - exit=0) [04:53:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:54:32] miraheze/CreateWiki - Universal-Omega the build passed. [04:56:39] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/deleteWiki.php --wiki=testwiki --delete --deletewiki sagan4masonwiki (END - exit=0) [04:56:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:00:24] miraheze/CreateWiki - Universal-Omega the build passed. [05:00:50] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/restoreManageWikiBackups.php --wiki=sagan4masonwiki --filename sagan4masonwiki_managewiki_backup_87545fc9d9fe39b13e19.json (END - exit=256) [05:00:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:01:18] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/restoreManageWikiBackup.php --wiki=sagan4masonwiki --filename sagan4masonwiki_managewiki_backup_87545fc9d9fe39b13e19.json (END - exit=0) [05:01:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:02:28] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=sagan4masonwiki --no-updates sagan4masonwiki_xml_732944d9131d03960bc1.xml (START) [05:02:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:02:54] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=sagan4masonwiki (END - exit=0) [05:02:57] !log [agent@mwtask181] starting deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all [05:02:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:03:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:03:06] !log [agent@mwtask181] finished deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 9s [05:03:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:03:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/81f85bfbaccf...f013f471f831 [05:03:54] [02miraheze/ssl] 07MirahezeSSLBot 03f013f47 - Bot: Add SSL cert for meta.sagan4.org [05:04:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/f013f471f831...53aa9eb6484a [05:04:10] [02miraheze/ssl] 07MirahezeSSLBot 0353aa9eb - Bot: Add SSL cert for mason.sagan4.org [05:05:29] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=sagan4masonwiki --no-updates sagan4masonwiki_xml_732944d9131d03960bc1.xml (END - exit=0) [05:05:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:07:26] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4masonwiki . --search-recursively (START) [05:07:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:07:30] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4masonwiki . --search-recursively (END - exit=2) [05:07:31] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4masonwiki . --search-recursively (START) [05:07:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:07:33] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4masonwiki . --search-recursively (END - exit=2) [05:07:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:07:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:07:53] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=sagan4masonwiki (END - exit=0) [05:07:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:07:57] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4masonwiki . --search-recursively (START) [05:07:59] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4masonwiki . --search-recursively (END - exit=2) [05:07:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:08:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:08:20] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4masonwiki . --search-recursively (START) [05:08:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:08:27] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4masonwiki . --search-recursively (END - exit=0) [05:08:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:08:34] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=sagan4masonwiki (END - exit=0) [05:08:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:09:57] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=sagan4masonwiki (END - exit=0) [05:10:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:20:51] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.89 ms [05:22:52] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 174.10 ms [05:26:33] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=sagan4betawiki (END - exit=0) [05:26:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:26:37] !log [agent@mwtask181] starting deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all [05:26:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:26:47] !log [agent@mwtask181] finished deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 9s [05:26:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:27:49] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=sagan4betawiki --no-updates sagan4betawiki_xml_c389ff4731efecd8eb6d.xml (START) [05:27:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:29:11] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=sagan4masonwiki (END - exit=0) [05:29:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:04] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=sagan4masonwiki (END - exit=0) [05:33:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:35:04] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=sagan4masonwiki (END - exit=0) [05:35:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:35:41] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4masonwiki . --search-recursively (START) [05:35:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:37:28] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4masonwiki . --search-recursively (END - exit=0) [05:37:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:37:45] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=sagan4masonwiki (START) [05:37:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:38:21] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=sagan4betawiki --no-updates sagan4betawiki_xml_c389ff4731efecd8eb6d.xml (START) [05:38:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:39:08] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=sagan4masonwiki (END - exit=0) [05:39:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:42:36] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=sagan4betawiki --no-updates sagan4betawiki_xml_c389ff4731efecd8eb6d.xml (END - exit=0) [05:42:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:50:55] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=sagan4betawiki --no-updates sagan4betawiki_xml_c389ff4731efecd8eb6d.xml (END - exit=0) [05:50:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:56:43] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/restoreManageWikiBackup.php --wiki=sagan4betawiki --filename sagan4betawiki_managewiki_backup_d2e9543bcb3667226b03.json (END - exit=0) [05:56:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:00:09] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4betawiki . --search-recursively (START) [06:00:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:00:13] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4betawiki . --search-recursively (END - exit=2) [06:00:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:01:11] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4betawiki . --search-recursively (START) [06:01:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:14:21] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4betawiki . --search-recursively (END - exit=0) [06:14:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:20:33] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 175.60 ms [06:24:34] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 178.31 ms [06:31:41] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=sagan4alphawiki --no-updates sagan4alphawiki_xml_38716137bb8e7ad2cc28.xml (END - exit=0) [06:31:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:35:30] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:40:30] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [07:36:59] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 191.50 ms [07:41:03] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 182.05 ms [07:56:46] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 162.55 ms [07:58:47] RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 161.20 ms [08:42:34] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 174.48 ms [08:48:41] RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 170.93 ms [09:03:51] PROBLEM - db181 Backups SQL on db181 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864183 seconds old and 112234 bytes [09:14:39] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:21:32] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 17.41, 20.58, 9.09 [09:25:23] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.47, 9.56, 7.10 [09:43:41] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [10:59:58] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 143.50 ms [11:01:59] RECOVERY - ping6 on cp28 is OK: PING OK - Packet loss = 0%, RTA = 162.85 ms [12:12:21] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/CreateWiki/compare/aa45e329c2c1...5842160c1148 [12:12:22] [02miraheze/CreateWiki] 07translatewiki 035842160 - Localisation updates from https://translatewiki.net. [12:12:23] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±15] 13https://github.com/miraheze/ImportDump/compare/dd2f4c00b8bc...b66e9a146227 [12:12:25] [02miraheze/ImportDump] 07translatewiki 03b66e9a1 - Localisation updates from https://translatewiki.net. [12:12:28] [02miraheze/GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/GlobalNewFiles/compare/3bcd7411b986...34db58fbd088 [12:12:29] [02miraheze/GlobalNewFiles] 07translatewiki 0334db58f - Localisation updates from https://translatewiki.net. [12:12:32] [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/DataDump/compare/a3b7add129fb...937c6b556bf7 [12:12:33] [02miraheze/DataDump] 07translatewiki 03937c6b5 - Localisation updates from https://translatewiki.net. [12:12:36] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/34ab495b1b50...8b5926926f50 [12:12:37] [02miraheze/landing] 07translatewiki 038b59269 - Localisation updates from https://translatewiki.net. [12:12:40] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/ManageWiki/compare/e0653dcc83ab...16a2555b0398 [12:12:43] [02miraheze/ManageWiki] 07translatewiki 0316a2555 - Localisation updates from https://translatewiki.net. [12:12:46] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/9323c3c6565b...f149a075482d [12:12:48] [02miraheze/MirahezeMagic] 07translatewiki 03f149a07 - Localisation updates from https://translatewiki.net. [12:12:49] [02miraheze/RequestSSL] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/RequestSSL/compare/02f6bf3a64be...42cd87dc8f70 [12:12:52] [02miraheze/RequestSSL] 07translatewiki 0342cd87d - Localisation updates from https://translatewiki.net. [12:12:55] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/193382bb3f40...a8b26edfddc2 [12:12:58] [02miraheze/WikiDiscover] 07translatewiki 03a8b26ed - Localisation updates from https://translatewiki.net. [12:13:22] miraheze/landing - translatewiki the build passed. [12:16:16] miraheze/RequestSSL - translatewiki the build has errored. [12:16:36] miraheze/GlobalNewFiles - translatewiki the build passed. [12:16:42] miraheze/DataDump - translatewiki the build passed. [12:16:59] miraheze/WikiDiscover - translatewiki the build passed. [12:17:09] miraheze/ManageWiki - translatewiki the build passed. [12:20:59] miraheze/CreateWiki - translatewiki the build passed. [12:21:17] miraheze/ImportDump - translatewiki the build passed. [12:25:47] PROBLEM - db161 Backups SQL on db161 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864172 seconds old and 111810 bytes [12:26:08] miraheze/MirahezeMagic - translatewiki the build passed. [12:32:33] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:32:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:42] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:32:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:14] !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:37:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:22] !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s [12:37:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:46:38] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 152.83 ms [12:48:39] RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 145.51 ms [13:37:58] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 185.91 ms [13:52:08] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 178.70 ms [14:22:55] [02MirahezeMagic] 07amire80 opened pull request 03#474: Correct message spelling - "whereas" - 13https://github.com/miraheze/MirahezeMagic/pull/474 [14:25:12] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 198.15 ms [14:27:44] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 157.56 ms [14:29:00] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=sagan4betawiki (START) [14:29:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:29:45] RECOVERY - ping6 on cp28 is OK: PING OK - Packet loss = 0%, RTA = 153.57 ms [14:31:16] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 213.26 ms [14:33:14] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=sagan4betawiki (END - exit=0) [14:33:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:35:27] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 210.27 ms [14:37:21] miraheze/MirahezeMagic - amire80 the build passed. [14:39:29] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 201.58 ms [14:43:13] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/53aa9eb6484a...6243a2cd194f [14:43:14] [02miraheze/ssl] 07MirahezeSSLBot 036243a2c - Bot: Add SSL cert for beta.sagan4.org [14:44:43] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4alphawiki . --search-recursively (START) [14:44:45] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4alphawiki . --search-recursively (END - exit=2) [14:44:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:44:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:45:29] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4alphawiki . --search-recursively (START) [14:45:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:45:43] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 212.39 ms [14:50:38] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 154.71 ms [14:51:42] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 197.72 ms [14:52:36] RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 155.90 ms [14:56:41] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 158.68 ms [14:58:39] RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 150.06 ms [14:59:42] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=dreamlogoswiki thedreamlogowikiwiki_xml_166eee6d679667b9207a.xml --no-updates (START) [14:59:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:10:39] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [15:23:19] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/restoreManageWikiBackup.php --wiki=sagan4alphawiki --filename sagan4alphawiki_managewiki_backup_80b7df68fc6ff16ba3f6.json (END - exit=0) [15:23:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:25:21] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=sagan4alphawiki --procs 6 (START) [15:25:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:25:30] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:26:29] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=sagan4alphawiki --procs 6 --memory-limit max (START) [15:26:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:27:00] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=sagan4alphawiki --procs 6 --memory-limit max (END - exit=0) [15:27:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:27:46] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=sagan4alphawiki --procs 6 (END - exit=0) [15:27:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:28:16] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=sagan4alphawiki --procs 6 (START) [15:28:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:28:34] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=sagan4alphawiki --procs 6 (END - exit=0) [15:28:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:36:11] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:39:48] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 198.09 ms [15:41:47] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 206.76 ms [15:42:22] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4alphawiki . --search-recursively (END - exit=0) [15:42:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:50:41] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=dreamlogoswiki thedreamlogowikiwiki_xml_166eee6d679667b9207a.xml --no-updates (END - exit=0) [15:50:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:00:30] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:06:20] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=dreamlogoswiki . --search-recursively (START) [16:06:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:06:48] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4alphawiki . --search-recursively (START) [16:06:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:07:05] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4alphawiki . --search-recursively (END - exit=0) [16:07:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:07:56] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=sagan4alphawiki (END - exit=0) [16:08:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:08:07] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=sagan4alphawiki (START) [16:08:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:11:06] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 207.77 ms [16:13:05] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 197.21 ms [16:23:51] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=dreamlogoswiki . --search-recursively (END - exit=0) [16:23:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:39:16] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 192.63 ms [16:41:14]