[00:00:31] miraheze/MirahezeMagic - Universal-Omega the build has errored. [00:03:04] PROBLEM - cp36 Disk Space on cp36 is CRITICAL: DISK CRITICAL - free space: / 5001MiB (5% inode=98%); [00:03:16] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 5134MiB (5% inode=98%); [00:05:04] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 20538MiB (23% inode=98%); [00:05:16] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 20694MiB (23% inode=98%); [00:05:58] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [00:06:06] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 50% [00:09:08] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240210000821.xml (START) [00:09:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:10:24] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240210000821.xml (END - exit=256) [00:10:25] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (START) [00:10:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:10:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:12:48] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (END - exit=0) [00:12:49] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=scruffywiki --active --update (END - exit=0) [00:12:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:12:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:13:38] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240210000821.xml (START) [00:13:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:13:51] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240210000821.xml (END - exit=256) [00:13:52] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (START) [00:13:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:13:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:03] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (END - exit=0) [00:16:04] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=scruffywiki --active --update (END - exit=0) [00:16:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:57] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 45% [00:18:32] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240210000821.xml (START) [00:18:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:44] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240210000821.xml (END - exit=256) [00:18:45] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (START) [00:18:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:56] PROBLEM - wiki.leapit.no - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.leapit.no. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:20:03] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 38% [00:23:18] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [00:23:19] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [00:23:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:27] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [00:23:28] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [00:23:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:31] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240210000821.xml (START) [00:23:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:04] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240210000821.xml (END - exit=0) [00:24:05] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (START) [00:24:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:37] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (END - exit=0) [00:24:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:11] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (END - exit=0) [00:27:12] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=scruffywiki --active --update (END - exit=0) [00:27:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:59] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 43% [00:32:03] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240210000821.xml (START) [00:32:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:19] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240210000821.xml (END - exit=0) [00:32:20] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (START) [00:32:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:57] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 38% [00:34:58] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (END - exit=0) [00:34:59] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=scruffywiki --active --update (END - exit=0) [00:35:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:35:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:35:49] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [00:48:46] PROBLEM - wiki.leapit.no - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.leapit.no All nameservers failed to answer the query. [00:54:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b446d765000d...f9dac90b659d [00:54:05] [02miraheze/mw-config] 07Universal-Omega 03f9dac90 - Fix support for $wmgLogToDisk [00:54:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f9dac90b659d...e48c79b95bd0 [00:54:50] [02miraheze/mw-config] 07Universal-Omega 03e48c79b - Enable $wgShowExceptionDetails on test151 [00:55:03] miraheze/mw-config - Universal-Omega the build passed. [00:55:10] !log [universalomega@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [00:55:11] !log [universalomega@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [00:55:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:42] !log [universalomega@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [00:55:43] !log [universalomega@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [00:55:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:46] miraheze/mw-config - Universal-Omega the build passed. [00:55:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:02:36] !log [@mwtask181] starting deploy of {'config': True} to all [01:02:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:02:45] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 8s [01:02:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:05:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e48c79b95bd0...68f9501d0103 [01:05:04] [02miraheze/mw-config] 07Universal-Omega 0368f9501 - Enable $wgShowExceptionDetails on mwtask181 [01:05:27] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [01:05:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:05:36] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [01:05:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:05:58] miraheze/mw-config - Universal-Omega the build passed. [01:13:45] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 38% [01:18:35] RECOVERY - wiki.leapit.no - reverse DNS on sslhost is OK: SSL OK - wiki.leapit.no reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:23:01] !log [@test151] starting deploy of {'config': True} to test151 [01:23:02] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [01:23:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:25:31] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240209225206.xml (START) [01:25:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:25:53] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240209225206.xml (END - exit=256) [01:25:54] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (START) [01:25:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:25:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:25] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (END - exit=2) [01:26:26] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=scruffywiki --active --update (END - exit=0) [01:26:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:31] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240209225206.xml (START) [01:26:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:49] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240209225206.xml (END - exit=256) [01:26:49] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (START) [01:26:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:49] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (END - exit=2) [01:27:50] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=scruffywiki --active --update (END - exit=0) [01:27:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:29:56] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240209225206.xml (START) [01:29:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:30:14] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=scruffywiki --no-updates /home/universalomega/ImportDump/scruffywiki-20240209225206.xml (END - exit=256) [01:30:15] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (START) [01:30:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:30:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:35:29] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=scruffywiki (END - exit=2) [01:35:30] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=scruffywiki --active --update (END - exit=0) [01:35:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:35:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:29] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [01:46:10] [02mediawiki-repos] 07labster opened pull request 03#10: Update Extension:Purge to the supported repo (T11834) - 13https://github.com/miraheze/mediawiki-repos/pull/10 [01:47:36] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 40% [01:51:48] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:53:34] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 39% [02:04:29] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 40% [02:07:22] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [02:08:28] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 39% [02:20:24] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 40% [02:21:38] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [02:24:34] [02mediawiki-repos] 07Universal-Omega closed pull request 03#10: Update Extension:Purge to the supported repo (T11834) - 13https://github.com/miraheze/mediawiki-repos/pull/10 [02:24:36] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/24068be924c0...75215eb0388c [02:24:38] [02miraheze/mediawiki-repos] 07labster 0375215eb - Update Extension:Purge to the supported repo (T11834) (#10) [02:30:21] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 39% [02:39:22] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [02:40:17] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 40% [02:42:17] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 39% [02:46:52] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=scruffywiki --search-recursively /home/universalomega/scruffywiki/images (START) [02:46:53] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=scruffywiki --search-recursively /home/universalomega/scruffywiki/images (END - exit=0) [02:46:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:46:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:19] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=scruffywiki --search-recursively /home/universalomega/scruffywiki/images (START) [02:47:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:25] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=scruffywiki --search-recursively /home/universalomega/scruffywiki/images (END - exit=0) [02:47:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:38] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=scruffywiki --search-recursively /home/universalomega/scruffywiki/images --overwrite (START) [02:47:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:57] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=scruffywiki --search-recursively /home/universalomega/scruffywiki/images --overwrite (END - exit=0) [02:47:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:48:45] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=scruffywiki --search-recursively /home/universalomega/scruffywiki/images --overwrite (START) [02:48:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:49:08] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=scruffywiki --search-recursively /home/universalomega/scruffywiki/images --overwrite (END - exit=0) [02:49:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:50:55] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=scruffywiki --search-recursively /home/universalomega/scruffywiki/images --overwrite (START) [02:50:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:51:18] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=scruffywiki --search-recursively /home/universalomega/scruffywiki/images --overwrite (END - exit=0) [02:51:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:05:09] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 40% [03:05:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:09:14] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [03:33:03] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 39% [03:37:01] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 40% [03:39:01] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 39% [03:39:46] !log [universalomega@mwtask181] starting deploy of {'force': True, 'versions': '1.41', 'upgrade_extensions': 'Purge'} to all [03:39:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:39:56] !log [universalomega@mwtask181] finished deploy of {'force': True, 'versions': '1.41', 'upgrade_extensions': 'Purge'} to all - SUCCESS in 9s [03:39:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:45:07] @.labster ^ deployed the change to new repo [03:45:41] And it works again, thanks! [03:47:27] <.labster> 👍 [05:43:31] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 40% [06:04:11] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 120.83, 58.09, 23.37 [06:04:18] PROBLEM - db181 SSH on db181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:05:02] PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:05:09] PROBLEM - db181 Puppet on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:05:09] PROBLEM - db181 APT on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:06:14] RECOVERY - db181 SSH on db181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [06:06:58] RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.054 seconds response time. miraheze.org returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [06:07:05] RECOVERY - db181 APT on db181 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [06:07:05] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:11:31] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=lazulumwiki --no-updates --username-prefix=wikia:lazulum /home/reception/ImportDump/lazulumwiki-20240210035834.xml (START) [06:11:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:13:43] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=lazulumwiki --no-updates --username-prefix=wikia:lazulum /home/reception/ImportDump/lazulumwiki-20240210035834.xml (END - exit=0) [06:13:44] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=lazulumwiki (START) [06:13:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:13:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:14:03] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=lazulumwiki (END - exit=0) [06:14:04] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=lazulumwiki --active --update (END - exit=0) [06:14:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:14:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:26:05] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.31, 1.88, 11.22 [06:28:05] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.18, 1.30, 9.87 [06:29:16] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 39% [08:44:36] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 40% [09:06:36] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 39% [09:09:33] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [09:13:13] PROBLEM - cp36 Current Load on cp36 is CRITICAL: LOAD CRITICAL - total load average: 13.50, 7.85, 3.96 [09:25:46] PROBLEM - cp37 Current Load on cp37 is CRITICAL: LOAD CRITICAL - total load average: 10.98, 7.47, 4.05 [09:29:46] PROBLEM - cp37 Current Load on cp37 is WARNING: LOAD WARNING - total load average: 4.58, 6.99, 4.74 [09:31:46] RECOVERY - cp37 Current Load on cp37 is OK: LOAD OK - total load average: 1.91, 5.17, 4.34 [09:33:13] PROBLEM - cp36 Current Load on cp36 is WARNING: LOAD WARNING - total load average: 2.10, 5.95, 7.32 [09:35:13] RECOVERY - cp36 Current Load on cp36 is OK: LOAD OK - total load average: 1.63, 4.48, 6.61 [09:57:44] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:09:17] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [10:12:28] PROBLEM - wiki.yuanpi.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.yuanpi.eu.org' expires in 7 day(s) (Sun 18 Feb 2024 10:10:37 AM GMT +0000). [10:27:33] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [11:28:21] PROBLEM - zhacg.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'zhacg.wiki' expires in 7 day(s) (Sun 18 Feb 2024 11:16:03 AM GMT +0000). [11:40:49] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [11:52:47] I’ll help 10 people to earn $20k within 72 hours but you will pay me 10% of your profit when you receive it. Note only interested people should apply, drop a message let’s get started by asking (HOW) [12:05:45] !log [alex@test151] starting deploy of {'l10n': True, 'versions': '1.41'} to test151 [12:05:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:06:43] The requestssl i18n on mirabeta is way outdated, that's why I'm running that [12:10:12] !log [alex@test151] finished deploy of {'l10n': True, 'versions': '1.41'} to test151 - SUCCESS in 267s [12:10:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:10:26] nice [12:10:40] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [14:49:04] PROBLEM - miraheze.gq - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.gq could not be found [14:55:25] PROBLEM - miraheze.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.tk could not be found [14:59:15] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [15:20:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:29:04] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [15:35:04] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9694MiB (10% inode=98%); [15:35:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:01:00] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [16:30:57] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:39:16] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9710MiB (10% inode=98%); [16:56:36] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 40% [17:00:48] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [17:25:04] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 146.63 ms [17:27:05] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 143.70 ms [17:27:38] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/9fbbbfc49937...f594d8ed571e [17:27:39] [02miraheze/dns] 07paladox 03f594d8e - Delete zones/miraheze.gq [17:28:14] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/3c9848bbadf9 [17:28:16] [02miraheze/ssl] 07paladox 033c9848b - Remove miraheze.gq [17:28:17] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [17:28:18] [02ssl] 07paladox opened pull request 03#774: Remove miraheze.gq - 13https://github.com/miraheze/ssl/pull/774 [17:28:31] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/3c9848bbadf9...4d35f2b4d1b1 [17:28:34] [02miraheze/ssl] 07paladox 034d35f2b - Delete certificates/miraheze.gq.crt [17:28:35] [02ssl] 07paladox synchronize pull request 03#774: Remove miraheze.gq - 13https://github.com/miraheze/ssl/pull/774 [17:28:36] [02ssl] 07paladox closed pull request 03#774: Remove miraheze.gq - 13https://github.com/miraheze/ssl/pull/774 [17:28:39] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/8c41f2c48df9...029fa3c83a39 [17:28:42] [02miraheze/ssl] 07paladox 03029fa3c - Remove miraheze.gq (#774) [17:28:44] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [17:28:46] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [17:28:47] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [17:29:03] !log [paladox@test151] starting deploy of {'world': True, 'l10n': True, 'versions': ['1.41', '1.42']} to test151 [17:29:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:30:47] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/d1b65c718aa5 [17:30:49] [02miraheze/ssl] 07paladox 03d1b65c7 - Remove miraheze.tk [17:30:52] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [17:30:55] [02ssl] 07paladox opened pull request 03#775: Remove miraheze.tk - 13https://github.com/miraheze/ssl/pull/775 [17:31:04] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/d1b65c718aa5...97b3f880abab [17:31:07] [02miraheze/ssl] 07paladox 0397b3f88 - Delete certificates/miraheze.tk.crt [17:31:09] [02ssl] 07paladox synchronize pull request 03#775: Remove miraheze.tk - 13https://github.com/miraheze/ssl/pull/775 [17:31:12] [02ssl] 07paladox closed pull request 03#775: Remove miraheze.tk - 13https://github.com/miraheze/ssl/pull/775 [17:31:14] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [17:31:16] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/029fa3c83a39...5b3fbc98f20f [17:31:17] [02miraheze/ssl] 07paladox 035b3fbc9 - Remove miraheze.tk (#775) [17:31:20] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [17:32:04] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/f594d8ed571e...ec2050d91bfb [17:32:06] [02miraheze/dns] 07paladox 03ec2050d - Delete zones/miraheze.tk [17:35:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5b3fbc98f20f...117f4a26f88d [17:35:40] [02miraheze/ssl] 07MirahezeSSLBot 03117f4a2 - Bot: Update SSL cert for miraheze.wiki [17:36:37] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/117f4a26f88d...0e134c0c1147 [17:36:39] [02miraheze/ssl] 07paladox 030e134c0 - Enable miraheze.wiki event [17:38:55] !log [paladox@test151] finished deploy of {'world': True, 'l10n': True, 'versions': ['1.41', '1.42']} to test151 - SUCCESS in 592s [17:38:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:54:19] RECOVERY - miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Fri 10 May 2024 04:35:32 PM GMT +0000. [18:12:59] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [18:21:49] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 40% [18:24:33] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03toggle-all [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/3cf730095e5d [18:24:36] [02miraheze/ManageWiki] 07Universal-Omega 033cf7300 - ToggleExtension: support running on all wikis at once [18:24:38] [02ManageWiki] 07Universal-Omega created branch 03toggle-all - 13https://github.com/miraheze/ManageWiki [18:24:39] [02ManageWiki] 07Universal-Omega opened pull request 03#451: ToggleExtension: support running on all wikis at once - 13https://github.com/miraheze/ManageWiki/pull/451 [18:26:01] [02miraheze/ManageWiki] 07github-actions[bot] pushed 031 commit to 03toggle-all [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/3cf730095e5d...102bc89e846e [18:26:04] [02miraheze/ManageWiki] 07github-actions 03102bc89 - CI: lint code to MediaWiki standards [18:26:05] [02ManageWiki] 07github-actions[bot] synchronize pull request 03#451: ToggleExtension: support running on all wikis at once - 13https://github.com/miraheze/ManageWiki/pull/451 [18:28:46] miraheze/ManageWiki - Universal-Omega the build passed. [18:30:32] [02ManageWiki] 07Universal-Omega reviewed pull request 03#451 commit - 13https://github.com/miraheze/ManageWiki/pull/451#discussion_r1485213032 [18:42:01] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [19:12:59] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [19:33:06] [02ManageWiki] 07redbluegreenhat reviewed pull request 03#451 commit - 13https://github.com/miraheze/ManageWiki/pull/451#discussion_r1485223247 [19:34:38] [02ManageWiki] 07Universal-Omega reviewed pull request 03#451 commit - 13https://github.com/miraheze/ManageWiki/pull/451#discussion_r1485223875 [19:36:29] [02ManageWiki] 07redbluegreenhat reviewed pull request 03#451 commit - 13https://github.com/miraheze/ManageWiki/pull/451#discussion_r1485224601 [19:38:36] [02ManageWiki] 07paladox commented on pull request 03#451: ToggleExtension: support running on all wikis at once - 13https://github.com/miraheze/ManageWiki/pull/451#issuecomment-1937104012 [19:40:59] [02ManageWiki] 07redbluegreenhat reviewed pull request 03#451 commit - 13https://github.com/miraheze/ManageWiki/pull/451#discussion_r1485226506 [19:43:06] [02ManageWiki] 07Universal-Omega commented on pull request 03#451: ToggleExtension: support running on all wikis at once - 13https://github.com/miraheze/ManageWiki/pull/451#issuecomment-1937104802 [19:43:58] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=polcompballwiki --no-updates /home/universalomega/pcbwiki.xml (START) [19:44:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:26] [02ManageWiki] 07Universal-Omega synchronize pull request 03#451: ToggleExtension: support running on all wikis at once - 13https://github.com/miraheze/ManageWiki/pull/451 [20:00:27] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03toggle-all [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/102bc89e846e...9e465a3de1cf [20:00:30] [02miraheze/ManageWiki] 07Universal-Omega 039e465a3 - Use --confirm [20:01:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-14 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/290818745535 [20:01:25] [02miraheze/puppet] 07paladox 032908187 - Fix "monitoring::hosts" for cp27/28/41/51 [20:01:26] [02puppet] 07paladox created branch 03paladox-patch-14 - 13https://github.com/miraheze/puppet [20:01:29] [02puppet] 07paladox opened pull request 03#3751: Fix "monitoring::hosts" for cp27/28/41/51 - 13https://github.com/miraheze/puppet/pull/3751 [20:01:33] [02ManageWiki] 07Universal-Omega synchronize pull request 03#451: ToggleExtension: support running on all wikis at once - 13https://github.com/miraheze/ManageWiki/pull/451 [20:01:34] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03toggle-all [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/9e465a3de1cf...46fcbbb3d9cc [20:01:36] [02miraheze/ManageWiki] 07Universal-Omega 0346fcbbb - Update fatalError message [20:01:40] [02puppet] 07paladox closed pull request 03#3751: Fix "monitoring::hosts" for cp27/28/41/51 - 13https://github.com/miraheze/puppet/pull/3751 [20:01:41] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-14 [20:01:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/700db25c32bb...dc68029acbea [20:01:47] [02miraheze/puppet] 07paladox 03dc68029 - Fix "monitoring::hosts" for cp27/28/41/51 (#3751) [20:01:49] [02puppet] 07paladox deleted branch 03paladox-patch-14 - 13https://github.com/miraheze/puppet [20:05:17] RECOVERY - Host cp28 is UP: PING OK - Packet loss = 0%, RTA = 157.62 ms [20:05:27] RECOVERY - cp28 SSH on cp28 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [20:05:29] RECOVERY - cp28 Puppet on cp28 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:05:36] RECOVERY - cp28 ferm_active on cp28 is OK: OK ferm input default policy is set [20:05:46] RECOVERY - cp28 Nginx Backend for phorge171 on cp28 is OK: TCP OK - 0.000 second response time on localhost port 8202 [20:05:58] RECOVERY - cp28 Nginx Backend for mw161 on cp28 is OK: TCP OK - 0.000 second response time on localhost port 8115 [20:06:04] RECOVERY - cp28 Varnish Backends on cp28 is OK: All 18 backends are healthy [20:06:11] miraheze/ManageWiki - Universal-Omega the build passed. [20:06:17] RECOVERY - cp28 Nginx Backend for mw172 on cp28 is OK: TCP OK - 0.000 second response time on localhost port 8118 [20:06:22] RECOVERY - cp28 PowerDNS Recursor on cp28 is OK: DNS OK: 0.029 seconds response time. miraheze.org returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [20:06:27] RECOVERY - cp28 Nginx Backend for swiftproxy161 on cp28 is OK: TCP OK - 0.000 second response time on localhost port 8206 [20:06:27] RECOVERY - cp28 Current Load on cp28 is OK: LOAD OK - total load average: 0.00, 0.05, 0.01 [20:06:32] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.11 ms [20:06:42] RECOVERY - cp28 Nginx Backend for mw151 on cp28 is OK: TCP OK - 0.000 second response time on localhost port 8113 [20:06:47] RECOVERY - cp28 Nginx Backend for test151 on cp28 is OK: TCP OK - 0.000 second response time on localhost port 8181 [20:06:47] RECOVERY - cp28 Nginx Backend for swiftproxy171 on cp28 is OK: TCP OK - 0.000 second response time on localhost port 8207 [20:06:47] RECOVERY - cp28 Nginx Backend for mw171 on cp28 is OK: TCP OK - 0.000 second response time on localhost port 8117 [20:06:47] PROBLEM - cp28 Disk Space on cp28 is WARNING: DISK WARNING - free space: / 30312MiB (6% inode=99%); [20:07:12] RECOVERY - cp28 HTTP 4xx/5xx ERROR Rate on cp28 is OK: OK - NGINX Error Rate is 0% [20:07:12] RECOVERY - cp28 Nginx Backend for mwtask181 on cp28 is OK: TCP OK - 0.000 second response time on localhost port 8160 [20:07:12] RECOVERY - cp28 Nginx Backend for mw162 on cp28 is OK: TCP OK - 0.000 second response time on localhost port 8116 [20:08:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0e134c0c1147...8fed95bc8945 [20:08:19] [02miraheze/ssl] 07MirahezeSSLBot 038fed95b - Bot: Update SSL cert for wiki.andreijiroh.uk.eu.org [20:11:01] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [20:14:44] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 143.33 ms [20:15:36] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/ec2050d91bfb...a05a24d1f579 [20:15:37] [02miraheze/dns] 07Universal-Omega 03a05a24d - Add polcompball.wikitide.org to Miraheze [20:16:46] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 143.33 ms [20:19:45] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=polcompballwiki --search-recursively /home/universalomega/pcbwiki (START) [20:19:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:56] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 157.51 ms [20:24:59] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.10 ms [20:25:16] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=polcompballwiki --search-recursively /home/universalomega/pcbwiki (END - exit=2) [20:25:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:29:56] RECOVERY - wiki.andreijiroh.uk.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.andreijiroh.uk.eu.org' will expire on Fri 10 May 2024 07:08:08 PM GMT +0000. [20:30:59] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8fed95bc8945...0f25379fd15a [20:31:02] [02miraheze/ssl] 07Universal-Omega 030f25379 - Add polcompball.wikitide.org redirect [20:57:01] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 206.14 ms [20:59:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.01 ms [21:09:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.38 ms [21:11:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.01 ms [21:17:01] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 143.32 ms [21:18:25] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/a05a24d1f579...9fab56b82507 [21:18:26] [02miraheze/dns] 07Universal-Omega 039fab56b - Point www.farthestfrontier.wiki to Miraheze [21:19:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 143.26 ms [21:19:54] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/9fab56b82507...2372c3f92b1e [21:19:55] [02miraheze/dns] 07Universal-Omega 032372c3f - Point www.rosettacode.org to Miraheze [21:20:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:21:27] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03redirects [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/b6e18b55abc5 [21:21:28] [02miraheze/ssl] 07Universal-Omega 03b6e18b5 - Add redirects [21:21:30] [02ssl] 07Universal-Omega created branch 03redirects - 13https://github.com/miraheze/ssl [21:21:32] [02ssl] 07Universal-Omega opened pull request 03#776: Add redirects - 13https://github.com/miraheze/ssl/pull/776 [21:29:03] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/2372c3f92b1e...70cda1134111 [21:29:05] [02miraheze/dns] 07Universal-Omega 0370cda11 - Revert for now [21:29:27] CosmicAlpha: see tech channel [21:29:29] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/70cda1134111...6e8d52a5a107 [21:29:32] [02miraheze/dns] 07Universal-Omega 036e8d52a - Revert for now [21:29:37] * RhinosF1 was paged by !_sre [21:45:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 152.74 ms [21:47:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 157.54 ms [21:55:01] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 157.60 ms [21:57:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 157.51 ms [22:03:22] PROBLEM - cp36 Disk Space on cp36 is CRITICAL: DISK CRITICAL - free space: / 5307MiB (5% inode=98%); [22:21:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 157.46 ms [22:23:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 158.39 ms [22:40:12] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [22:41:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/68f9501d0103...fce2453fc45b [22:41:18] [02miraheze/mw-config] 07Universal-Omega 03fce2453 - Fix oldid without diff [22:42:15] miraheze/mw-config - Universal-Omega the build passed. [22:52:54] PROBLEM - wiki.col6.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.col6.de' expires in 15 day(s) (Mon 26 Feb 2024 10:49:53 PM GMT +0000). [22:53:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0f25379fd15a...ec2d269a35b2 [22:53:10] [02miraheze/ssl] 07MirahezeSSLBot 03ec2d269 - Bot: Update SSL cert for wiki.col6.de [22:53:49] !log [@test151] starting deploy of {'config': True} to test151 [22:53:50] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [22:53:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:57:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 143.35 ms [22:57:34] !log [@mwtask181] starting deploy of {'config': True} to all [22:57:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:57:44] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [22:57:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:59:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.59 ms [23:14:16] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 5295MiB (5% inode=98%); [23:15:52]