[00:04:32] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21465MiB (24% inode=98%); [00:05:04] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 19434MiB (20% inode=98%); [00:05:08] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21443MiB (24% inode=98%); [00:17:16] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [00:35:06] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=ehswiki ehswiki_xml_09fd901a24119c670c3f.xml --no-updates (START) [00:35:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:35:32] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=ehswiki ehswiki_xml_09fd901a24119c670c3f.xml --no-updates (END - exit=0) [00:35:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:37:05] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=ehswiki . --search-recursively (START) [00:37:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:37:32] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=ehswiki . --search-recursively (END - exit=0) [00:37:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:41:51] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=ehswiki (START) [00:41:55] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=ehswiki (END - exit=0) [00:41:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:42:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:42] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=growevertreewiki growevertreewiki_xml_93c4344b9510b593db10.xml.gz --no-updates (START) [01:21:43] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=growevertreewiki growevertreewiki_xml_93c4344b9510b593db10.xml.gz --no-updates (END - exit=256) [01:21:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:55] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=growevertreewiki growevertreewiki_xml_93c4344b9510b593db10.xml --no-updates (START) [01:21:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:22:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:00] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=wodehousewiki wodehousewiki_xml_1ecf2a7229ea88cad02d.xml --no-updates (START) [01:27:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:23] PROBLEM - www.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Fri 03 May 2024 01:21:33 AM GMT +0000). [01:27:36] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=wodehousewiki wodehousewiki_xml_1ecf2a7229ea88cad02d.xml --no-updates (END - exit=0) [01:27:36] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e5961bd96dc4...9fc8021fb033 [01:27:38] [02miraheze/ssl] 07WikiTideSSLBot 039fc8021 - Bot: Update SSL cert for www.thesimswiki.com [01:27:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:31:00] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=growevertreewiki growevertreewiki_xml_93c4344b9510b593db10.xml --no-updates (END - exit=0) [01:31:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:23] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=singaporeshuttlebuseswiki singaporeshuttlebuseswiki_xml_6e6361e2f22e5aa830aa.xml --no-updates (START) [01:34:28] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=singaporeshuttlebuseswiki singaporeshuttlebuseswiki_xml_6e6361e2f22e5aa830aa.xml --no-updates (END - exit=0) [01:34:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:36:16] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=singaporeshuttlebuseswiki . --search-recursively (START) [01:36:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:36:33] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=singaporeshuttlebuseswiki . --search-recursively (END - exit=0) [01:36:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:36:48] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=singaporeshuttlebuseswiki (START) [01:36:50] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=singaporeshuttlebuseswiki (END - exit=0) [01:36:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:11] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=growevertreewiki . --search-recursively (START) [01:37:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:25] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=wodehousewiki (START) [01:37:26] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:37:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:38] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=wodehousewiki (END - exit=0) [01:37:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:38:26] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Fri 03 May 2024 01:21:33 AM GMT +0000). [01:39:53] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=theneonchronicleswiki theneonchronicleswiki_xml_ae44d96dc9552b9f5c47.xml --no-updates (START) [01:39:54] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=theneonchronicleswiki theneonchronicleswiki_xml_ae44d96dc9552b9f5c47.xml --no-updates (END - exit=65280) [01:40:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:40:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:40:16] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=theneonchronicleswikiwiki theneonchronicleswiki_xml_ae44d96dc9552b9f5c47.xml --no-updates (START) [01:40:17] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=theneonchronicleswikiwiki theneonchronicleswiki_xml_ae44d96dc9552b9f5c47.xml --no-updates (END - exit=65280) [01:40:19] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.37 ms [01:40:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:40:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:42:00] PROBLEM - wiki.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Fri 03 May 2024 01:21:33 AM GMT +0000). [01:42:17] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 163.84 ms [01:49:15] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=theneonchronicleswikiwiki theneonchronicleswiki_xml_ae44d96dc9552b9f5c47.xml --no-updates (START) [01:49:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:49:47] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=theneonchronicleswikiwiki theneonchronicleswiki_xml_ae44d96dc9552b9f5c47.xml --no-updates (END - exit=0) [01:49:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:54:52] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=theneonchronicleswiki . --search-recursively (START) [01:54:53] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=theneonchronicleswiki . --search-recursively (END - exit=65280) [01:54:59] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=theneonchronicleswikiwiki . --search-recursively (START) [01:55:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:55:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:55:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:57:17] RECOVERY - www.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Tue 16 Jul 2024 12:27:30 AM GMT +0000. [01:57:45] PROBLEM - rockrevolution.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rockrevolution.wiki' expires in 15 day(s) (Fri 03 May 2024 01:54:39 AM GMT +0000). [01:57:57] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9fc8021fb033...df80c279906c [01:57:58] [02miraheze/ssl] 07WikiTideSSLBot 03df80c27 - Bot: Update SSL cert for rockrevolution.wiki [02:01:50] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=theneonchronicleswikiwiki . --search-recursively (END - exit=0) [02:01:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:06:24] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=theneonchronicleswikiwiki (START) [02:06:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:06:33] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=theneonchronicleswikiwiki (END - exit=0) [02:06:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:07:21] RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Tue 16 Jul 2024 12:27:30 AM GMT +0000. [02:11:23] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/c8189630107b...82c1d5aef1d2 [02:11:24] [02miraheze/dns] 07AgentIsai 0382c1d5a - Add more redirects [02:11:29] RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Tue 16 Jul 2024 12:27:30 AM GMT +0000. [02:12:34] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=mirapediawiki mirapediawiki_xml_365879aff73076fce798.xml --no-updates (START) [02:12:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:12:48] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=mirapediawiki mirapediawiki_xml_365879aff73076fce798.xml --no-updates (END - exit=0) [02:12:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:14:12] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=mirapediawiki . --search-recursively (START) [02:14:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:26] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=mirapediawiki . --search-recursively (END - exit=0) [02:15:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:16:19] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=mirapediawiki (START) [02:16:23] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=mirapediawiki (END - exit=0) [02:16:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:16:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:16:42] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/82c1d5aef1d2...a29a7785e977 [02:16:43] [02miraheze/dns] 07AgentIsai 03a29a778 - Add mirapedia.wt [02:17:39] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/a29a7785e977...8a64cdd8e698 [02:17:42] [02miraheze/dns] 07AgentIsai 038a64cdd - Remove duplicate [02:20:26] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/restoreManageWikiBackup.php --wiki=growevertreewiki --filename growevertreewiki_managewiki_backup_0af1653a8d6b566ba94d.json (END - exit=0) [02:20:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:26:30] RECOVERY - rockrevolution.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rockrevolution.wiki' will expire on Tue 16 Jul 2024 12:57:51 AM GMT +0000. [02:33:17] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=d20pembrokewiki . --search-recursively (START) [02:33:24] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=d20pembrokewiki . --search-recursively (END - exit=0) [02:33:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:33:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:58] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=kilnwiki . --search-recursively (START) [02:48:05] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=kilnwiki . --search-recursively (END - exit=0) [02:48:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:48:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:51:18] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=thegreatwarwiki . --search-recursively (START) [02:51:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:29] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=thegreatwarwiki . --search-recursively (END - exit=0) [03:00:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:24:38] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 174.02 ms [03:26:36] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.93 ms [03:35:18] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4betawiki . --search-recursively (START) [03:35:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:35:36] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4betawiki . --search-recursively (END - exit=0) [03:35:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:28] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:40:59] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/restoreManageWikiBackup.php --wiki=thewhiteroomwiki --filename thewhiteroomwiki_managewiki_backup_25da651336f5dbefdf08.json (END - exit=0) [03:41:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:41:41] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=thewhiteroomwiki . --search-recursively (START) [03:41:41] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 222.95 ms [03:41:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:43:31] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=growevertreewiki . --search-recursively (END - exit=0) [03:43:39] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.03 ms [03:43:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:47:43] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 177.62 ms [03:49:41] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.15 ms [04:16:17] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=growevertreewiki (START) [04:16:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:17:55] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=growevertreewiki (END - exit=0) [04:18:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:19:53] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=growevertreewiki --update (END - exit=0) [04:19:54] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/updateSpecialPages.php --wiki=growevertreewiki (END - exit=0) [04:20:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:20:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:01:09] PROBLEM - holocron.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'holocron.net' expires in 15 day(s) (Fri 03 May 2024 05:58:11 AM GMT +0000). [06:01:20] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/df80c279906c...dc0bd216f5d1 [06:01:21] [02miraheze/ssl] 07WikiTideSSLBot 03dc0bd21 - Bot: Update SSL cert for holocron.net [06:30:17] RECOVERY - holocron.net - LetsEncrypt on sslhost is OK: OK - Certificate 'holocron.net' will expire on Tue 16 Jul 2024 05:01:14 AM GMT +0000. [06:51:46] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [07:12:28] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:20:33] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [08:18:05] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:21:46] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [08:34:23] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.22 ms [08:36:22] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.49 ms [08:42:27] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [08:47:41] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:50:33] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [09:48:53] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.66 ms [09:50:51] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 164.56 ms [10:40:40] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.73 ms [10:44:38] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.31 ms [11:26:30] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 174.44 ms [11:28:28] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 180.74 ms [11:34:19] PROBLEM - worldtobuild.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for worldtobuild.wiki could not be found [11:41:04] PROBLEM - worldtobuild.wiki - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [12:05:31] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 173.44 ms [12:07:29] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 179.29 ms [12:48:05] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [13:00:39] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query qadrishattari.xyz. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:01:35] PROBLEM - wiki.buryland.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.buryland.net All nameservers failed to answer the query. [13:10:54] PROBLEM - wiki.buryland.net - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [13:29:17] RECOVERY - wiki.qadrishattari.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.qadrishattari.xyz reverse DNS resolves to cp36.wikitide.net - CNAME OK [13:31:05] RECOVERY - wiki.buryland.net - reverse DNS on sslhost is OK: SSL OK - wiki.buryland.net reverse DNS resolves to cp37.wikitide.net - CNAME OK [13:40:38] RECOVERY - wiki.buryland.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.buryland.net' will expire on Thu 20 Jun 2024 01:58:22 AM GMT +0000. [14:13:23] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.98, 6.71, 5.72 [14:15:19] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.79, 6.58, 5.78 [14:16:51] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [14:23:05] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.57, 7.19, 6.34 [14:25:01] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.23, 6.07, 6.02 [15:57:48] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [16:07:05] [02dns] 07Universal-Omega created branch 03remove-cpweb2 - 13https://github.com/miraheze/dns [16:07:06] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove-cpweb2 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/105812bd6dbc [16:07:09] [02miraheze/dns] 07Universal-Omega 03105812b - Remove cpweb2 [16:07:10] [02dns] 07Universal-Omega opened pull request 03#509: Remove cpweb2 - 13https://github.com/miraheze/dns/pull/509 [16:08:31] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove-cpweb2 [+0/-0/±1] 13https://github.com/miraheze/dns/compare/105812bd6dbc...e5db87123369 [16:08:34] [02miraheze/dns] 07Universal-Omega 03e5db871 - Use geoip!cp [16:08:35] [02dns] 07Universal-Omega synchronize pull request 03#509: Remove cpweb2 - 13https://github.com/miraheze/dns/pull/509 [16:09:04] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove-cpweb2 [+0/-0/±1] 13https://github.com/miraheze/dns/compare/e5db87123369...71546f2d4502 [16:09:05] [02miraheze/dns] 07Universal-Omega 0371546f2 - Use geoip!cp [16:09:06] [02dns] 07Universal-Omega synchronize pull request 03#509: Remove cpweb2 - 13https://github.com/miraheze/dns/pull/509 [16:11:14] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove-cpweb2 [+0/-0/±1] 13https://github.com/miraheze/dns/compare/71546f2d4502...191abcf4ebc6 [16:11:15] [02miraheze/dns] 07Universal-Omega 03191abcf - Use geoip!cp [16:11:17] [02dns] 07Universal-Omega synchronize pull request 03#509: Remove cpweb2 - 13https://github.com/miraheze/dns/pull/509 [16:11:56] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.59, 7.33, 6.47 [16:17:04] PROBLEM - cp41 Disk Space on cp41 is WARNING: DISK WARNING - free space: / 10225MiB (10% inode=98%); [16:19:40] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.47, 6.39, 6.47 [16:22:02] [02dns] 07Universal-Omega closed pull request 03#509: Remove cpweb2 - 13https://github.com/miraheze/dns/pull/509 [16:22:04] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/dns/compare/8a64cdd8e698...b9e6c766c0b1 [16:22:05] [02miraheze/dns] 07Universal-Omega 03b9e6c76 - Remove cpweb2 (#509) [16:22:07] [02dns] 07Universal-Omega deleted branch 03remove-cpweb2 - 13https://github.com/miraheze/dns [16:22:09] [02miraheze/dns] 07Universal-Omega deleted branch 03remove-cpweb2 [16:27:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:29:25] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp37.wikitide.net - CNAME OK [16:32:56] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:33:14] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.81, 6.89, 6.34 [16:35:10] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.52, 6.71, 6.35 [16:39:33] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03update-cp [+0/-0/±1] 13https://github.com/miraheze/dns/commit/c60eed4a10f5 [16:39:34] [02miraheze/dns] 07Universal-Omega 03c60eed4 - Update cp config [16:39:36] [02dns] 07Universal-Omega created branch 03update-cp - 13https://github.com/miraheze/dns [16:39:37] [02dns] 07Universal-Omega opened pull request 03#510: Update cp config - 13https://github.com/miraheze/dns/pull/510 [16:40:42] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03update-cp [+0/-0/±1] 13https://github.com/miraheze/dns/compare/c60eed4a10f5...4823521e17d1 [16:40:43] [02miraheze/dns] 07Universal-Omega 034823521 - Update [16:40:44] [02dns] 07Universal-Omega synchronize pull request 03#510: Update cp config - 13https://github.com/miraheze/dns/pull/510 [16:41:58] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.88, 7.77, 6.87 [16:44:04] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:47:46] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.16, 7.65, 7.23 [16:51:38] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 7.70, 8.02, 7.49 [16:53:34] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.48, 7.37, 7.31 [16:56:13] [02dns] 07Universal-Omega closed pull request 03#510: Update cp config - 13https://github.com/miraheze/dns/pull/510 [16:56:14] [02miraheze/dns] 07Universal-Omega deleted branch 03update-cp [16:56:17] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/b9e6c766c0b1...b9bd75abfa44 [16:56:18] [02miraheze/dns] 07Universal-Omega 03b9bd75a - Update cp config (#510) [16:56:19] [02dns] 07Universal-Omega deleted branch 03update-cp - 13https://github.com/miraheze/dns [16:57:25] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.56, 7.57, 7.37 [16:57:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:58:10] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:59:04] [02CreateWiki] 07Reception123 closed pull request 03#508: no dismiss for more details; bump priorities - 13https://github.com/miraheze/CreateWiki/pull/508 [16:59:07] [02miraheze/CreateWiki] 07Reception123 deleted branch 03Reception123-patch-1 [16:59:08] [02miraheze/CreateWiki] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/3083633da9a0...851cb6a310a1 [16:59:11] [02miraheze/CreateWiki] 07Reception123 03851cb6a - no dismiss for more details; bump priorities (#508) [16:59:14] [02CreateWiki] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/CreateWiki [16:59:21] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.82, 7.06, 7.20 [16:59:24] !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [16:59:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:59:35] !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [16:59:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:00:19] !log [reception@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all [17:00:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:00:30] !log [reception@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 11s [17:00:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:01:17] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.30, 6.04, 6.80 [17:01:46] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 51% [17:01:52] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 53% [17:05:46] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 31% [17:05:52] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 31% [17:07:20] miraheze/CreateWiki - Reception123 the build passed. [17:09:05] [02puppet] 07Universal-Omega closed pull request 03#3834: Add a database-admins group - 13https://github.com/miraheze/puppet/pull/3834 [17:09:07] [02miraheze/puppet] 07Universal-Omega deleted branch 03database-admins [17:09:09] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/puppet/compare/48c884dc7cfc...b17e9215f3ec [17:09:10] [02miraheze/puppet] 07Universal-Omega 03b17e921 - Add a database-admins group (#3834) [17:09:11] [02puppet] 07Universal-Omega deleted branch 03database-admins - 13https://github.com/miraheze/puppet [17:09:47] [02mw-config] 07Universal-Omega closed pull request 03#5531: thumb_handler: redirect default server to primary domain - 13https://github.com/miraheze/mw-config/pull/5531 [17:09:50] [02miraheze/mw-config] 07Universal-Omega deleted branch 03thumb_handler-redirect [17:09:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/333281590b42...2172a68e36e4 [17:09:56] [02miraheze/mw-config] 07Universal-Omega 032172a68 - thumb_handler: redirect default server to primary domain (#5531) [17:09:58] [02mw-config] 07Universal-Omega deleted branch 03thumb_handler-redirect - 13https://github.com/miraheze/mw-config [17:10:40] miraheze/mw-config - Universal-Omega the build passed. [17:17:29] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/MirahezeMagic/rebuildVersionCache.php --wiki=metawiki (END - exit=256) [17:17:46] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/rebuildVersionCache.php --wiki=metawiki (END - exit=0) [17:17:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:18:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:23:19] !log [@test151] starting deploy of {'config': True} to test151 [17:23:20] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [17:23:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:23:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:29:15] PROBLEM - db171 Puppet on db171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:29:56] PROBLEM - db161 Puppet on db161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:32:37] !log [@mwtask181] starting deploy of {'config': True} to all [17:32:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:46] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [17:32:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:33:36] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:37:10] !log [@mwtask171] starting deploy of {'config': True} to all [17:37:19] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [17:37:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:37:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:49:11] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b17e9215f3ec...9e58bae44e58 [17:49:12] [02miraheze/puppet] 07Universal-Omega 039e58bae - T12050: add alex to database-admins and mediawiki-roots [17:49:27] PROBLEM - db151 Puppet on db151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:52:42] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9e58bae44e58...b4ea8db825ef [17:52:43] [02miraheze/puppet] 07Universal-Omega 03b4ea8db - Fix database-admins [17:53:12] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:53:34] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:55:15] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [17:55:27] RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:56:06] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:07:52] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bff89d0c0d2b...7dad1907672c [18:07:54] [02miraheze/puppet] 07Reception123 037dad190 - update rename wiki script with "good version" [18:07:56] [02puppet] 07Universal-Omega synchronize pull request 03#3814: update rename wiki script with "good version" - 13https://github.com/miraheze/puppet/pull/3814 [18:09:14] miraheze/puppet - Universal-Omega the build passed. [18:09:20] [02puppet] 07Universal-Omega deleted a comment on pull request 03#3817: rename-wiki: various - 13https://github.com/miraheze/puppet/pull/3817#issuecomment-2045955290 [18:10:37] [02puppet] 07RhinosF1 synchronize pull request 03#3817: rename-wiki: various - 13https://github.com/miraheze/puppet/pull/3817 [18:10:46] [02puppet] 07Universal-Omega synchronize pull request 03#3823: Add apt source for libssl1.1 - 13https://github.com/miraheze/puppet/pull/3823 [18:10:48] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03libssl1.1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6d0d9f4da626...e0cf798b9e30 [18:10:49] [02miraheze/puppet] 07Universal-Omega 03e0cf798 - Add comment [18:12:25] miraheze/puppet - RhinosF1 the build passed. [18:12:42] [02puppet] 07Universal-Omega closed pull request 03#3817: rename-wiki: various - 13https://github.com/miraheze/puppet/pull/3817 [18:12:45] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7dad1907672c...f2c7e9786e48 [18:12:47] [02miraheze/puppet] 07RhinosF1 03f2c7e97 - rename-wiki: various (#3817) [18:12:50] [02puppet] 07Universal-Omega synchronize pull request 03#3814: update rename wiki script with "good version" - 13https://github.com/miraheze/puppet/pull/3814 [18:14:08] miraheze/puppet - Universal-Omega the build passed. [18:17:00] [02puppet] 07Universal-Omega reviewed pull request 03#3814 commit - 13https://github.com/miraheze/puppet/pull/3814#discussion_r1569271617 [18:17:01] [02puppet] 07Universal-Omega reviewed pull request 03#3814 commit - 13https://github.com/miraheze/puppet/pull/3814#discussion_r1569272846 [18:17:04] [02puppet] 07Universal-Omega reviewed pull request 03#3814 commit - 13https://github.com/miraheze/puppet/pull/3814#discussion_r1569275110 [18:21:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2172a68e36e4...9194a05f1164 [18:21:45] [02miraheze/mw-config] 07Universal-Omega 039194a05 - Use mw-lb.wikitide.net [18:22:35] miraheze/mw-config - Universal-Omega the build passed. [18:23:11] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [18:23:49] !log [@test151] starting deploy of {'config': True} to test151 [18:23:50] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [18:23:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:32:27] !log [@mwtask181] starting deploy of {'config': True} to all [18:32:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:32:36] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [18:32:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:28] !log [@mwtask171] starting deploy of {'config': True} to all [18:37:36] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [18:37:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:40:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9194a05f1164...46fb1de2f0a6 [18:40:04] [02miraheze/mw-config] 07Universal-Omega 0346fb1de - Add getAllowedDomains functions [18:40:55] miraheze/mw-config - Universal-Omega the build passed. [18:42:50] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03cookie-domains [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/37ee9ff58f87 [18:42:51] [02miraheze/mw-config] 07Universal-Omega 0337ee9ff - Support all allowed domains in cookie settings [18:42:52] [02mw-config] 07Universal-Omega created branch 03cookie-domains - 13https://github.com/miraheze/mw-config [18:42:56] [02mw-config] 07Universal-Omega opened pull request 03#5540: Support all allowed domains in cookie settings - 13https://github.com/miraheze/mw-config/pull/5540 [18:43:53] miraheze/mw-config - Universal-Omega the build passed. [18:53:10] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:53:54] !log [@test151] starting deploy of {'config': True} to test151 [18:53:55] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [18:54:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:57] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b4ea8db825ef...1310f11bf87d [18:55:00] [02miraheze/puppet] 07Universal-Omega 031310f11 - Try permission fix [18:56:51] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1310f11bf87d...f5d243ab268b [18:56:52] [02miraheze/puppet] 07Universal-Omega 03f5d243a - Again [18:58:29] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 169.57 ms [19:00:27] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 164.38 ms [19:02:48] !log [@mwtask181] starting deploy of {'config': True} to all [19:02:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:02:58] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [19:03:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:06:59] !log [@mwtask171] starting deploy of {'config': True} to all [19:07:07] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [19:07:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:36] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f5d243ab268b...5a04d9575b76 [19:08:39] [02miraheze/puppet] 07Universal-Omega 035a04d95 - Again [19:11:56] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5a04d9575b76...33c95870c9d8 [19:11:59] [02miraheze/puppet] 07Universal-Omega 0333c9587 - Update mariadb also [19:23:05] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [19:39:51] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/b9bd75abfa44...0b1ebaf57f9a [19:39:52] [02miraheze/dns] 07Universal-Omega 030b1ebaf - Add health.wikitide.net [19:41:03] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/33c95870c9d8...799b1768a574 [19:41:04] [02miraheze/puppet] 07Universal-Omega 03799b176 - varnish: update [19:46:50] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03varnish-healthcheck [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/456a3ba00441 [19:46:51] [02miraheze/puppet] 07Universal-Omega 03456a3ba - varnish: update healthcheck host [19:46:52] [02puppet] 07Universal-Omega created branch 03varnish-healthcheck - 13https://github.com/miraheze/puppet [19:46:55] [02puppet] 07Universal-Omega opened pull request 03#3835: varnish: update healthcheck host - 13https://github.com/miraheze/puppet/pull/3835 [19:49:47] [02puppet] 07Universal-Omega closed pull request 03#3835: varnish: update healthcheck host - 13https://github.com/miraheze/puppet/pull/3835 [19:49:49] [02miraheze/puppet] 07Universal-Omega deleted branch 03varnish-healthcheck [19:49:52] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/799b1768a574...51c09dbaac7c [19:49:53] [02miraheze/puppet] 07Universal-Omega 0351c09db - varnish: update healthcheck host (#3835) [19:49:55] [02puppet] 07Universal-Omega deleted branch 03varnish-healthcheck - 13https://github.com/miraheze/puppet [19:49:58] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/0b1ebaf57f9a...c87437eb1d83 [19:50:01] [02miraheze/dns] 07Universal-Omega 03c87437e - Update healthcheck host [19:53:26] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 3 backends are down. mw172 mw181 mw182 [19:53:41] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw182 [19:55:26] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [19:55:40] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [19:59:26] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [20:00:49] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [20:05:56] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/51c09dbaac7c...863c9b8b07bf [20:05:58] [02miraheze/puppet] 07Universal-Omega 03863c9b8 - Add health.wikitide.net server [20:14:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/863c9b8b07bf...12bfe8461e3d [20:14:44] [02miraheze/puppet] 07Universal-Omega 0312bfe84 - Fix [20:16:50] RECOVERY - cp26 HTTPS on cp26 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3776 bytes in 1.164 second response time [20:17:48] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=moddedminecraftwiki ftb_gamepedia_pages_full.xml --no-updates --username-prefix=w:c:ftb (START) [20:17:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:26] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [20:24:01] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/12bfe8461e3d...e7c7218c52a7 [20:24:03] [02miraheze/puppet] 07Universal-Omega 03e7c7218 - Remove healthcheck from *.miraheze.org [20:25:03] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:27:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:54:58] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [21:37:06] PROBLEM - mw182 APT on mw182 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [21:39:07] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 65 packages available for upgrade (38 critical updates). [21:45:38] PROBLEM - mon181 APT on mon181 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [21:47:38] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (31 critical updates). [22:27:00] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:30:17] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=moddedminecraftwiki ftb_gamepedia_pages_full.xml --no-updates --username-prefix=w:c:ftb (END - exit=0) [22:30:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:33:08] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9696MiB (10% inode=98%); [22:34:32] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9706MiB (10% inode=98%); [22:56:55] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [23:01:11] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=moddedminecraftwiki (START) [23:01:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:13:40] [02miraheze/mw-config] 07Universal-Omega pushed 032 commits to 03cleanup [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/b10c7e779e21...06b04ee19a00 [23:13:42] [02miraheze/mw-config] 07Universal-Omega 0345e5a9a - MirahezeFunctions: minor cleanup [23:13:45] [02miraheze/mw-config] 07Universal-Omega 0306b04ee - Update [23:13:47] [02mw-config] 07Universal-Omega synchronize pull request 03#5535: MirahezeFunctions: minor cleanup - 13https://github.com/miraheze/mw-config/pull/5535 [23:14:07] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03cleanup [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/06b04ee19a00...e4cafc363cd9 [23:14:08] [02miraheze/mw-config] 07github-actions 03e4cafc3 - CI: lint code to MediaWiki standards [23:14:09] [02mw-config] 07github-actions[bot] synchronize pull request 03#5535: MirahezeFunctions: minor cleanup - 13https://github.com/miraheze/mw-config/pull/5535 [23:14:37] miraheze/mw-config - Universal-Omega the build passed. [23:15:24] [02mw-config] 07Universal-Omega closed pull request 03#5535: MirahezeFunctions: minor cleanup - 13https://github.com/miraheze/mw-config/pull/5535 [23:15:25] [02miraheze/mw-config] 07Universal-Omega deleted branch 03cleanup [23:15:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/46fb1de2f0a6...9ea64804fb8f [23:15:30] [02miraheze/mw-config] 07Universal-Omega 039ea6480 - MirahezeFunctions: minor cleanup (#5535) [23:15:31] [02mw-config] 07Universal-Omega deleted branch 03cleanup - 13https://github.com/miraheze/mw-config [23:16:18] miraheze/mw-config - Universal-Omega the build passed. [23:17:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03cookie-domains [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/37ee9ff58f87...e30c45802c5f [23:17:07] [02miraheze/mw-config] 07Universal-Omega 03e30c458 - Support all allowed domains in cookie settings [23:17:09] [02mw-config] 07Universal-Omega synchronize pull request 03#5540: Support all allowed domains in cookie settings - 13https://github.com/miraheze/mw-config/pull/5540 [23:18:00] miraheze/mw-config - Universal-Omega the build passed. [23:18:35] [02mw-config] 07Universal-Omega synchronize pull request 03#5540: Support all allowed domains in cookie settings - 13https://github.com/miraheze/mw-config/pull/5540 [23:18:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03cookie-domains [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e30c45802c5f...eba400daefaa [23:18:40] [02miraheze/mw-config] 07Universal-Omega 03eba400d - Fix [23:19:30] miraheze/mw-config - Universal-Omega the build passed. [23:24:39] [02mw-config] 07Universal-Omega synchronize pull request 03#5540: Support all allowed domains in cookie settings - 13https://github.com/miraheze/mw-config/pull/5540 [23:24:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03cookie-domains [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/eba400daefaa...8c0ca71be974 [23:24:42] [02miraheze/mw-config] 07Universal-Omega 038c0ca71 - Fix [23:25:32] miraheze/mw-config - Universal-Omega the build passed. [23:27:23] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [23:27:59] [02mw-config] 07Universal-Omega closed pull request 03#5540: Support all allowed domains in cookie settings - 13https://github.com/miraheze/mw-config/pull/5540 [23:28:00] [02miraheze/mw-config] 07Universal-Omega deleted branch 03cookie-domains [23:28:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9ea64804fb8f...897e816ddc8a [23:28:02] [02miraheze/mw-config] 07Universal-Omega 03897e816 - Support all allowed domains in cookie settings (#5540) [23:28:04] [02mw-config] 07Universal-Omega deleted branch 03cookie-domains - 13https://github.com/miraheze/mw-config [23:28:58] miraheze/mw-config - Universal-Omega the build passed. [23:30:36] [02miraheze/mw-config] 07Universal-Omega pushed 0331 commits to 03more-domains [+0/-0/±38] 13https://github.com/miraheze/mw-config/compare/e19a29c83190...6e50aa7bfdcf [23:30:37] [02miraheze/mw-config] 07Universal-Omega 036e50aa7 - Merge branch 'master' into more-domains [23:30:40] [02mw-config] 07Universal-Omega synchronize pull request 03#5529: Allow using orain.org and wiki.surf also - 13https://github.com/miraheze/mw-config/pull/5529 [23:31:29] miraheze/mw-config - Universal-Omega the build passed. [23:32:58] !log [@mwtask181] starting deploy of {'config': True} to all [23:33:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:33:08] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [23:33:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:37:39] !log [@mwtask171] starting deploy of {'config': True} to all [23:37:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:37:47] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [23:37:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:23] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures