[00:04:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.06, 10.16, 10.26 [00:04:32] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.11, 3.33, 2.67 [00:06:30] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.53, 2.99, 2.62 [00:07:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.75, 11.04, 9.92 [00:07:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.70, 9.58, 10.05 [00:09:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.21, 10.08, 9.72 [00:48:59] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.93, 6.77, 4.87 [00:50:56] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.36, 5.86, 4.77 [00:54:56] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:07:04] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 8038 MB (6% inode=99%); [01:07:34] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1723944247 secs [01:19:04] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7966 MB (5% inode=99%); [01:23:34] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.002523511648 secs [01:34:15] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.24, 6.09, 3.51 [01:36:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.20, 4.75, 3.36 [01:37:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.34, 9.76, 8.14 [01:38:30] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.09, 3.44, 2.80 [01:40:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.29, 3.07, 2.74 [01:41:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.75, 10.64, 8.79 [01:41:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.85, 11.00, 9.24 [01:43:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 12.00, 10.88, 9.09 [01:43:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.25, 11.00, 9.45 [01:45:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.19, 9.89, 9.23 [01:47:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.43, 9.75, 9.06 [01:52:48] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [01:58:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.15, 10.38, 9.65 [02:00:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.26, 9.83, 9.55 [03:05:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 2.51, 4.04, 3.20 [03:07:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.21, 3.26, 3.01 [03:13:34] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1655978262 secs [03:16:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.29, 10.12, 9.27 [03:18:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.13, 11.40, 9.79 [03:20:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.90, 11.12, 9.91 [03:21:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.40, 9.35, 9.40 [03:30:06] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.67, 3.75, 2.03 [03:32:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.08, 4.43, 3.24 [03:32:06] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.81, 2.61, 1.81 [03:34:01] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.12, 3.86, 3.17 [03:37:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.61, 4.15, 3.45 [03:39:48] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.62, 3.62, 3.34 [03:41:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.42, 2.91, 3.11 [03:51:34] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.09730100632 secs [03:55:04] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 8004 MB (6% inode=99%); [04:01:04] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7967 MB (5% inode=99%); [04:13:34] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset -0.108039856 secs [04:15:34] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.07643780112 secs [04:34:15] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.37, 5.94, 3.29 [04:36:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.20, 4.44, 3.06 [04:37:20] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=lewdhammerwiki --no-updates --username-prefix=wikia:lewdhammer /home/reception/ImportDump/lewdhammerwiki-20230419231250.xml (START) [04:37:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:38:01] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tttewikiwiki --no-updates --username-prefix=wikia:ttte /home/reception/ImportDump/tttewikiwiki-20230419150730.xml (START) [04:38:04] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=lewdhammerwiki --no-updates --username-prefix=wikia:lewdhammer /home/reception/ImportDump/lewdhammerwiki-20230419231250.xml (END - exit=0) [04:38:05] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=lewdhammerwiki (START) [04:38:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:38:06] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tttewikiwiki --no-updates --username-prefix=wikia:ttte /home/reception/ImportDump/tttewikiwiki-20230419150800.xml (START) [04:38:08] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tttewikiwiki --no-updates --username-prefix=wikia:ttte /home/reception/ImportDump/tttewikiwiki-20230419150904.xml (START) [04:38:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:38:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:38:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:38:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:38:29] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=lewdhammerwiki (END - exit=0) [04:38:30] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=lewdhammerwiki --active --update (END - exit=0) [04:38:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:38:37] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tttewikiwiki --no-updates --username-prefix=wikia:ttte /home/reception/ImportDump/tttewikiwiki-20230419150904.xml (END - exit=0) [04:38:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:38:38] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=tttewikiwiki (START) [04:38:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:38:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:38:53] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tttewikiwiki --no-updates --username-prefix=wikia:ttte /home/reception/ImportDump/tttewikiwiki-20230419150800.xml (END - exit=0) [04:38:54] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=tttewikiwiki (START) [04:39:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:39:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:40:46] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=tttewikiwiki (END - exit=256) [04:40:48] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=tttewikiwiki --active --update (END - exit=0) [04:40:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:40:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:41:45] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tttewikiwiki --no-updates --username-prefix=wikia:ttte /home/reception/ImportDump/tttewikiwiki-20230419150730.xml (END - exit=0) [04:41:46] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=tttewikiwiki (START) [04:41:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:41:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.76, 10.14, 8.74 [04:41:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:43:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.68, 9.81, 8.78 [04:52:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.24, 2.98, 2.15 [04:54:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.71, 2.76, 2.17 [05:00:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.42, 11.00, 9.85 [05:02:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.21, 9.58, 9.46 [05:14:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.07, 12.22, 10.40 [05:15:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.75, 11.28, 9.48 [05:16:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.42, 11.19, 10.25 [05:19:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.42, 12.82, 10.96 [05:21:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.13, 11.76, 10.81 [05:23:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.92, 11.12, 10.19 [05:25:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.74, 9.45, 10.08 [05:27:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.03, 9.39, 9.72 [06:00:15] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.67, 5.79, 3.03 [06:02:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.60, 4.52, 2.91 [06:11:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.80, 10.18, 8.68 [06:11:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 18.20, 12.87, 9.76 [06:15:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.15, 12.00, 10.17 [06:17:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.27, 9.93, 9.05 [06:23:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.26, 9.14, 9.64 [06:25:45] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=tttewikiwiki (END - exit=0) [06:25:46] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=tttewikiwiki (END - exit=0) [06:25:47] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=tttewikiwiki --active --update (END - exit=0) [06:25:48] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=tttewikiwiki --active --update (END - exit=0) [06:25:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:25:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:25:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:26:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:48:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.41, 11.14, 9.83 [06:50:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.29, 9.30, 9.31 [06:53:34] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1241832376 secs [07:00:07] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.41, 4.09, 2.24 [07:02:06] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.81, 3.24, 2.16 [07:05:34] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.09356415272 secs [07:11:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.73, 10.93, 9.10 [07:13:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.41, 10.84, 9.30 [07:15:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.68, 9.99, 9.19 [07:28:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.97, 10.94, 9.68 [07:30:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.61, 10.17, 9.53 [07:44:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.27, 10.14, 9.13 [07:44:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.01, 10.41, 9.49 [07:46:51] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.91, 10.10, 9.50 [07:46:51] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.29, 9.83, 9.14 [08:13:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.56, 10.21, 9.13 [08:15:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.64, 9.59, 9.03 [08:27:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.29, 9.92, 9.25 [08:28:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.10, 10.58, 9.75 [08:31:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.56, 9.79, 9.41 [08:32:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.92, 9.76, 9.65 [09:05:44] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:07:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.68, 9.77, 9.07 [09:09:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.53, 10.47, 9.39 [09:11:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.79, 10.05, 9.36 [09:12:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.62, 10.34, 9.55 [09:14:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.36, 9.96, 9.50 [09:41:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.42, 11.01, 8.91 [09:45:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.22, 11.22, 9.33 [09:47:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.26, 11.07, 9.50 [09:49:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.75, 11.76, 9.94 [09:53:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.46, 10.30, 9.78 [09:53:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.96, 11.04, 10.57 [09:54:22] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.12, 6.16, 3.79 [09:55:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.86, 9.58, 9.56 [09:56:20] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.14, 4.58, 3.50 [09:57:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.39, 9.86, 10.20 [10:01:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.19, 10.26, 10.33 [10:03:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.52, 9.43, 10.00 [10:03:51] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [10:10:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.94, 11.16, 10.18 [10:10:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.64, 10.42, 10.20 [10:12:08] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.77, 9.59, 9.73 [10:12:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.28, 9.94, 10.06 [10:16:22] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [10:31:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 18.10, 12.89, 10.36 [10:32:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.06, 10.78, 9.45 [10:37:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.74, 11.31, 10.69 [10:38:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.58, 10.36, 9.99 [10:40:00] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.67, 9.04, 9.54 [10:41:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 10.09, 9.87, 10.18 [10:45:17] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [10:47:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.98, 10.96, 10.57 [10:49:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.02, 11.86, 10.91 [10:51:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.88, 2.74, 1.74 [10:51:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.07, 9.69, 10.24 [10:53:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.34, 2.64, 1.83 [10:53:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.19, 9.60, 10.16 [11:10:49] [02miraheze/CustomHeader] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/CustomHeader/compare/3ab091348204...6f0d9ad84aaa [11:10:50] [02miraheze/CustomHeader] 07translatewiki 036f0d9ad - Localisation updates from https://translatewiki.net. [11:10:52] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/98dd2c0eccc2...8084dd88e10c [11:10:54] [02miraheze/CreateWiki] 07translatewiki 038084dd8 - Localisation updates from https://translatewiki.net. [11:10:56] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/9cf84293fdba...3aeeebe007d3 [11:10:57] [02miraheze/ImportDump] 07translatewiki 033aeeebe - Localisation updates from https://translatewiki.net. [11:11:00] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/3cd468661061...f69c0648c2a8 [11:11:01] [02miraheze/ManageWiki] 07translatewiki 03f69c064 - Localisation updates from https://translatewiki.net. [11:11:02] [02miraheze/YouTube] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/YouTube/compare/0419d019a2ff...d663901be2f4 [11:11:04] [02miraheze/YouTube] 07translatewiki 03d663901 - Localisation updates from https://translatewiki.net. [11:11:07] [02miraheze/SpriteSheet] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/SpriteSheet/compare/fd2be7dede3f...5e92f971fb59 [11:11:08] [02miraheze/SpriteSheet] 07translatewiki 035e92f97 - Localisation updates from https://translatewiki.net. [11:11:11] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/MirahezeMagic/compare/0d045376e643...6fb32fd2ee0d [11:11:13] [02miraheze/MirahezeMagic] 07translatewiki 036fb32fd - Localisation updates from https://translatewiki.net. [11:14:37] miraheze/CustomHeader - translatewiki the build passed. [11:18:55] miraheze/ManageWiki - translatewiki the build passed. [11:20:04] miraheze/YouTube - translatewiki the build passed. [11:20:17] miraheze/CreateWiki - translatewiki the build passed. [11:21:37] miraheze/SpriteSheet - translatewiki the build passed. [11:22:15] miraheze/ImportDump - translatewiki the build passed. [11:27:54] miraheze/MirahezeMagic - translatewiki the build passed. [11:32:15] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.97, 7.03, 4.52 [11:34:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.84, 4.95, 4.06 [11:37:26] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.87, 3.06, 1.84 [11:39:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.41, 2.35, 1.72 [11:55:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.88, 3.63, 2.39 [11:57:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.74, 3.28, 2.40 [12:11:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.12, 10.29, 9.07 [12:13:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.83, 9.19, 8.81 [12:16:04] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.68, 6.60, 4.47 [12:17:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.60, 10.17, 9.30 [12:19:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.73, 10.63, 9.05 [12:19:40] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.82, 7.40, 5.81 [12:20:05] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 4.31, 6.26, 4.90 [12:21:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.20, 10.28, 9.14 [12:23:40] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.20, 6.57, 5.87 [12:23:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.89, 10.18, 9.67 [12:25:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.27, 9.79, 9.22 [12:29:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.00, 10.88, 10.00 [12:31:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.32, 10.25, 9.62 [12:31:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.93, 10.39, 9.94 [12:33:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.33, 9.80, 9.53 [12:33:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.97, 9.44, 9.64 [12:43:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.34, 12.92, 11.20 [12:45:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.35, 11.91, 11.04 [12:50:15] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.53, 5.86, 4.02 [12:52:16] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.16, 7.46, 4.81 [12:54:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.55, 6.47, 4.79 [12:55:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.62, 10.91, 10.87 [12:57:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.59, 11.13, 10.96 [13:00:06] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.38, 3.02, 1.94 [13:02:06] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.26, 2.28, 1.79 [13:06:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.54, 11.53, 10.14 [13:06:41] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.75, 3.27, 2.74 [13:08:36] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.52, 2.98, 2.70 [13:11:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.10, 3.44, 2.24 [13:11:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.59, 12.92, 11.53 [13:12:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.76, 10.63, 10.24 [13:12:27] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.49, 3.20, 2.85 [13:13:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.78, 2.73, 2.12 [13:13:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.12, 11.91, 11.32 [13:14:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.79, 9.51, 9.86 [13:14:22] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.29, 3.49, 2.99 [13:16:17] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.46, 3.36, 3.00 [13:17:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.43, 11.91, 11.42 [13:18:12] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.00, 3.11, 2.94 [13:23:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.43, 11.69, 11.55 [13:25:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.51, 11.78, 11.58 [13:31:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.12, 10.57, 10.18 [13:31:13] PROBLEM - cp32 APT on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:31:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.69, 11.62, 11.60 [13:33:04] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.63, 11.10, 10.41 [13:33:11] RECOVERY - cp32 APT on cp32 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [13:33:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.11, 11.95, 11.74 [13:37:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.79, 11.40, 10.84 [13:37:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.67, 11.81, 11.78 [13:39:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.71, 12.27, 11.93 [13:41:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.21, 11.64, 11.05 [13:43:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.48, 11.17, 10.92 [13:43:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.21, 11.83, 11.91 [13:45:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.04, 11.56, 11.78 [13:47:00] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.46, 11.33, 11.00 [13:47:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.19, 11.02, 11.56 [13:49:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.39, 9.76, 10.46 [13:49:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.64, 12.27, 11.91 [13:53:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.64, 11.50, 11.75 [13:54:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.75, 9.48, 10.18 [13:59:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.17, 11.89, 11.76 [14:01:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.58, 11.18, 11.54 [14:01:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.31, 11.37, 10.76 [14:04:15] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.94, 5.08, 2.86 [14:04:58] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [14:06:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.12, 4.14, 2.80 [14:06:22] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.29, 3.76, 2.73 [14:07:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.28, 11.51, 11.43 [14:12:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.62, 3.76, 3.10 [14:13:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.73, 11.39, 11.50 [14:13:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.85, 11.46, 11.56 [14:14:05] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.50, 2.92, 2.86 [14:17:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.65, 12.09, 11.62 [14:20:29] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:22:28] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [14:23:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.00, 11.86, 11.52 [14:25:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.03, 11.23, 11.34 [14:27:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.71, 11.20, 11.90 [14:30:13] PROBLEM - cp32 APT on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:34:01] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:37:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.34, 8.78, 9.91 [14:38:20] RECOVERY - cp32 APT on cp32 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [14:43:32] it's redeploy world time [14:44:00] I'm a bit afraid to do it given that apparently mw12* is having issues with overloading, but we'll see [14:48:12] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.36, 3.45, 2.90 [14:50:07] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.07, 3.28, 2.90 [14:51:46] @Reception123 and paladox: mwtask141 doesn't like my key [14:51:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.30, 8.13, 9.86 [14:51:58] though I can login to bast just fine [14:52:16] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 14.48, 10.12, 5.02 [14:52:26] that doesn't make much sense... as puppet would have run for sure [14:52:59] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.91, 2.74, 1.64 [14:53:15] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.036 second response time [14:53:16] I would double check your proxy-jump config just in case [14:53:29] config looks good [14:53:58] also I don't get why all these 502s keep happening? [14:54:11] PROBLEM - wiki.hylormc.tk - reverse DNS on sslhost is UNKNOWN: [14:54:14] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.09, 3.02, 1.58 [14:54:17] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:19] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:45] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.006 second response time [14:55:17] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:20] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.26, 3.69, 2.15 [14:55:30] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:36] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.86, 3.21, 2.31 [14:55:58] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.604 second response time [14:56:13] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 3.31, 3.17, 1.82 [14:56:14] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.325 second response time [14:56:17] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.249 second response time [14:56:22] https://www.irccloud.com/pastebin/T3N6yJg0 [14:56:25] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw131 [14:56:25] That’s why [14:56:42] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.264 second response time [14:56:48] oh, that would make sense [14:56:50] Running manually now [14:56:54] but the 502 issue is strange too [14:57:03] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 16.49, 17.35, 8.05 [14:57:07] it's been happening since the swift issue but that's been fixed (at least in a way) [14:57:08] why the high load on mwtask141 by the way? [14:57:15] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.404 second response time [14:57:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.10, 3.11, 2.13 [14:57:24] Tech:SAL seems to indicate that nothing is really going on there [14:57:25] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.222 second response time [14:57:33] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.34, 3.15, 2.39 [14:57:36] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Syntax error at 'environment' (file: /etc/puppetlabs/puppet/environments/production/modules/mediawiki/manifests/extensionsetup.pp, line: 346, column: 9) on node mwtask141.miraheze.org [14:57:48] you know [14:57:50] Orange_Star: appears to be related to your composer change [14:58:05] I think we just discovered by Composer wasn't running xD [14:58:18] looking at it [14:58:22] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [14:59:06] missing coma, of course [14:59:37] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=esnarutowiki --no-updates /home/reception/esnaruto_pages_full.xml --username-prefix=wikia:naruto (START) [14:59:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:59:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.51, 10.81, 9.88 [14:59:59] ah, the classic error [15:00:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 0.96, 6.79, 6.30 [15:00:34] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f20707524a6b...10a4b1b4af99 [15:00:35] [02miraheze/puppet] 07Reception123 0310a4b1b - syntax fix [15:01:26] changes pulled on puppet141, running puppet on mwtask now [15:01:39] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: no parameter named 'created' (file: /etc/puppetlabs/puppet/environments/production/modules/mediawiki/manifests/extensionsetup.pp, line: 341) on Exec[webauthn_composer] (file: /etc/puppetlabs/puppet/environments/production/modules/mediawiki/manifests/extensionsetup.pp, line: 341) on node mwtask141.miraheze.org [15:01:47] Slightly different error now [15:01:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.55, 10.73, 9.98 [15:02:02] hmm I see [15:02:16] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/10a4b1b4af99...5bf52439c10b [15:02:18] Should be creates [15:02:19] [02miraheze/puppet] 07Reception123 035bf5243 - fix typo [15:05:42] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [15:05:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.54, 9.65, 9.72 [15:06:01] Orange_Star: ^ you should be able to get in now [15:07:43] yes, I can now [15:08:16] though using a FIDO2 key, man I have to complete 2FA two times, once for bast, another for mwtask [15:09:57] well, as it seems Composer is run alongside Puppet after all, I'm more inclined to just wait and see if Composer runs on mw* automatically [15:10:38] PROBLEM - mon141 Current Load on mon141 is WARNING: WARNING - load average: 0.61, 1.65, 3.60 [15:12:38] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 0.18, 1.17, 3.20 [15:19:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.07, 10.70, 10.06 [15:21:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.70, 10.82, 10.17 [15:23:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.42, 11.13, 10.34 [15:23:20] PROBLEM - wiki.hylormc.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.hylormc.tk could not be found [15:28:49] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.75, 4.13, 2.85 [15:29:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.48, 11.70, 10.89 [15:30:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.71, 3.57, 2.80 [15:32:09] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [15:32:44] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.63, 2.83, 2.61 [15:35:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.53, 6.31, 3.56 [15:36:00] PROBLEM - cp32 APT on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:37:58] RECOVERY - cp32 APT on cp32 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [15:40:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.85, 11.41, 10.89 [15:43:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.08, 3.16, 3.29 [15:43:36] !log [www-data@mwtask141] starting deploy of {'world': True} to all [15:43:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:43:50] ? [15:43:57] who's that? [15:44:43] me [15:45:12] it has to be done - puppet just installes the packages on mwtask141, and then deploy-mediawiki copies it all to mw* [15:45:33] if puppet was deploying straight to mw*, their puppet runs would have failed as well [15:45:38] makes sense [15:45:38] but they didn't [15:48:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.70, 10.84, 10.20 [15:48:48] !log [www-data@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 311s [15:48:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:49:36] Composer package shows up in Special:Version [15:50:37] seems to be working now [15:50:54] yeah, it's working, I was able to register a key [15:50:55] I don't have a physical key, but was able to add one via ios/macos passkeys [15:51:58] we bleeding edge now :) [15:52:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.31, 11.55, 11.45 [15:52:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.61, 9.55, 9.86 [15:53:31] meanwhile beta is broken, since the extension was never added on the REL1_40 branch [15:53:55] ah yeah that's the inconvenient of between update periods [15:53:58] you need to add to both branches [15:54:12] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.79, 3.59, 2.96 [15:56:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.12, 12.47, 11.80 [15:56:25] RECOVERY - mon141 APT on mon141 is OK: APT OK: 11 packages available for upgrade (0 critical updates). [15:56:27] !log [reception@puppet141] Upgraded packages libapr1, apache2-bin, ghostscript, libgs9, and libgs9-common on mon141 [15:56:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:57:14] !log [reception@puppet141] Upgraded packages ghostscript, libgs9, and libgs9-common on jobchron121 [15:57:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:57:44] !log [reception@puppet141] Upgraded packages ghostscript, libgs9, libgs9-common, xserver-common, and xvfb on mw131 [15:57:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:58:05] RECOVERY - jobchron121 APT on jobchron121 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [15:58:07] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.34, 4.67, 3.50 [15:58:10] RECOVERY - mw131 APT on mw131 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [15:58:42] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.01, 12.08, 10.76 [15:58:59]