[00:03:16] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.10, 3.45, 2.87 [00:04:26] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.2980043292 secs [00:05:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.72, 3.16, 2.84 [00:12:29] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.22, 5.29, 3.83 [00:16:22] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.32, 3.92, 3.60 [00:18:18] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.51, 2.97, 3.29 [00:30:45] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.66, 7.77, 5.30 [00:32:26] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.02233591676 secs [00:34:38] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.91, 6.54, 5.43 [00:35:51] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.32, 5.23, 3.82 [00:45:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.41, 3.59, 3.78 [00:49:51] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.83, 4.06, 3.92 [00:51:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.43, 3.53, 3.75 [00:55:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.53, 2.38, 3.22 [00:56:09] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.98, 5.88, 5.08 [00:58:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.83, 7.02, 5.58 [00:58:19] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1234410107 secs [01:00:09] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.98, 6.80, 5.73 [01:02:27] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:04:58] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [01:10:18] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07946747541 secs [01:20:25] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.07, 3.44, 3.15 [01:26:09] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.12, 6.19, 4.13 [01:26:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.72, 3.98, 3.62 [01:28:09] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.24, 4.24, 3.66 [01:28:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.35, 3.97, 3.64 [01:30:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.49, 3.61, 3.56 [01:34:01] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.23, 3.24, 3.40 [01:37:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.69, 3.79, 3.59 [01:39:53] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.00, 3.07, 3.35 [02:01:55] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:03:44] [02mw-config] 07brandon-wm synchronize pull request 03#5172: T10684: Enable Special:Mute - 13https://github.com/miraheze/mw-config/pull/5172 [02:03:53] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [02:04:40] miraheze/mw-config - brandon-wm the build passed. [02:04:58] [02mw-config] 07brandon-wm synchronize pull request 03#5172: T10684: Enable Special:Mute - 13https://github.com/miraheze/mw-config/pull/5172 [02:05:57] miraheze/mw-config - brandon-wm the build passed. [02:06:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.95, 3.71, 3.15 [02:10:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.47, 3.68, 3.30 [02:12:05] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.12, 3.23, 3.19 [02:28:09] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.59, 3.21, 2.59 [02:30:06] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.33, 2.89, 2.55 [02:30:31] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:32:47] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.3502269387 secs [02:33:37] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.46, 5.18, 2.95 [02:33:41] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1329237223 secs [02:38:50] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [02:40:48] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.03517386317 secs [02:41:41] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.09536603093 secs [02:47:39] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.38, 2.24, 3.59 [02:49:38] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.08, 1.51, 3.16 [03:13:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.95, 3.76, 3.24 [03:15:55] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.99, 3.26, 3.11 [03:21:51] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.57, 4.46, 3.63 [03:25:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.34, 3.75, 3.54 [03:31:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.28, 3.18, 3.38 [03:35:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.43, 3.86, 3.66 [03:41:51] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.56, 3.84, 3.68 [03:43:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.31, 3.56, 3.59 [03:45:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.15, 3.00, 3.37 [04:00:51] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:03:18] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.22, 4.59, 2.60 [04:06:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.74, 3.56, 3.06 [04:08:24] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.31, 3.56, 3.10 [04:08:41] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [04:09:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.90, 3.15, 2.70 [04:12:17] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.35, 3.37, 3.10 [04:59:02] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.31, 2.43, 1.39 [05:00:53] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:01:02] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.79, 2.44, 1.52 [05:02:51] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [05:26:52] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1436252594 secs [05:28:57] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.65, 3.60, 1.93 [05:31:23] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:32:56] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.68, 2.87, 2.05 [05:33:21] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [05:44:53] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.09381976724 secs [06:06:18] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.294988662 secs [06:14:54] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset -0.1108440161 secs [06:16:18] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.01232796907 secs [06:18:54] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.09146124125 secs [06:31:27] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:33:29] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [06:58:59] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=panchikowiki --no-updates --username-prefix=w /home/reception/ImportDump/panchikowiki-20230408183656.xml (START) [06:59:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:59:04] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=panchikowiki --no-updates --username-prefix=w /home/reception/ImportDump/panchikowiki-20230408183656.xml (END - exit=0) [06:59:04] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=panchikowiki (START) [06:59:08] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=androgynywiki --no-updates --username-prefix=wikia:tales-of-androgyny /home/reception/ImportDump/androgynywiki-20230409050438.xml (START) [06:59:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:59:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:59:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:27] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=omniversalbattlefieldwiki --no-updates --username-prefix=wikia:omniversal-battlefield /home/reception/omniversalbattlefield_pages_full.xml (START) [07:02:28] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=omniversalbattlefieldwiki --no-updates --username-prefix=wikia:omniversal-battlefield /home/reception/omniversalbattlefield_pages_full.xml (END - exit=65280) [07:02:29] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=omniversalbattlefieldwiki (START) [07:02:30] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=omniversalbattlefieldwiki (END - exit=65280) [07:02:31] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=omniversalbattlefieldwiki --active --update (END - exit=65280) [07:02:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:06:03] PROBLEM - es141 Current Load on es141 is WARNING: WARNING - load average: 3.63, 2.65, 1.35 [07:06:23] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=androgynywiki --no-updates --username-prefix=wikia:tales-of-androgyny /home/reception/ImportDump/androgynywiki-20230409050438.xml (END - exit=0) [07:06:24] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=androgynywiki (START) [07:06:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:06:32] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=panchikowiki (END - exit=0) [07:06:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:06:33] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=panchikowiki --active --update (END - exit=0) [07:06:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:06:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:35] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=androgynywiki (END - exit=0) [07:07:36] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=androgynywiki --active --update (END - exit=0) [07:07:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:01] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 0.74, 1.92, 1.24 [07:33:50] PROBLEM - wiki.litek.top - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.litek.top could not be found [08:00:36] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.02, 3.56, 1.95 [08:02:27] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:04:36] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.38, 3.76, 2.44 [08:07:21] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [08:08:36] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.32, 3.07, 2.51 [08:26:26] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1583125293 secs [08:49:04] [02mw-config] 07ugochimobi opened pull request 03#5189: T10711: wgExtraLanguageNames for fuutropediawiki and factfinder3dwiki - 13https://github.com/miraheze/mw-config/pull/5189 [08:50:05] miraheze/mw-config - ugochimobi the build passed. [08:50:13] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [08:52:26] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.0572668314 secs [08:58:24] [02mw-config] 07redbluegreenhat reviewed pull request 03#5172 commit - 13https://github.com/miraheze/mw-config/pull/5172#discussion_r1161560518 [09:19:55] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [11:08:47] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=omniversalbattlefieldwiki --no-updates --username-prefix=wikia:omniversal-battlefield /home/reception/omniversalbattlefield_pages_full.xml (START) [11:08:48] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=omniversalbattlefieldwiki --no-updates --username-prefix=wikia:omniversal-battlefield /home/reception/omniversalbattlefield_pages_full.xml (END - exit=65280) [11:08:49] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=omniversalbattlefieldwiki (START) [11:08:50] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=omniversalbattlefieldwiki (END - exit=65280) [11:08:51] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=omniversalbattlefieldwiki --active --update (END - exit=65280) [11:08:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:08:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:09:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:09:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:09:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:10:49] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/fe0310e89b7d...a01e9d3b89e7 [11:10:52] [02miraheze/CreateWiki] 07translatewiki 03a01e9d3 - Localisation updates from https://translatewiki.net. [11:10:54] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/7935d3663e67...9cf84293fdba [11:10:56] [02miraheze/ImportDump] 07translatewiki 039cf8429 - Localisation updates from https://translatewiki.net. [11:10:59] [02miraheze/MatomoAnalytics] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/23332d0b0881...cdcffa0ecca9 [11:11:02] [02miraheze/MatomoAnalytics] 07translatewiki 03cdcffa0 - Localisation updates from https://translatewiki.net. [11:11:05] [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/2fa5030b36ab...78cd4bb1dcb0 [11:11:07] [02miraheze/DataDump] 07translatewiki 0378cd4bb - Localisation updates from https://translatewiki.net. [11:11:08] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/77b546e318d5...66b9db8d1970 [11:11:11] [02miraheze/MirahezeMagic] 07translatewiki 0366b9db8 - Localisation updates from https://translatewiki.net. [11:11:29] PROBLEM - vise.dayid.org - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.405372381210327 seconds [11:18:24] miraheze/MatomoAnalytics - translatewiki the build passed. [11:18:56] miraheze/DataDump - translatewiki the build passed. [11:20:14] miraheze/CreateWiki - translatewiki the build passed. [11:21:03] miraheze/ImportDump - translatewiki the build passed. [11:24:30] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/bb8828b72712...a8b32e54d633 [11:24:31] [02miraheze/dns] 07Reception123 03a8b32e5 - add beamngmodding.com zone [11:24:36] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.60, 3.69, 2.15 [11:25:40] miraheze/MirahezeMagic - translatewiki the build passed. [11:26:36] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.94, 3.09, 2.11 [11:31:36] [02mw-config] 07brandon-wm synchronize pull request 03#5172: T10684: Enable Special:Mute - 13https://github.com/miraheze/mw-config/pull/5172 [11:31:46] [02mw-config] 07brandon-wm reviewed pull request 03#5172 commit - 13https://github.com/miraheze/mw-config/pull/5172#discussion_r1161650668 [11:32:41] miraheze/mw-config - brandon-wm the build passed. [11:49:31] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/de74ab201baa...f99533b9fa25 [11:49:34] [02miraheze/ssl] 07MirahezeSSLBot 03f99533b - Bot: Add SSL cert for wiki.hsdfz.org.cn [11:54:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/f99533b9fa25...1738929954a1 [11:54:53] [02miraheze/ssl] 07MirahezeSSLBot 031738929 - Bot: Add SSL cert for beamngmodding.com [12:00:36] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.34, 3.26, 2.16 [12:01:20] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:02:36] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.81, 3.43, 2.37 [12:02:49] [02mw-config] 07Reception123 closed pull request 03#5189: T10711: wgExtraLanguageNames for fuutropediawiki and factfinder3dwiki - 13https://github.com/miraheze/mw-config/pull/5189 [12:02:50] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4015695f58a7...3642cdb773a7 [12:02:51] [02miraheze/mw-config] 07ugochimobi 033642cdb - T10711: wgExtraLanguageNames for fuutropediawiki and factfinder3dwiki (#5189) [12:03:32] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [12:03:48] miraheze/mw-config - Reception123 the build passed. [12:04:21] !log swift download miraheze-dcmultiversewiki-local-public -D dcmultiversewiki-images --object-threads=1 [12:04:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:04:36] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.39, 2.66, 2.21 [12:09:17] RECOVERY - vise.dayid.org - reverse DNS on sslhost is OK: SSL OK - vise.dayid.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [12:14:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.70, 5.97, 3.63 [12:16:54] !log [@mwtask141] starting deploy of {'config': True} to all [12:16:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:17:46] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 52s [12:17:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:21:08] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.64, 3.59, 2.27 [12:23:08] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.82, 3.76, 2.51 [12:25:20] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.63, 4.01, 2.77 [12:25:58] PROBLEM - es141 Current Load on es141 is CRITICAL: CRITICAL - load average: 4.42, 3.40, 2.08 [12:28:13] !log [@test131] starting deploy of {'config': True} to all [12:28:14] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [12:28:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:28:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:29:13] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.86, 2.75, 2.21 [12:29:46] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 2.20, 3.27, 2.37 [12:31:12] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.02, 2.38, 2.13 [12:31:16] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.25, 3.44, 3.00 [12:31:33] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [12:33:15] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.71, 3.10, 2.92 [12:35:37] PROBLEM - es141 Current Load on es141 is CRITICAL: CRITICAL - load average: 4.04, 3.91, 2.92 [12:41:10] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.51, 3.72, 3.23 [12:43:40] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [12:45:10] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.74, 3.82, 3.39 [12:47:09] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.70, 4.13, 3.55 [12:52:15] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [12:56:19] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.2097361386 secs [13:01:01] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:01:44] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:08:09] PROBLEM - swiftobject111 Disk Space on swiftobject111 is WARNING: DISK WARNING - free space: / 58600 MB (10% inode=84%); [13:08:18] PROBLEM - es141 Current Load on es141 is WARNING: WARNING - load average: 0.57, 2.61, 3.71 [13:09:40] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:12:18] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 0.49, 1.63, 3.06 [13:13:13] [02mw-config] 07redbluegreenhat reviewed pull request 03#5172 commit - 13https://github.com/miraheze/mw-config/pull/5172#discussion_r1161709342 [13:15:15] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 1.13, 2.21, 3.71 [13:17:14] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 0.99, 1.82, 3.38 [13:21:51] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 1.67, 2.78, 7.60 [13:24:18] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.01919481158 secs [13:25:49] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.36, 2.62, 6.44 [13:32:07] [1/2] Been getting the occasional 503 for this URL [2/2] rts%7Cskins.citizen.icons%2Cstyles%7Cskins.citizen.icons.wmui&only=styles&skin=citizen> [13:32:38] [1/20] here's the x- values for the most recent 503 [13:32:38] [2/20] ``` [13:32:39] [3/20] x-cache [13:32:39] [4/20] cp32 MISS UNCACHEABLE [13:32:39] [5/20] x-content-type-options [13:32:40] [6/20] nosniff [13:32:40] [7/20] X-Firefox-Spdy [13:32:40] [8/20] h2 [13:32:41] [9/20] x-powered-by [13:32:41] [10/20] PHP/7.4.33 [13:32:41] [11/20] x-request-id [13:32:42] [12/20] 0525e7d4fc040b4781dac2fd [13:32:42] [13/20] x-varnish [13:32:43] [14/20] 810780008 [13:32:43] [15/20] x-wiki-visibility [13:32:44] [16/20] Public [13:32:44] [17/20] x-xss-protection [13:32:45] [18/20] 1; mode=block [13:32:45] [19/20] ``` [13:32:46] [20/20] couldn't seem to find the x-served-by header though [13:35:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.58, 10.01, 8.54 [13:35:36] [02mw-config] 07brandon-wm synchronize pull request 03#5172: T10684: Enable Special:Mute - 13https://github.com/miraheze/mw-config/pull/5172 [13:36:38] miraheze/mw-config - brandon-wm the build passed. [13:37:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.19, 10.41, 8.84 [13:39:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.32, 10.43, 9.41 [13:39:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.73, 10.31, 9.01 [13:41:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.50, 12.63, 10.31 [13:41:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.59, 11.00, 9.41 [13:43:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.73, 10.71, 9.50 [13:45:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.88, 11.11, 9.76 [13:47:32] [02mw-config] 07redbluegreenhat closed pull request 03#5172: T10684: Enable Special:Mute - 13https://github.com/miraheze/mw-config/pull/5172 [13:47:35] !log [alex@mwtask141] starting deploy of {'config': True} to all [13:47:35] [02miraheze/mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3642cdb773a7...421e3e47cc14 [13:47:38] [02miraheze/mw-config] 07brandon-wm 03421e3e4 - T10684: Enable Special:Mute (#5172) [13:47:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:47:46] !log [alex@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 11s [13:47:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:48:37] miraheze/mw-config - redbluegreenhat the build passed. [13:55:15] I think I went too fast here [13:56:03] !log [alex@mwtask141] starting deploy of {'config': True} to all [13:56:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:10] !log [alex@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [13:56:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:40] !log [@test131] starting deploy of {'config': True} to all [13:56:41] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [13:56:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:58:02] Reception123: I deployed the new changes, but there seems to have been no effect? [14:04:09] RECOVERY - swiftobject111 Disk Space on swiftobject111 is OK: DISK OK - free space: / 58638 MB (11% inode=84%); [14:07:48] Orange_Star: don't forget --pull=config [14:08:09] PROBLEM - swiftobject111 Disk Space on swiftobject111 is WARNING: DISK WARNING - free space: / 58565 MB (10% inode=84%); [14:09:56] !log [alex@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [14:10:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:10:07] !log [alex@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [14:10:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:10:49] oh, you're right [14:11:05] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:15:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.74, 10.45, 11.71 [14:24:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.41, 10.39, 11.92 [14:26:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.11, 10.86, 11.89 [14:29:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.29, 11.91, 11.40 [14:31:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.59, 11.00, 11.11 [14:32:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.12, 11.02, 11.79 [14:35:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.34, 11.03, 11.03 [14:36:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.37, 10.99, 11.49 [14:38:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.70, 11.21, 11.54 [14:39:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.80, 10.96, 10.96 [14:40:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.92, 11.84, 11.73 [14:40:40] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [14:42:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.38, 11.85, 11.76 [14:43:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.36, 11.09, 10.98 [14:45:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.04, 11.06, 10.98 [14:46:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.77, 12.14, 11.91 [14:49:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.96, 11.47, 11.12 [14:51:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.78, 11.65, 11.23 [14:55:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.98, 12.10, 11.43 [14:57:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.81, 11.83, 11.41 [15:05:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.13, 11.46, 11.15 [15:07:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.03, 10.52, 10.85 [15:08:10] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.21, 10.06, 11.69 [15:12:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.65, 12.55, 12.22 [15:14:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.65, 11.27, 11.79 [15:16:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.12, 11.75, 11.89 [15:18:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.30, 11.13, 11.64 [15:19:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.54, 10.66, 10.60 [15:20:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.86, 12.23, 11.96 [15:21:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.23, 10.08, 10.43 [15:24:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.66, 11.59, 11.76 [15:25:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.57, 9.25, 10.03 [15:25:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.25, 11.80, 11.81 [15:27:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.12, 11.98, 11.91 [15:31:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.65, 12.15, 11.92 [15:33:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.48, 11.82, 11.84 [15:36:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.58, 12.02, 10.90 [15:38:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.62, 10.80, 10.59 [15:41:48] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.38, 11.68, 11.67 [15:43:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.48, 11.56, 11.63 [15:49:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.94, 9.15, 9.98 [15:51:50] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [15:55:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.92, 10.57, 10.80 [15:57:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.34, 10.30, 10.67 [16:05:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.82, 11.18, 10.30 [16:08:17] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1566018462 secs [16:08:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.60, 11.27, 10.53 [16:11:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.01, 11.24, 10.63 [16:12:39] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.37, 10.87, 10.48 [16:13:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.48, 10.48, 10.43 [16:14:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.15, 9.72, 10.10 [16:19:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.04, 9.73, 10.17 [16:20:40] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:34:17] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.006673127413 secs [16:35:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.92, 9.82, 9.20 [16:37:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.61, 9.74, 9.25 [16:42:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.08, 11.28, 9.85 [16:48:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.65, 11.66, 10.58 [16:49:50] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [16:56:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.37, 11.92, 10.91 [16:59:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.28, 10.51, 9.78 [17:01:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.78, 9.76, 9.59 [17:02:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.80, 11.64, 11.25 [17:21:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.94, 9.02, 9.97 [17:25:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.34, 10.25, 9.52 [17:27:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.46, 10.00, 9.52 [17:27:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.63, 12.05, 10.91 [17:29:07] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.47, 2.99, 2.19 [17:29:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.21, 10.95, 10.65 [17:31:06] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.72, 2.80, 2.22 [17:35:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.10, 11.95, 10.30 [17:37:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.41, 11.45, 10.30 [17:37:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.15, 11.09, 10.69 [17:39:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.66, 12.38, 10.76 [17:41:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.37, 11.81, 10.75 [17:43:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.48, 13.60, 11.53 [17:47:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.81, 11.69, 11.48 [17:49:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.52, 11.52, 11.37 [17:57:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.08, 8.92, 9.98 [17:57:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.58, 9.06, 10.07 [18:06:26] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.45, 3.52, 2.59 [18:08:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.19, 2.99, 2.51 [18:12:17] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.25, 5.37, 2.88 [18:12:38] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:14:17] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.90, 4.20, 2.78 [18:15:24] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.89, 3.61, 2.69 [18:23:08] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.07, 3.14, 2.90 [18:42:13] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [18:43:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.23, 11.48, 9.46 [18:45:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.86, 10.86, 9.47 [18:47:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.58, 9.73, 9.23 [18:51:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.88, 10.72, 9.73 [18:53:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.41, 10.87, 9.39 [18:53:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.36, 11.21, 10.02 [18:55:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.40, 10.46, 9.44 [18:55:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.34, 11.06, 10.10 [18:57:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.14, 9.86, 9.36 [18:59:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.67, 9.54, 9.72 [19:00:54] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.99, 3.64, 2.78 [19:02:50] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.79, 2.81, 2.57 [19:05:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.36, 10.30, 9.59 [19:09:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.83, 10.58, 10.00 [19:13:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.23, 10.86, 10.10 [19:13:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.61, 11.44, 10.44 [19:17:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.12, 11.61, 10.61 [19:17:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.54, 11.16, 10.56 [19:17:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1573960483 secs [19:19:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.02, 11.42, 10.65 [19:19:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.50, 12.86, 11.23 [19:43:57] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07810303569 secs [19:45:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.30, 9.36, 11.80 [19:45:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.34, 10.19, 11.45 [19:53:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.37, 7.96, 10.07 [19:53:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.40, 8.46, 10.09 [20:15:50] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [20:25:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.46, 11.86, 9.79 [20:26:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.99, 11.31, 9.33 [20:27:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.98, 11.63, 9.97 [20:28:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.04, 10.21, 9.15 [20:31:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.90, 9.67, 9.23 [20:35:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.91, 12.08, 10.70 [20:35:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.35, 12.18, 10.34 [20:41:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.81, 11.41, 10.73 [20:53:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.02, 10.68, 10.60 [20:53:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.11, 11.14, 11.92 [20:55:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.65, 11.69, 12.02 [20:57:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.77, 11.52, 10.99 [20:57:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.54, 11.13, 11.79 [21:03:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.34, 8.84, 9.97 [21:09:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.99, 10.39, 10.64 [21:13:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.37, 10.17, 10.58 [21:14:59] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:23:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.76, 8.69, 9.77 [21:44:34] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [22:01:10] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:03:10] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [22:19:49] PROBLEM - wiki.widwwa.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.widwwa.co.uk' expires in 15 day(s) (Wed 26 Apr 2023 21:52:35 GMT +0000). [22:24:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1738929954a1...5fe2077258b1 [22:24:27] [02miraheze/ssl] 07MirahezeSSLBot 035fe2077 - Bot: Update SSL cert for wiki.widwwa.co.uk [22:32:07] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:34:07] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [22:38:40] [1/21] Been getting 503's on pages in addition to missing CSS [22:38:40] [2/21] Here's the x- values of the latest 503 I got [22:38:40] [3/21] ``` [22:38:40] [4/21] x-cache [22:38:41] [5/21] cp33 MISS UNCACHEABLE [22:38:41] [6/21] x-content-type-options [22:38:41] [7/21] nosniff [22:38:42] [8/21] X-Firefox-Spdy [22:38:42] [9/21] h2 [22:38:42] [10/21] x-powered-by [22:38:43] [11/21] PHP/7.4.33 [22:38:43] [12/21] x-request-id [22:38:44] [13/21] e058141ea935396f770f1786 [22:38:44] [14/21] x-varnish [22:38:45] [15/21] 101608083 [22:38:45] [16/21] x-wiki-visibility [22:38:46] [17/21] Public [22:38:46] [18/21] x-xss-protection [22:38:47] [19/21] 1; mode=block [22:38:47] [20/21] ``` [22:38:48] [21/21] It'll sometimes refresh and the page appears, but when that happens some or all CSS is missing [22:39:17] The 503 page does mention SMW and being on maintenance mode, so is it related to the wiki being migrated to SMW recently? [22:49:37] RECOVERY - wiki.widwwa.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.widwwa.co.uk' will expire on Sun 09 Jul 2023 21:24:17 GMT +0000. [22:57:31] https://phabricator.miraheze.org/p/Monicageller777/ spam bot [22:58:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.49, 9.68, 8.90 [22:58:51] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.26, 3.01, 1.79 [23:00:34] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.30, 9.10, 8.77 [23:00:50] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 3.16, 2.80, 1.85 [23:01:03] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:04:38] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.48, 3.15, 2.73 [23:05:17] [02puppet] 07The-Voidwalker closed pull request 03#3181: update grants for cargouser - 13https://github.com/miraheze/puppet/pull/3181 [23:05:18] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d33ea083f28b...0be907ef7c84 [23:05:20] [02miraheze/puppet] 07The-Voidwalker 030be907e - update grants for cargouser (#3181) [23:05:23] [02miraheze/puppet] 07The-Voidwalker deleted branch 03patch-LimitCargoDB [23:05:24] [02puppet] 07The-Voidwalker deleted branch 03patch-LimitCargoDB - 13https://github.com/miraheze/puppet [23:08:29] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.98, 3.25, 2.90 [23:29:01] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:30:44] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 4.00, 3.49, 2.89 [23:32:01] PROBLEM - en.religiononfire.mar.in.ua - LetsEncrypt on sslhost is CRITICAL: connect to address en.religiononfire.mar.in.ua and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [23:33:44] PROBLEM - atnarsia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for atnarsia.com could not be found [23:34:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.25, 3.96, 3.18 [23:36:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.36, 3.75, 3.20 [23:38:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.88, 3.38, 3.12