[00:07:10] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.02, 6.58, 4.32 [00:10:50] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [00:14:51] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.78, 7.75, 6.20 [00:16:47] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.02, 6.26, 5.86 [00:31:31] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.20, 9.11, 6.87 [00:37:17] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.21, 6.66, 6.71 [00:40:38] RECOVERY - graylog131 NTP time on graylog131 is OK: NTP OK: Offset -0.0008296072483 secs [00:42:14] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zymonic.com All nameservers failed to answer the query. [00:54:44] PROBLEM - swiftac111 Backups Swift Account Container on swiftac111 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/swift-account-container-backup.log [01:12:16] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406045913696289 seconds [01:16:20] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1177940667 secs [01:23:30] [02ManageWiki] 07Universal-Omega opened pull request 03#392: Add support for using name instead of var in extensions cache - 13https://github.com/miraheze/ManageWiki/pull/392 [01:25:55] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.15, 3.17, 2.60 [01:26:52] miraheze/ManageWiki - Universal-Omega the build passed. [01:27:53] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.52, 3.00, 2.61 [01:36:21] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.09972926974 secs [01:41:06] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [01:41:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.69, 4.08, 2.95 [01:43:18] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.62, 3.47, 2.86 [01:45:23] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.76, 3.66, 2.98 [01:46:08] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.28, 5.65, 3.25 [01:47:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.07, 3.02, 2.83 [01:48:07] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.62, 4.15, 3.01 [02:04:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.22, 9.50, 7.78 [02:06:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.32, 8.36, 7.58 [02:09:59] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:37:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.50, 4.23, 2.81 [02:39:38] PROBLEM - cp32 APT on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:41:40] RECOVERY - cp32 APT on cp32 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [02:45:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.41, 3.73, 3.41 [02:47:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.10, 3.11, 3.22 [03:01:44] RECOVERY - db131 Backups SQL on db131 is OK: FILE_AGE OK: /var/log/sql-backup.log is 102 seconds old and 275 bytes [03:02:19] PROBLEM - db112 Backups SQL on db112 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864136 seconds old and 364 bytes [03:02:59] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.32, 4.90, 3.57 [03:04:00] PROBLEM - db112 Backups SQL on db112 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864238 seconds old and 364 bytes [03:08:52] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.38, 3.65, 3.46 [03:10:49] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.90, 3.13, 3.30 [03:26:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.78, 6.25, 4.23 [03:32:26] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.38, 3.97, 3.85 [03:38:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.44, 2.45, 3.24 [03:42:30] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.2191419601 secs [03:52:30] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.01636949182 secs [03:54:57] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.21, 3.81, 3.24 [03:56:55] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.13, 3.58, 3.23 [03:58:53] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.97, 4.54, 3.65 [03:59:09] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1028255522 secs [04:00:50] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.04, 3.74, 3.47 [04:02:48] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.23, 2.94, 3.21 [04:05:09] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09603181481 secs [04:26:18] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.42, 4.28, 3.23 [04:28:16] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.31, 3.59, 3.10 [04:30:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.03, 3.29, 3.07 [04:52:38] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1701358259 secs [05:24:38] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.08676815033 secs [05:48:15] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:58:15] [02miraheze/RemovePII] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-phan-config-0.12.1 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/commit/884be15ee6c0 [05:58:18] [02miraheze/RemovePII] 07dependabot[bot] 03884be15 - Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 [05:58:21] [02RemovePII] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-phan-config-0.12.1 - 13https://github.com/miraheze/RemovePII [05:58:22] [02RemovePII] 07dependabot[bot] opened pull request 03#67: Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/RemovePII/pull/67 [05:58:23] [02RemovePII] 07dependabot[bot] labeled pull request 03#67: Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/RemovePII/pull/67 [05:58:25] [02RemovePII] 07dependabot[bot] labeled pull request 03#67: Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/RemovePII/pull/67 [06:00:39] miraheze/RemovePII - dependabot[bot] the build has errored. [06:07:42] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.48, 4.53, 2.66 [06:08:27] PROBLEM - cp32 APT on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:09:57] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zymonic.com All nameservers failed to answer the query. [06:13:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.06, 3.61, 2.89 [06:13:42] RECOVERY - cp32 APT on cp32 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:15:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.70, 3.01, 2.76 [06:18:16] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [06:38:41] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1956752241 secs [06:39:58] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.402597665786743 seconds [06:40:03] PROBLEM - cp22 NTP time on cp22 is CRITICAL: NTP CRITICAL: Offset 0.7698574066 secs [06:47:09] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1006385088 secs [06:57:09] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08002182841 secs [07:02:03] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.0005888938904 secs [07:02:43] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.02168110013 secs [07:16:03] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.448725611 secs [07:18:03] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.0001749694347 secs [07:50:00] PROBLEM - dft.wikiyri.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dft.wikiyri.org could not be found [07:51:16] PROBLEM - en.wikiyri.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns2.byet.org.', 'ns1.pony.icu.', 'ns4.byet.org.', 'ns1.byet.org.', 'ns3.byet.org.', 'ns5.byet.org.', 'ns2.pony.icu.'], 'CNAME': None} [07:55:34] PROBLEM - wikiyri.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikiyri.org could not be found [07:57:45] PROBLEM - wikiyri.org - LetsEncrypt on sslhost is CRITICAL: connect to address wikiyri.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:03:46] PROBLEM - bn.wikiyri.org - LetsEncrypt on sslhost is CRITICAL: connect to address bn.wikiyri.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:10:18] PROBLEM - data.wikiyri.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns2.byet.org.', 'ns1.pony.icu.', 'ns4.byet.org.', 'ns1.byet.org.', 'ns3.byet.org.', 'ns5.byet.org.', 'ns2.pony.icu.'], 'CNAME': '11776.BODIS.COM.'} [08:15:25] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1757653356 secs [08:22:21] PROBLEM - en.wikiyri.org - LetsEncrypt on sslhost is CRITICAL: connect to address en.wikiyri.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:23:38] PROBLEM - dft.wikiyri.org - LetsEncrypt on sslhost is CRITICAL: connect to address dft.wikiyri.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:27:25] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.06757691503 secs [08:35:31] PROBLEM - bn.wikiyri.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bn.wikiyri.org could not be found [08:48:25] PROBLEM - data.wikiyri.org - LetsEncrypt on sslhost is CRITICAL: connect to address data.wikiyri.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:52:17] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [09:01:43] RECOVERY - bn.wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'bn.wikiyri.org' will expire on Fri 12 May 2023 03:51:17 GMT +0000. [09:07:44] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.14, 2.57, 1.54 [09:09:08] PROBLEM - cp32 APT on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:09:42] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.69, 2.72, 1.73 [09:13:37] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.41, 4.77, 2.83 [09:14:14] RECOVERY - cp32 APT on cp32 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [09:15:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.30, 3.79, 2.70 [09:17:13] RECOVERY - data.wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'data.wikiyri.org' will expire on Sat 03 Jun 2023 17:21:16 GMT +0000. [09:17:33] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.05, 2.85, 2.48 [09:20:44] RECOVERY - en.wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.wikiyri.org' will expire on Fri 12 May 2023 15:47:14 GMT +0000. [09:21:53] RECOVERY - dft.wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dft.wikiyri.org' will expire on Mon 19 Jun 2023 16:40:54 GMT +0000. [09:22:18] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [09:27:40] RECOVERY - wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiyri.org' will expire on Fri 12 May 2023 03:48:20 GMT +0000. [10:03:09] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1611083448 secs [10:23:09] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.06445080042 secs [10:56:18] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:09:53] RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: SSL OK - wiki.zymonic.com reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:16:55] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=huevoversowiki --no-updates --username-prefix=es /home/reception/ImportDump/huevoversowiki-20230418101558.xml (START) [11:17:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:17:11] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=huevoversowiki --no-updates --username-prefix=wikia:theeggverse /home/reception/ImportDump/huevoversowiki-20230416143732.xml (START) [11:17:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:17:17] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=boomerangfuwiki --no-updates --username-prefix=wikia:boomerangfu /home/reception/ImportDump/boomerangfuwiki-20230416200130.xml (START) [11:17:22] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=simdemocracywiki --no-updates --username-prefix=w /home/reception/ImportDump/simdemocracywiki-20230417192914.xml (START) [11:17:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:17:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:18:11] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=simdemocracywiki --no-updates --username-prefix=w /home/reception/ImportDump/simdemocracywiki-20230417192914.xml (END - exit=0) [11:18:12] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=simdemocracywiki (START) [11:18:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:18:17] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=huevoversowiki --no-updates --username-prefix=wikia:theeggverse /home/reception/ImportDump/huevoversowiki-20230416143732.xml (END - exit=0) [11:18:18] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=huevoversowiki (START) [11:18:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:18:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:18:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:20:22] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=huevoversowiki --no-updates --username-prefix=es /home/reception/ImportDump/huevoversowiki-20230418101558.xml (END - exit=0) [11:20:24] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=huevoversowiki (START) [11:20:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:20:26] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=boomerangfuwiki --no-updates --username-prefix=wikia:boomerangfu /home/reception/ImportDump/boomerangfuwiki-20230416200130.xml (END - exit=0) [11:20:27] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=boomerangfuwiki (START) [11:20:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:20:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:20:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:21:21] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=simdemocracywiki (END - exit=0) [11:21:22] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=simdemocracywiki --active --update (END - exit=0) [11:21:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:21:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:22:39] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=boomerangfuwiki (END - exit=0) [11:22:40] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=boomerangfuwiki --active --update (END - exit=0) [11:22:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:22:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:22:58] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 6.56, 4.12, 1.93 [11:23:32] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=huevoversowiki (END - exit=0) [11:23:33] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=huevoversowiki --active --update (END - exit=0) [11:23:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:23:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:24:36] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=huevoversowiki (END - exit=0) [11:24:37] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=huevoversowiki --active --update (END - exit=0) [11:24:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:24:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:24:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.14, 3.46, 1.96 [11:26:18] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:26:58] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 0.37, 2.35, 1.74 [11:33:09] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1889382601 secs [11:42:44] PROBLEM - es141 PowerDNS Recursor on es141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:42:47] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.52, 5.76, 3.37 [11:44:41] RECOVERY - es141 PowerDNS Recursor on es141 is OK: DNS OK: 0.166 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [11:44:47] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.22, 5.07, 3.44 [11:49:09] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.04823416471 secs [11:56:52] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:58:19] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:58:54] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [12:03:09] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.1146575511 secs [12:05:09] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.08045238256 secs [12:05:51] PROBLEM - es141 PowerDNS Recursor on es141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:07:50] RECOVERY - es141 PowerDNS Recursor on es141 is OK: DNS OK: 2.320 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [12:10:42] PROBLEM - cp32 APT on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:12:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.87, 3.65, 2.55 [12:14:09] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.57, 3.32, 2.57 [12:15:43] RECOVERY - cp32 APT on cp32 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [12:44:41] PROBLEM - wiki.joust.ro - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.joust.ro All nameservers failed to answer the query. [12:57:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.14, 11.02, 9.74 [12:58:19] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [12:59:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.17, 11.07, 9.92 [13:00:53] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [13:03:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.98, 9.64, 9.61 [13:05:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.90, 10.04, 9.01 [13:07:03] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.85, 5.71, 3.70 [13:07:05] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.56, 8.67, 8.64 [13:11:01] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.49, 5.65, 4.23 [13:14:03] RECOVERY - wiki.joust.ro - reverse DNS on sslhost is OK: SSL OK - wiki.joust.ro reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:29:46] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:51:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.71, 9.61, 8.44 [13:55:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.43, 12.41, 9.96 [13:59:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.49, 11.33, 10.11 [14:00:56] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.34, 10.16, 9.38 [14:01:28] [02miraheze/ImportDump] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-phan-config-0.12.1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/361366056295 [14:01:31] [02miraheze/ImportDump] 07dependabot[bot] 033613660 - Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 [14:01:34] [02ImportDump] 07dependabot[bot] labeled pull request 03#40: Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/ImportDump/pull/40 [14:01:35] [02ImportDump] 07dependabot[bot] labeled pull request 03#40: Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/ImportDump/pull/40 [14:01:38] [02ImportDump] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-phan-config-0.12.1 - 13https://github.com/miraheze/ImportDump [14:01:40] [02ImportDump] 07dependabot[bot] opened pull request 03#40: Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/ImportDump/pull/40 [14:02:20] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [14:03:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.56, 11.37, 10.36 [14:04:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.14, 11.00, 9.87 [14:05:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.51, 11.54, 10.56 [14:06:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.33, 9.94, 9.61 [14:07:21] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.30, 11.65, 10.71 [14:09:23] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.45, 3.22, 2.90 [14:12:14] miraheze/ImportDump - dependabot[bot] the build passed. [14:15:21] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.19, 3.61, 3.17 [14:17:21] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.60, 3.62, 3.23 [14:19:21] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.42, 4.03, 3.43 [14:23:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.37, 11.58, 11.80 [14:25:08] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.08, 5.47, 3.33 [14:25:21] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.61, 3.77, 3.50 [14:32:20] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [14:33:02] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.56, 6.60, 5.16 [14:35:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.19, 11.73, 11.43 [14:35:21] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.84, 3.70, 3.53 [14:36:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.72, 10.40, 9.82 [14:37:21] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.70, 3.70, 3.55 [14:39:21] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.15, 3.21, 3.39 [14:40:00] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.71, 12.29, 10.62 [14:43:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.97, 10.76, 10.46 [14:45:53] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.15, 9.94, 10.19 [14:47:10] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.59, 10.68, 11.73 [14:49:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.20, 12.03, 12.11 [14:51:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.69, 10.97, 11.71 [14:56:20] [02mw-config] 07redbluegreenhat closed pull request 03#5195: $smwgNamespacesWithSemanticLinks changes on sagan4alphawiki - 13https://github.com/miraheze/mw-config/pull/5195 [15:03:31] PROBLEM - deepdark.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for deepdark.wiki could not be found [15:04:07] [02PDFEmbed] 07dependabot[bot] opened pull request 03#14: Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/PDFEmbed/pull/14 [15:04:21] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:05:12] miraheze/PDFEmbed - dependabot[bot] the build passed. [15:05:37] [02PDFEmbed] 07dependabot[bot] labeled pull request 03#14: Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/PDFEmbed/pull/14 [15:07:04] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.32, 8.36, 10.07 [15:07:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.16, 10.32, 9.52 [15:09:09] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1779869497 secs [15:09:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.78, 10.06, 9.51 [15:11:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.32, 10.46, 10.47 [15:12:15] [02miraheze/PDFEmbed] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-phan-config-0.12.1 [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/commit/b681a78ed245 [15:12:18] [02miraheze/PDFEmbed] 07dependabot[bot] 03b681a78 - Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 [15:12:49] [02PDFEmbed] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-phan-config-0.12.1 - 13https://github.com/miraheze/PDFEmbed [15:13:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.52, 11.03, 10.70 [15:13:17] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 10.16, 9.85, 9.50 [15:17:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.30, 11.40, 10.78 [15:19:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.74, 11.10, 10.78 [15:25:52] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.72, 3.24, 2.25 [15:26:00] PROBLEM - deepdark.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address deepdark.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:26:59] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.06, 9.69, 10.18 [15:27:01] PROBLEM - studio.niuboss123.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query studio.niuboss123.com. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:27:52] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.76, 2.91, 2.24 [15:31:09] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.05055844784 secs [15:31:24] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.68, 3.33, 3.05 [15:33:21] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.36, 2.90, 2.92 [15:34:22] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [15:43:24] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.36, 7.11, 4.38 [15:51:19] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.22, 6.09, 5.55 [15:51:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.30, 10.55, 9.89 [15:53:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.97, 11.03, 9.28 [15:53:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.64, 10.39, 9.95 [15:55:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.47, 10.62, 9.34 [15:56:30] PROBLEM - studio.niuboss123.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - studio.niuboss123.com All nameservers failed to answer the query. [15:57:49] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.16, 11.78, 10.54 [15:59:18] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.11, 9.16, 9.11 [16:01:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.47, 11.62, 10.75 [16:06:14] [02miraheze/GlobalNewFiles] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-phan-config-0.12.1 [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/commit/141dd4832a04 [16:06:15] [02miraheze/GlobalNewFiles] 07dependabot[bot] 03141dd48 - Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 [16:06:16] [02GlobalNewFiles] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-phan-config-0.12.1 - 13https://github.com/miraheze/GlobalNewFiles [16:06:19] [02GlobalNewFiles] 07dependabot[bot] opened pull request 03#42: Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/GlobalNewFiles/pull/42 [16:06:20] [02GlobalNewFiles] 07dependabot[bot] labeled pull request 03#42: Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/GlobalNewFiles/pull/42 [16:06:22] [02GlobalNewFiles] 07dependabot[bot] labeled pull request 03#42: Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/GlobalNewFiles/pull/42 [16:09:55] miraheze/GlobalNewFiles - dependabot[bot] the build passed. [16:15:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.62, 12.27, 11.28 [16:23:51] [02ManageWiki] 07Reception123 closed pull request 03#392: Add support for using name instead of var in extensions cache - 13https://github.com/miraheze/ManageWiki/pull/392 [16:23:52] [02miraheze/ManageWiki] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/724602d892a5...3cd468661061 [16:23:54] [02miraheze/ManageWiki] 07Universal-Omega 033cd4686 - Add support for using name instead of var in extensions cache (#392) [16:26:01] RECOVERY - studio.niuboss123.com - reverse DNS on sslhost is OK: SSL OK - studio.niuboss123.com reverse DNS resolves to cp23.miraheze.org - CNAME OK [16:27:08] miraheze/ManageWiki - Reception123 the build passed. [16:27:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.29, 10.89, 10.25 [16:29:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.40, 11.44, 10.53 [16:29:42] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.52, 3.12, 2.98 [16:30:53] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:31:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.82, 11.17, 10.54 [16:33:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.28, 11.74, 10.81 [16:37:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.44, 10.77, 10.70 [16:37:30] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.73, 3.24, 3.14 [16:45:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.10, 9.49, 10.13 [16:57:09] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1538678408 secs [16:58:50] [02miraheze/MatomoAnalytics] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-phan-config-0.12.1 [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/commit/17efb46ab6de [16:58:52] [02miraheze/MatomoAnalytics] 07dependabot[bot] 0317efb46 - Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 [16:58:53] [02MatomoAnalytics] 07dependabot[bot] opened pull request 03#98: Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/98 [16:58:56] [02MatomoAnalytics] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-phan-config-0.12.1 - 13https://github.com/miraheze/MatomoAnalytics [16:58:59] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#98: Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/98 [16:59:02] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#98: Update mediawiki/mediawiki-phan-config requirement from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/98 [16:59:29] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1647795737 secs [17:02:48] miraheze/MatomoAnalytics - dependabot[bot] the build passed. [17:03:52] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.85, 4.17, 2.85 [17:04:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.30, 10.83, 10.22 [17:05:29] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.05497789383 secs [17:05:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.76, 11.24, 11.93 [17:05:52] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.91, 3.22, 2.66 [17:06:47] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.20, 6.99, 3.87 [17:08:47] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.09, 6.30, 4.03 [17:11:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.90, 12.37, 12.06 [17:18:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.21, 11.17, 10.62 [17:20:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.81, 10.32, 10.36 [17:22:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.92, 9.85, 10.20 [17:23:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.80, 11.10, 11.90 [17:28:53] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [17:29:09] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09186497331 secs [17:33:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.45, 10.70, 11.18 [17:34:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.04, 11.14, 10.35 [17:36:44] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.11, 3.27, 2.52 [17:37:55] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.05, 3.54, 3.15 [17:38:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.08, 11.29, 10.62 [17:38:42] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.95, 3.25, 2.59 [17:39:52] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.72, 3.49, 3.17 [17:40:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.00, 2.70, 2.46 [17:45:42] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.26, 3.20, 3.20 [17:49:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.05, 11.67, 11.94 [17:56:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.87, 10.05, 10.17 [17:58:53] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [18:01:43] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.99, 7.53, 4.37 [18:01:59] PROBLEM - es141 Current Load on es141 is CRITICAL: CRITICAL - load average: 5.88, 3.60, 1.99 [18:02:58] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.04, 2.73, 1.59 [18:04:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.51, 2.99, 1.83 [18:05:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.86, 11.03, 11.13 [18:06:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.02, 10.89, 10.45 [18:06:58] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.68, 3.65, 2.21 [18:07:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.27, 9.98, 10.77 [18:07:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-1/±0] 13https://github.com/miraheze/puppet/commit/5190c0a5dcb7 [18:07:44] [02miraheze/puppet] 07paladox 035190c0a - Switch es131/141 to os131/141 [18:07:46] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [18:07:48] [02puppet] 07paladox opened pull request 03#3201: Switch es131/141 to os131/141 - 13https://github.com/miraheze/puppet/pull/3201 [18:08:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-1/±0] 13https://github.com/miraheze/puppet/compare/5190c0a5dcb7...95b1352ad5c8 [18:08:07] [02miraheze/puppet] 07paladox 0395b1352 - Rename es141.yaml to os141.yaml [18:08:08] [02puppet] 07paladox synchronize pull request 03#3201: Switch es131/141 to os131/141 - 13https://github.com/miraheze/puppet/pull/3201 [18:08:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/95b1352ad5c8...ae26d5b43eea [18:08:25] [02miraheze/puppet] 07paladox 03ae26d5b - Update os141.yaml [18:08:26] [02puppet] 07paladox synchronize pull request 03#3201: Switch es131/141 to os131/141 - 13https://github.com/miraheze/puppet/pull/3201 [18:08:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.46, 9.58, 10.00 [18:08:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ae26d5b43eea...c3832fced65a [18:08:43] [02miraheze/puppet] 07paladox 03c3832fc - Update site.pp [18:08:46] [02puppet] 07paladox synchronize pull request 03#3201: Switch es131/141 to os131/141 - 13https://github.com/miraheze/puppet/pull/3201 [18:09:29] !log reinstall es131/141 as os131/141 [18:09:50] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.014 second response time [18:10:22] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:24] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:11:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:11:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.57, 7.58, 9.61 [18:11:44] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/b3dccbe4a7de [18:11:46] [02miraheze/dns] 07paladox 03b3dccbe - Switch to using opensearch [18:11:47] [02dns] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/dns [18:11:49] [02dns] 07paladox opened pull request 03#406: Switch to using opensearch - 13https://github.com/miraheze/dns/pull/406 [18:12:28] [02dns] 07paladox closed pull request 03#406: Switch to using opensearch - 13https://github.com/miraheze/dns/pull/406 [18:12:30] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/791227c378c6...bc037a3c2472 [18:12:31] [02miraheze/dns] 07paladox 03bc037a3 - Switch to using opensearch (#406) [18:12:33] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-2 [18:12:34] [02dns] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/dns [18:13:13] PROBLEM - es141 APT on es141 is CRITICAL: connect to address 2a10:6740::6:501 port 5666: No route to hostconnect to host 2a10:6740::6:501 port 5666: No route to host [18:13:17] PROBLEM - es131 Puppet on es131 is CRITICAL: connect to address 2a10:6740::6:401 port 5666: No route to hostconnect to host 2a10:6740::6:401 port 5666: No route to host [18:13:17] PROBLEM - Host es131 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:401) [18:13:19] PROBLEM - es141 NTP time on es141 is CRITICAL: connect to address 2a10:6740::6:501 port 5666: No route to hostconnect to host 2a10:6740::6:501 port 5666: No route to host [18:13:19] PROBLEM - es141 conntrack_table_size on es141 is CRITICAL: connect to address 2a10:6740::6:501 port 5666: No route to hostconnect to host 2a10:6740::6:501 port 5666: No route to host [18:13:39] PROBLEM - es141 Disk Space on es141 is CRITICAL: connect to address 2a10:6740::6:501 port 5666: No route to hostconnect to host 2a10:6740::6:501 port 5666: No route to host [18:13:51] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.031 second response time [18:14:32] PROBLEM - es141 Puppet on es141 is CRITICAL: connect to address 2a10:6740::6:501 port 5666: No route to hostconnect to host 2a10:6740::6:501 port 5666: No route to host [18:14:32] PROBLEM - es141 SSH on es141 is CRITICAL: connect to address 2a10:6740::6:501 and port 22: No route to host [18:14:36] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.576 second response time [18:14:47] PROBLEM - es141 PowerDNS Recursor on es141 is CRITICAL: connect to address 2a10:6740::6:501 port 5666: No route to hostconnect to host 2a10:6740::6:501 port 5666: No route to host [18:14:47] PROBLEM - es141 ferm_active on es141 is CRITICAL: connect to address 2a10:6740::6:501 port 5666: No route to hostconnect to host 2a10:6740::6:501 port 5666: No route to host [18:14:54] PROBLEM - ping6 on es141 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:501) [18:15:16] PROBLEM - Host es141 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:501) [18:15:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0ef31f3fee80...0a09e673f5a9 [18:15:29] [02miraheze/puppet] 07paladox 030a09e67 - graylog: switch to opensearch [18:19:16] RECOVERY - Host es131 is UP: PING OK - Packet loss = 0%, RTA = 1.38 ms [18:19:30] PROBLEM - es131 ferm_active on es131 is CRITICAL: connect to address 2a10:6740::6:401 port 5666: Connection refusedconnect to host 2a10:6740::6:401 port 5666: Connection refused PROBLEM - es131 NTP time on es131 is CRITICAL: connect to address 2a10:6740::6:401 port 5666: Connection refusedconnect to host 2a10:6740::6:401 port 5666: Connection refused [18:19:30] PROBLEM - es131 Current Load on es131 is CRITICAL: connect to address 2a10:6740::6:401 port 5666: Connection refusedconnect to host 2a10:6740::6:401 port 5666: Connection refused [18:19:30] PROBLEM - es131 SSH on es131 is CRITICAL: connect to address 2a10:6740::6:401 and port 22: Connection refused [18:19:40] PROBLEM - es131 Disk Space on es131 is CRITICAL: connect to address 2a10:6740::6:401 port 5666: Connection refusedconnect to host 2a10:6740::6:401 port 5666: Connection refused [18:19:55] PROBLEM - es131 APT on es131 is CRITICAL: connect to address 2a10:6740::6:401 port 5666: Connection refusedconnect to host 2a10:6740::6:401 port 5666: Connection refused [18:20:20] PROBLEM - es131 PowerDNS Recursor on es131 is CRITICAL: connect to address 2a10:6740::6:401 port 5666: Connection refusedconnect to host 2a10:6740::6:401 port 5666: Connection refused [18:20:20] PROBLEM - es131 conntrack_table_size on es131 is CRITICAL: connect to address 2a10:6740::6:401 port 5666: Connection refusedconnect to host 2a10:6740::6:401 port 5666: Connection refused [18:22:56] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:31] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw121 [18:24:50] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.426 second response time [18:26:22] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw121 [18:26:31] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [18:27:27] RECOVERY - es131 SSH on es131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:28:22] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [18:30:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 1.59, 2.74, 3.92 [18:31:17] RECOVERY - Host es141 is UP: PING OK - Packet loss = 0%, RTA = 0.38 ms [18:33:15] RECOVERY - ping6 on es141 is OK: PING OK - Packet loss = 0%, RTA = 0.33 ms [18:35:16] PROBLEM - graylog131 Puppet on graylog131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:36:58] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.34, 1.96, 3.18 [18:37:05] RECOVERY - es131 APT on es131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:37:14] RECOVERY - es131 PowerDNS Recursor on es131 is OK: DNS OK: 0.149 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [18:37:15] PROBLEM - es131 conntrack_table_size on es131 is UNKNOWN: NRPE: Unable to read output [18:37:54] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.007 second response time [18:38:04] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1429393291 secs [18:38:08] PROBLEM - es131 Puppet on es131 is UNKNOWN: NRPE: Unable to read output [18:38:17] RECOVERY - es131 NTP time on es131 is OK: NTP OK: Offset -0.007455289364 secs [18:38:20] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.009 second response time [18:38:22] PROBLEM - es131 ferm_active on es131 is UNKNOWN: NRPE: Unable to read output [18:38:26] RECOVERY - es131 Current Load on es131 is OK: OK - load average: 0.15, 0.42, 0.27 [18:38:34] RECOVERY - es131 Disk Space on es131 is OK: DISK OK - free space: / 197341 MB (98% inode=99%); [18:38:45] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 4.39, 3.30, 2.47 [18:38:48] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:16] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.006 second response time [18:39:19] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 2 backends are down. mw131 mw142 [18:39:49]