[00:05:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.35, 3.90, 3.98 [00:09:29] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.08, 3.37, 3.69 [00:21:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.27, 3.78, 4.00 [00:23:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.88, 5.07, 4.42 [00:30:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.15, 3.41, 1.96 [00:32:42] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.14, 2.53, 1.81 [00:35:16] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.84, 6.52, 4.50 [00:37:16] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 6.37, 6.45, 4.73 [00:41:03] PROBLEM - wiki.yuanpi.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yuanpi.eu.org could not be found [00:41:16] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.80, 8.00, 5.81 [00:43:16] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 13.63, 9.56, 6.62 [00:46:55] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zymonic.com All nameservers failed to answer the query. [00:47:16] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.28, 7.77, 6.68 [00:49:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.80, 3.20, 4.00 [00:51:16] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.76, 6.30, 6.32 [00:55:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.01, 5.14, 4.46 [01:01:16] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.72, 6.88, 6.64 [01:03:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.18, 3.19, 3.82 [01:03:16] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.35, 7.70, 6.97 [01:09:36] PROBLEM - es141 Current Load on es141 is WARNING: WARNING - load average: 3.25, 3.44, 3.04 [01:09:37] RECOVERY - wiki.yuanpi.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.yuanpi.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [01:11:36] PROBLEM - es141 Current Load on es141 is CRITICAL: CRITICAL - load average: 4.01, 3.86, 3.25 [01:13:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.70, 2.70, 3.28 [01:13:36] PROBLEM - es141 Current Load on es141 is WARNING: WARNING - load average: 3.29, 3.54, 3.20 [01:15:16] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.01, 7.61, 7.97 [01:15:36] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 1.85, 2.94, 3.02 [01:16:43] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.40469765663147 seconds [01:21:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.96, 4.36, 3.74 [01:21:16] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.70, 4.80, 6.64 [01:27:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.86, 3.94, 3.86 [01:28:42] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.77, 2.24, 1.36 [01:30:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.58, 3.22, 1.81 [01:32:42] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.16, 2.34, 1.67 [01:36:37] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [01:41:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.91, 3.05, 3.35 [02:05:14] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:36:11] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.zulaclub.net. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:49:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.93, 3.53, 2.99 [02:51:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.44, 4.59, 3.49 [03:00:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.20, 3.08, 1.65 [03:01:30] RECOVERY - db131 Backups SQL on db131 is OK: FILE_AGE OK: /var/log/sql-backup.log is 88 seconds old and 275 bytes [03:02:42] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.42, 3.04, 1.83 [03:06:09] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [03:16:55] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/858eaaa7a0e5 [03:16:58] [02miraheze/puppet] 07The-Voidwalker 03858eaaa - add chron to optimize matomo tables [03:17:01] [02puppet] 07The-Voidwalker created branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/puppet [03:17:02] [02puppet] 07The-Voidwalker opened pull request 03#3178: add chron to optimize matomo tables - 13https://github.com/miraheze/puppet/pull/3178 [03:19:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.25, 3.56, 4.00 [03:19:17] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/858eaaa7a0e5...9a0a795d2e33 [03:19:19] [02miraheze/puppet] 07The-Voidwalker 039a0a795 - Create optimizeMatomoTables.sh [03:19:21] [02puppet] 07The-Voidwalker synchronize pull request 03#3178: add chron to optimize matomo tables - 13https://github.com/miraheze/puppet/pull/3178 [03:23:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.09, 4.02, 4.03 [03:25:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.31, 3.56, 3.85 [03:31:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.68, 3.70, 3.73 [03:35:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.01, 3.57, 3.69 [03:39:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.22, 2.75, 3.32 [03:42:01] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.25, 3.43, 2.77 [03:44:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.31, 2.69, 2.57 [03:55:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.47, 3.80, 3.46 [03:57:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.37, 3.11, 3.24 [04:00:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.07, 3.36, 1.85 [04:01:06] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:02:42] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.50, 3.34, 2.06 [04:03:06] RECOVERY - cp23 APT on cp23 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [04:15:54] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.87, 3.42, 3.27 [04:17:49] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.33, 3.39, 3.27 [04:24:39] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1188389361 secs [04:39:06] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.zulaclub.net. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:41:13] PROBLEM - gs.sidem.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query gs.sidem.wiki. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:52:39] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08860877156 secs [04:55:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.28, 4.70, 3.19 [04:56:50] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:58:47] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [05:01:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.73, 3.99, 3.40 [05:03:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.61, 3.24, 3.20 [05:24:39] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.1129130721 secs [05:28:39] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.08727157116 secs [05:39:06] PROBLEM - gs.sidem.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - gs.sidem.wiki All nameservers failed to answer the query. [05:49:39] [02mw-config] 07AgentIsai opened pull request 03#5176: Add Google Translate to Meta translating - 13https://github.com/miraheze/mw-config/pull/5176 [05:50:39] miraheze/mw-config - AgentIsai the build passed. [05:51:05] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [05:54:25] [02puppet] 07AgentIsai opened pull request 03#3179: Add API key to PrivateSettings - 13https://github.com/miraheze/puppet/pull/3179 [05:56:36] [02puppet] 07AgentIsai synchronize pull request 03#3179: Add API key to PrivateSettings - 13https://github.com/miraheze/puppet/pull/3179 [05:57:01] [02mw-config] 07Reception123 closed pull request 03#5176: Add Google Translate to Meta translating - 13https://github.com/miraheze/mw-config/pull/5176 [05:57:03] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4ab2c3b6e93b...93081fd83c36 [05:57:05] [02miraheze/mw-config] 07AgentIsai 0393081fd - Add Google Translate to Meta translating (#5176) [05:57:05] !log [@test131] starting deploy of {'config': True} to all [05:57:06] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:57:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:57:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:57:33] [02puppet] 07Reception123 closed pull request 03#3179: Add API key to PrivateSettings - 13https://github.com/miraheze/puppet/pull/3179 [05:57:36] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/10f19f8822ce...e38ac3d39baf [05:57:38] [02miraheze/puppet] 07AgentIsai 03e38ac3d - Add API key to PrivateSettings (#3179) [05:58:01] miraheze/mw-config - Reception123 the build passed. [05:59:06] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [05:59:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:06:26] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1059807539 secs [06:08:55] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [06:09:03] RECOVERY - gs.sidem.wiki - reverse DNS on sslhost is OK: SSL OK - gs.sidem.wiki reverse DNS resolves to cp23.miraheze.org - CNAME OK [06:20:19] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [06:20:37] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [06:20:38] !log [reception@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [06:20:39] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [06:20:40] !log [reception@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [06:20:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:20:45] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [06:20:46] !log [reception@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [06:20:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:20:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:20:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:20:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:21:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:22:26] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.07495987415 secs [06:22:48] !log [@mwtask141] starting deploy of {'config': True} to all [06:22:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:22:56] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [06:23:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:23:50] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [06:23:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:23:57] !log [reception@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 6s [06:24:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:52] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.zulaclub.net. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:56:02] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.52, 3.12, 2.32 [06:57:58] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.61, 3.25, 2.50 [07:08:49] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [07:30:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.76, 3.54, 2.11 [07:32:42] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.54, 2.67, 1.97 [07:54:35] PROBLEM - db142 Backups SQL on db142 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209711 seconds old and 1753 bytes [08:00:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.20, 4.20, 2.63 [08:01:29] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:03:31] RECOVERY - cp23 APT on cp23 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [08:06:42] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.92, 3.64, 3.20 [08:08:42] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.95, 2.78, 2.94 [08:10:28] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=constantnoblewiki --e 1 -- 1000 [08:10:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:10:58] fuck, wrong syntax [08:11:16] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=constantnoblewiki 1 -- 1000 [08:11:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:12:34] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=constantnoblewiki --e 3800 -- 2800 [08:12:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:18:56] well, I finally understand the syntax, I'll have to start again from zero and just avoid the 2800-3800 range [08:36:58] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.131542325 secs [08:44:58] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.07706171274 secs [09:43:11] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zymonic.com All nameservers failed to answer the query. [09:58:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.99, 4.03, 2.20 [10:00:42] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.86, 3.14, 2.10 [10:12:59] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.4053122997283936 seconds [10:28:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.57, 3.64, 2.25 [10:32:42] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.41, 2.71, 2.21 [10:51:16] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.36, 7.02, 3.64 [10:53:16] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.86, 5.08, 3.34 [11:15:36] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [11:30:42] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.78, 3.32, 2.25 [11:32:42] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.71, 2.70, 2.15 [11:44:50] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:14:04] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [12:25:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.16, 3.46, 2.46 [12:27:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.79, 3.72, 2.69 [12:28:10] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:30:19] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [12:31:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.92, 5.26, 3.54 [12:35:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.46, 3.31, 3.14 [12:39:16] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.23, 4.97, 2.73 [12:43:04] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:43:16] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.64, 5.09, 3.42 [12:45:03] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [12:51:56] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=constantnoblewiki --e 1000 -- 1 [12:52:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:54:55] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.20, 4.04, 3.29 [13:04:11] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:06:08] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [13:10:22] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.92, 3.07, 3.85 [13:11:41] RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: SSL OK - wiki.zymonic.com reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:12:18] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.84, 3.45, 3.86 [13:16:55] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:18:52] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [13:25:49] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.59, 3.34, 3.89 [13:26:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/9ce98dbc0e07 [13:26:09] [02miraheze/puppet] 07paladox 039ce98db - Fix wmgTranslateGoogleTranslateMetaKey in PrivateSettings [13:26:11] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [13:26:12] [02puppet] 07paladox opened pull request 03#3180: Fix wmgTranslateGoogleTranslateMetaKey in PrivateSettings - 13https://github.com/miraheze/puppet/pull/3180 [13:26:33] [02puppet] 07paladox closed pull request 03#3180: Fix wmgTranslateGoogleTranslateMetaKey in PrivateSettings - 13https://github.com/miraheze/puppet/pull/3180 [13:26:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e38ac3d39baf...c6d77ccf0f8e [13:26:38] [02miraheze/puppet] 07paladox 03c6d77cc - Fix wmgTranslateGoogleTranslateMetaKey in PrivateSettings (#3180) [13:26:41] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [13:26:42] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [13:27:45] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.60, 4.30, 4.15 [13:30:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.06, 5.72, 3.34 [13:31:54] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=constantnoblewiki --e 2000 -- 1000 [13:31:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:33:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 0.89, 3.20, 3.93 [13:34:42] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.79, 2.92, 2.72 [13:37:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.35, 2.33, 3.38 [13:38:23] You know, I didn't expect refreshLinks to be this slow [13:38:53] At this rate it will take 7+ hours to run through all of that wiki's pages [13:39:32] Some shortcuts come to mind, maybe I can only run it on the page ids that Routhwick pointed out were misbehaving? [13:39:51] Or just run the rest tomorrow? [13:39:58] Reception123: ^ [13:40:37] Orange_Star: you can just use screen before [13:40:45] So it can be left running even after you leave [13:40:54] that too [13:41:01] That's what we usually do as refreshLinks can take a very long time [13:55:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.17, 3.87, 3.22 [13:55:43] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:57:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.01, 3.56, 3.19 [13:58:01] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:00:43] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 19.21, 10.23, 5.13 [14:01:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.64, 4.32, 3.56 [14:01:48] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:02:53] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [14:03:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.98, 3.41, 3.32 [14:03:48] most of them are semantic mediawiki jobs related to constantnoblewiki [14:03:49] RECOVERY - cp23 APT on cp23 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [14:05:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.35, 2.72, 3.08 [14:05:32] more like the vast majority [14:08:42] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.13, 3.40, 3.83 [14:09:42] current batch has finished, should I continue running refreshLinks? [14:12:42] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.44, 2.59, 3.38 [14:23:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.95, 3.12, 2.72 [14:25:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.58, 2.89, 2.69 [14:28:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 10.43, 5.10, 3.37 [14:30:42] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.15, 3.81, 3.11 [14:31:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.94, 3.70, 3.18 [14:32:42] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.08, 2.86, 2.85 [14:33:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.51, 3.05, 3.01 [14:35:39] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:41:04] PROBLEM - wiki.yuanpi.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yuanpi.eu.org could not be found [14:55:00] Sure [14:58:32] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.55, 4.07, 3.27 [15:00:28] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.70, 3.15, 3.02 [15:06:37] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:09:16] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.31, 6.73, 3.76 [15:10:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:13:16] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.04, 6.66, 4.54 [15:15:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.00, 4.79, 3.52 [15:19:48] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.82, 3.46, 3.26 [15:21:44] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.67, 3.07, 3.13 [15:31:30] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.03, 3.75, 2.55 [15:33:24] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.84, 2.70, 2.30 [15:35:13] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [15:35:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:43:16] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1230193973 secs [15:49:29] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [15:50:58] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.2557299137 secs [15:53:29] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 7 = Critical] [15:53:43] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.21, 4.51, 3.21 [15:59:31] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.68, 3.30, 3.15 [16:04:37] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [16:06:58] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.02014702559 secs [16:10:30] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:17:18] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.0903506875 secs [16:18:06] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/93081fd83c36...d934cde73889 [16:18:09] [02miraheze/mw-config] 07Reception123 03d934cde - rm global group perms from betawiki crats [16:18:28] !log [reception@test131] starting deploy of {'pull': 'config', 'config': True} to all [16:18:29] !log [reception@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [16:18:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:18:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:19:06] miraheze/mw-config - Reception123 the build passed. [16:28:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.40, 2.91, 1.92 [16:30:30] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:30:42] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 3.20, 2.99, 2.08 [16:39:56] !log [@mwtask141] starting deploy of {'config': True} to all [16:40:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:40:04] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [16:40:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:40:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.96, 9.77, 8.21 [16:42:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.95, 9.11, 8.16 [16:44:01] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:53:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.37, 3.34, 2.65 [16:55:16] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.18, 3.36, 2.75 [16:59:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.89, 3.66, 3.02 [17:01:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.69, 5.42, 3.80 [17:05:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.06, 3.52, 3.38 [17:07:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.43, 3.25, 3.30 [17:13:15] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [17:17:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.97, 3.56, 3.26 [17:19:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.49, 3.35, 3.24 [17:26:08] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.59, 3.74, 3.48 [17:30:00] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.28, 3.86, 3.58 [17:33:52] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.37, 3.26, 3.49 [17:35:48] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.75, 2.73, 3.26 [17:57:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.31, 3.33, 2.70 [17:59:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.87, 2.99, 2.64 [18:01:54] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=constantnoblewiki 2000 (START) [18:01:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:05:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.38, 3.52, 2.90 [18:07:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.99, 3.17, 2.87 [18:17:00] gosh: https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki?orgId=1 [18:19:21] I actually wonder if this is just SMW's normal reaction to refreshLinks or what. [18:45:37] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [18:47:08] It's running in a screen, if it affects regular wiki tasks ping me nonstop until I kill it. [19:00:39] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.2717772424 secs [19:06:37] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:06:39] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.0235106349 secs [19:25:47] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.11, 3.73, 2.73 [19:29:38] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.91, 3.03, 2.68 [19:31:31] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.224912852 secs [19:38:46] [02miraheze/mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d934cde73889...5047121b0189 [19:38:49] [02miraheze/mw-config] 07redbluegreenhat 035047121 - Disable Cargo [19:39:44] miraheze/mw-config - redbluegreenhat the build passed. [19:40:18] !log [@mwtask141] starting deploy of {'config': True} to all [19:40:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:24] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [19:40:28] !log [alex@mwtask141] starting deploy of {'config': True} to all [19:40:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:34] !log [alex@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [19:40:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:41:31] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.01676052809 secs [19:47:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.59, 3.84, 2.94 [19:49:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.41, 2.89, 2.70 [19:57:05] !log [@test131] starting deploy of {'config': True} to all [19:57:06] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [19:57:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.96, 3.73, 3.12 [19:59:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.55, 3.31, 3.04 [20:04:36] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [20:20:07] [02mw-config] 07AgentIsai opened pull request 03#5177: Sitenotice for Cargo being disabled - 13https://github.com/miraheze/mw-config/pull/5177 [20:20:46] [02mw-config] 07AgentIsai synchronize pull request 03#5177: Sitenotice for Cargo being disabled - 13https://github.com/miraheze/mw-config/pull/5177 [20:21:03] miraheze/mw-config - AgentIsai the build passed. [20:22:01] miraheze/mw-config - AgentIsai the build passed. [20:26:28] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 130.15 ms [20:28:58] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1371279061 secs [20:35:04] [02mw-config] 07MacFan4000 closed pull request 03#5177: Sitenotice for Cargo being disabled - 13https://github.com/miraheze/mw-config/pull/5177 [20:35:06] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5047121b0189...90884704a667 [20:35:08] [02miraheze/mw-config] 07AgentIsai 039088470 - Sitenotice for Cargo being disabled (#5177) [20:36:02] miraheze/mw-config - MacFan4000 the build passed. [20:40:25] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 104.36 ms [20:40:27] !log [@mwtask141] starting deploy of {'config': True} to all [20:40:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:40:33] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [20:40:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:41:48] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:42:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.69, 9.36, 8.00 [20:44:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.88, 8.80, 7.97 [20:46:40] !log [@test131] starting deploy of {'config': True} to all [20:46:41] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [20:46:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:46:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:50:37] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [20:53:06] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/90884704a667...752afb7e7d88 [20:53:09] [02miraheze/mw-config] 07MacFan4000 03752afb7 - Bump Sitenotice ID [20:54:09] miraheze/mw-config - MacFan4000 the build passed. [20:56:12] !log [@test131] starting deploy of {'config': True} to all [20:56:13] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [20:56:14] !log [@mwtask141] starting deploy of {'config': True} to all [20:56:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:21] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [20:56:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:02:58] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.07999834418 secs [21:09:43] [02miraheze/mediawiki] 07The-Voidwalker pushed 031 commit to 03REL1_39 [+0/-0/±2] 13https://github.com/miraheze/mediawiki/compare/2aeaefb0c1b6...a0b5eec8e8b2 [21:09:44] [02miraheze/mediawiki] 07The-Voidwalker 03a0b5eec - update CheckUser and GrowthExperiments [21:11:02] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [21:42:38] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.56, 3.14, 2.71 [21:44:33] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.86, 2.97, 2.70 [21:44:58] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1125166714 secs [22:03:57] [02miraheze/mediawiki] 07The-Voidwalker pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/a0b5eec8e8b2...a44a6f1f9bfb [22:03:59] [02miraheze/mediawiki] 07The-Voidwalker 03a44a6f1 - update CheckUser [22:08:58] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08085781336 secs [22:25:34] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03bump-auth-token [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/6a36460cd45e [22:25:37] [02miraheze/mw-config] 07MacFan4000 036a36460 - bump auth token version [22:25:38] [02mw-config] 07MacFan4000 created branch 03bump-auth-token - 13https://github.com/miraheze/mw-config [22:25:41] [02mw-config] 07MacFan4000 opened pull request 03#5178: bump auth token version - 13https://github.com/miraheze/mw-config/pull/5178 [22:26:05] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03bump-auth-token [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6a36460cd45e...22d7f518cc36 [22:26:08] [02miraheze/mw-config] 07github-actions 0322d7f51 - CI: lint code to MediaWiki standards [22:26:11] [02mw-config] 07github-actions[bot] synchronize pull request 03#5178: bump auth token version - 13https://github.com/miraheze/mw-config/pull/5178 [22:26:38] miraheze/mw-config - MacFan4000 the build passed. [22:42:39] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1073580682 secs [22:52:39] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07974007726 secs [23:11:53] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [23:16:39] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.3070254922 secs [23:26:39] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.02121067047 secs [23:36:01] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.70, 3.15, 2.43 [23:38:01] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.16, 3.44, 2.62 [23:40:01] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.78, 2.80, 2.49 [23:41:07] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [23:52:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.03, 3.31, 2.82 [23:58:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.98, 3.74, 3.21