[00:02:33] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 6.32, 4.59, 2.22 [00:04:01] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 8.24, 9.43, 4.68 [00:06:32] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 1.38, 3.33, 2.29 [00:19:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: LOAD CRITICAL - total load average: 3.56, 4.67, 2.96 [00:21:25] PROBLEM - cp33 Current Load on cp33 is WARNING: LOAD WARNING - total load average: 2.52, 3.80, 2.84 [00:23:25] RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 1.52, 3.19, 2.75 [00:26:01] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 0.46, 1.42, 3.60 [00:28:01] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 0.27, 1.04, 3.20 [01:04:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: LOAD CRITICAL - total load average: 9.10, 6.50, 3.83 [01:07:29] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.47, 3.12, 2.81 [01:08:32] PROBLEM - cp32 Current Load on cp32 is WARNING: LOAD WARNING - total load average: 1.12, 3.55, 3.22 [01:09:25] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.25, 2.81, 2.73 [01:10:32] RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 2.53, 3.17, 3.12 [01:13:22] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.43, 3.35, 2.98 [01:15:18] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.88, 3.20, 2.98 [01:29:23] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:43:56] PROBLEM - bast121 Current Load on bast121 is CRITICAL: LOAD CRITICAL - total load average: 3.33, 1.60, 0.72 [01:49:38] PROBLEM - bast121 PowerDNS Recursor on bast121 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:51:46] PROBLEM - bast121 SSH on bast121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:09] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.63, 3.39, 2.83 [01:53:39] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:54:56] PROBLEM - bast121 conntrack_table_size on bast121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:57:28] PROBLEM - bast121 ferm_active on bast121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:59:33] PROBLEM - bast121 NTP time on bast121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:00:33] PROBLEM - bast121 Disk Space on bast121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:02:13] RECOVERY - bast121 NTP time on bast121 is OK: NTP OK: Offset -0.00588285923 secs [02:02:16] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 49 minutes ago with 0 failures [02:02:16] RECOVERY - bast121 ferm_active on bast121 is OK: OK ferm input default policy is set [02:02:27] RECOVERY - bast121 Disk Space on bast121 is OK: DISK OK - free space: / 14951MiB (84% inode=93%); [02:02:39] RECOVERY - bast121 conntrack_table_size on bast121 is OK: OK: nf_conntrack is 0 % full [02:03:09] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 1.42, 3.76, 3.69 [02:03:34] RECOVERY - bast121 PowerDNS Recursor on bast121 is OK: DNS OK: 0.114 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [02:04:09] RECOVERY - bast121 SSH on bast121 is OK: SSH OK - OpenSSH_9.2p1 Debian-2 (protocol 2.0) [02:04:40] PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 3.93, 2.97, 2.22 [02:05:33] PROBLEM - mw133 Puppet on mw133 is UNKNOWN: NRPE: Unable to read output [02:06:40] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 2.36, 2.66, 2.20 [02:07:09] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 1.66, 2.57, 3.21 [02:11:13] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -0.01974862814 secs [02:15:39] PROBLEM - cp32 Current Load on cp32 is CRITICAL: LOAD CRITICAL - total load average: 2.64, 6.09, 4.07 [02:19:40] RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 0.94, 3.26, 3.35 [02:29:32] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [02:33:50] PROBLEM - bast121 Current Load on bast121 is WARNING: LOAD WARNING - total load average: 0.48, 0.22, 1.91 [02:35:50] RECOVERY - bast121 Current Load on bast121 is OK: LOAD OK - total load average: 0.08, 0.15, 1.68 [02:57:23] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:00:04] RECOVERY - db101 Backups SQL on db101 is OK: FILE_AGE OK: /var/log/sql-backup.log is 2 seconds old and 0 bytes [04:22:08] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [04:51:17] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query zulaclub.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:39:13] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1015818417 secs [05:50:10] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [06:10:06] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.03, 2.71, 1.22 [06:16:06] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.37, 3.43, 2.14 [06:18:06] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.46, 2.77, 2.05 [07:00:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.26, 3.59, 2.91 [07:02:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.67, 3.37, 2.93 [07:17:10] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [07:45:49] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [10:16:58] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.001150429249 secs [10:19:28] PROBLEM - wiki.theinverse.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.theinverse.co.uk' expires in 15 day(s) (Thu 28 Sep 2023 10:14:02 GMT +0000). [10:24:36] PROBLEM - elephants.ggreig.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'elephants.ggreig.com' expires in 15 day(s) (Thu 28 Sep 2023 10:03:13 GMT +0000). [10:29:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/14ab8057c123...63c4a95b50d2 [10:29:53] [02miraheze/ssl] 07MirahezeSSLBot 0363c4a95 - Bot: Update SSL cert for elephants.ggreig.com [10:36:52] PROBLEM - studio.niuboss123.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'studio.niuboss123.com' expires in 15 day(s) (Thu 28 Sep 2023 10:26:33 GMT +0000). [10:38:57] PROBLEM - dodgecircle.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dodgecircle.com' expires in 15 day(s) (Thu 28 Sep 2023 10:22:34 GMT +0000). [10:40:55] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 7.90, 6.86, 5.76 [10:42:49] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 4.36, 5.92, 5.55 [10:43:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/63c4a95b50d2...69a4aec3ad9f [10:43:44] [02miraheze/ssl] 07MirahezeSSLBot 0369a4aec - Bot: Update SSL cert for studio.niuboss123.com [10:46:11] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/69a4aec3ad9f...f6f27321b087 [10:46:14] [02miraheze/ssl] 07MirahezeSSLBot 03f6f2732 - Bot: Update SSL cert for dodgecircle.com [11:05:45] RECOVERY - studio.niuboss123.com - LetsEncrypt on sslhost is OK: OK - Certificate 'studio.niuboss123.com' will expire on Mon 11 Dec 2023 09:43:36 GMT +0000. [11:07:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a12b3a66a225...e4b7a8057de5 [11:07:22] [02miraheze/puppet] 07paladox 03e4b7a80 - memcached: set some sysctl to improve performance [11:08:24] RECOVERY - dodgecircle.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dodgecircle.com' will expire on Mon 11 Dec 2023 09:46:04 GMT +0000. [11:22:36] RECOVERY - elephants.ggreig.com - LetsEncrypt on sslhost is OK: OK - Certificate 'elephants.ggreig.com' will expire on Mon 11 Dec 2023 09:29:43 GMT +0000. [11:26:52] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [11:27:29] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4e4fdd6b4c4a...d8b66ab90cbc [11:27:31] [02miraheze/mw-config] 07paladox 03d8b66ab - Switch wgMessageCacheType to mem131 [11:27:51] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [11:27:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:28:00] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [11:28:07] !log [paladox@mwtask141] starting deploy of {'config': True} to all [11:28:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:28:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:28:15] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [11:28:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:28:38] miraheze/mw-config - paladox the build passed. [11:44:15] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.001639902592 secs [11:44:58] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.2850996554 secs [11:56:22] !log [@test131] starting deploy of {'config': True} to all [11:56:23] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [11:56:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:56:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:56:53] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [12:09:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.21, 3.67, 3.01 [12:21:18] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.80, 3.97, 3.60 [12:23:17] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.20, 4.09, 3.69 [12:24:35] PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 3.92, 2.49, 1.67 [12:24:45] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.09948715568 secs [12:26:32] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 0.98, 1.81, 1.52 [12:27:15] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.96, 3.99, 3.74 [12:29:14] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.04, 4.06, 3.80 [12:51:05] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.62, 3.81, 3.95 [12:53:04] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.22, 3.94, 3.98 [13:06:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.63, 3.47, 3.83 [13:16:53] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.62, 2.90, 3.35 [13:17:10] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:45:50] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:48:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.91, 3.40, 3.09 [14:04:31] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.26, 3.99, 3.65 [14:06:30] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.28, 3.73, 3.60 [14:09:11] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/commit/6a7141ed673b [14:09:14] [02miraheze/DataDump] 07paladox 036a7141e - Fix "PHP Notice: Undefined index: generatedumptype" [14:09:16] [02DataDump] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/DataDump [14:09:17] [02DataDump] 07paladox opened pull request 03#63: Fix "PHP Notice: Undefined index: generatedumptype" - 13https://github.com/miraheze/DataDump/pull/63 [14:12:01] miraheze/DataDump - paladox the build has errored. [14:15:41] [02DataDump] 07paladox synchronize pull request 03#63: Fix "PHP Notice: Undefined index: generatedumptype" - 13https://github.com/miraheze/DataDump/pull/63 [14:15:43] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/6a7141ed673b...2fd285f8007a [14:15:43] [02miraheze/DataDump] 07paladox 032fd285f - Update DataDumpPager.php [14:16:26] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.02, 3.69, 3.63 [14:18:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.58, 3.67, 3.63 [14:21:25] [02mw-config] 07R4356th opened pull request 03#5313: Change benpediawiki's custom language (T11042) - 13https://github.com/miraheze/mw-config/pull/5313 [14:21:54] miraheze/DataDump - paladox the build passed. [14:22:24] miraheze/mw-config - R4356th the build passed. [14:22:51] [02DataDump] 07paladox closed pull request 03#63: Fix "PHP Notice: Undefined index: generatedumptype" - 13https://github.com/miraheze/DataDump/pull/63 [14:22:53] [02miraheze/DataDump] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/f2d08e02f8c2...5c8d34dbde5e [14:22:55] [02miraheze/DataDump] 07paladox 035c8d34d - Fix "PHP Notice: Undefined index: generatedumptype" (#63) [14:22:57] [02miraheze/DataDump] 07paladox deleted branch 03paladox-patch-1 [14:22:59] [02DataDump] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/DataDump [14:23:34] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/192e84034cbc...9d9980d9a02a [14:23:36] [02miraheze/mediawiki] 07paladox 039d9980d - Update DataDump [14:23:47] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [14:23:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:24:16] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_40 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/6851c9502123...fa01faa04ceb [14:24:17] [02miraheze/mediawiki] 07paladox 03fa01faa - Update DataDump [14:24:33] !log [paladox@test131] starting deploy of {'world': True, 'l10n': True} to all [14:24:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.45, 3.82, 3.65 [14:29:28] [02mediawiki] 07R4356th commented on pull request 03#15526: Install RealMe - 13https://github.com/miraheze/mediawiki/pull/15526#issuecomment-1715839018 [14:30:15] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1014895439 secs [14:30:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.38, 3.66, 3.62 [14:30:24] !log [paladox@mwtask141] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 396s [14:30:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:30:41] miraheze/DataDump - paladox the build passed. [14:32:50] !log [paladox@test131] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 496s [14:32:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:40:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.68, 4.01, 3.74 [14:42:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.67, 3.84, 3.71 [14:42:44] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/commit/f0785edde93f [14:42:45] [02miraheze/DataDump] 07paladox 03f0785ed - Update [14:42:48] [02DataDump] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/DataDump [14:42:51] [02DataDump] 07paladox opened pull request 03#64: Update - 13https://github.com/miraheze/DataDump/pull/64 [14:48:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.13, 4.05, 3.84 [14:48:37] miraheze/DataDump - paladox the build passed. [14:50:21] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [14:50:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.66, 3.92, 3.81 [14:52:47] [02mw-config] 07R4356th synchronize pull request 03#5313: Change benpediawiki's custom language (T11042) - 13https://github.com/miraheze/mw-config/pull/5313 [14:53:52] miraheze/mw-config - R4356th the build passed. [14:53:54] [02mw-config] 07R4356th edited pull request 03#5313: Change benpediawiki's custom language and add one for sonaponawiki (T11042 and T11202) - 13https://github.com/miraheze/mw-config/pull/5313 [14:54:21] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [14:55:06] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/f0785edde93f...55f4c841cd17 [14:55:07] [02miraheze/DataDump] 07paladox 0355f4c84 - Update DataDumpPager.php [14:55:10] [02DataDump] 07paladox synchronize pull request 03#64: Update - 13https://github.com/miraheze/DataDump/pull/64 [14:58:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.43, 3.87, 3.79 [14:59:06] [02DataDump] 07paladox edited pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [14:59:45] [02DataDump] 07paladox edited pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [15:00:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.46, 3.67, 3.72 [15:02:08] miraheze/DataDump - paladox the build passed. [15:03:10] [02DataDump] 07paladox synchronize pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [15:03:12] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/55f4c841cd17...6f0d2e6e625e [15:03:14] [02miraheze/DataDump] 07paladox 036f0d2e6 - Update DataDumpPager.php [15:06:25] miraheze/DataDump - paladox the build has errored. [15:07:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f99e05196a0d [15:07:15] [02miraheze/puppet] 07paladox 03f99e051 - jobrunner-hi: increase DataDumpGenerateJob time limits to 6h [15:07:18] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [15:07:20] [02puppet] 07paladox opened pull request 03#3406: jobrunner-hi: increase DataDumpGenerateJob time limits to 6h - 13https://github.com/miraheze/puppet/pull/3406 [15:07:25] [02puppet] 07paladox closed pull request 03#3406: jobrunner-hi: increase DataDumpGenerateJob time limits to 6h - 13https://github.com/miraheze/puppet/pull/3406 [15:07:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e4b7a8057de5...71352989b465 [15:07:30] [02miraheze/puppet] 07paladox 037135298 - jobrunner-hi: increase DataDumpGenerateJob time limits to 6h (#3406) [15:07:33] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [15:07:35] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [15:08:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.02, 3.71, 3.69 [15:08:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/71352989b465...f93562d0ff34 [15:08:35] [02miraheze/puppet] 07paladox 03f93562d - jobrunner-hi: increase runners to 2 for hi queue [15:10:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.29, 3.50, 3.61 [15:11:44] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/6f0d2e6e625e...8b6cd1b04a8f [15:11:47] [02miraheze/DataDump] 07paladox 038b6cd1b - Update DataDumpGenerateJob.php [15:11:50] [02DataDump] 07paladox synchronize pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [15:12:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.61, 3.74, 3.67 [15:14:33] miraheze/DataDump - paladox the build has errored. [15:15:48] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/8b6cd1b04a8f...1fb9fed852f3 [15:15:49] [02miraheze/DataDump] 07paladox 031fb9fed - Update DataDumpGenerateJob.php [15:15:50] [02DataDump] 07paladox synchronize pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [15:16:49] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/1fb9fed852f3...65c2e1aef01a [15:16:51] [02miraheze/DataDump] 07paladox 0365c2e1a - Update DataDumpPager.php [15:16:54] [02DataDump] 07paladox synchronize pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [15:17:45] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:19:48] [02mw-config] 07R4356th commented on pull request 03#5307: T11148: added reviewer protection for infopediawiki - 13https://github.com/miraheze/mw-config/pull/5307#issuecomment-1715930493 [15:20:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.90, 3.99, 3.86 [15:30:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.22, 4.04, 3.91 [15:33:00] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/65c2e1aef01a...3c872a6f91a4 [15:33:02] [02miraheze/DataDump] 07paladox 033c872a6 - Update DataDumpPager.php [15:33:05] [02DataDump] 07paladox synchronize pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [15:34:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.89, 3.97, 3.91 [15:35:39] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/3c872a6f91a4...f10bfa410efc [15:35:40] [02miraheze/DataDump] 07paladox 03f10bfa4 - Update DataDumpPager.php [15:35:41] [02DataDump] 07paladox synchronize pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [15:35:52] miraheze/DataDump - paladox the build has errored. [15:36:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.10, 3.99, 3.93 [15:38:17] [02DataDump] 07paladox synchronize pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [15:38:20] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/f10bfa410efc...e13326816f21 [15:38:21] [02miraheze/DataDump] 07paladox 03e133268 - Update DataDumpPager.php [15:38:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.19, 3.64, 3.80 [15:38:26] miraheze/DataDump - paladox the build has errored. [15:41:40] miraheze/DataDump - paladox the build has errored. [15:44:23] [02DataDump] 07paladox synchronize pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [15:44:25] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/e13326816f21...00ea0149bb02 [15:44:26] [02miraheze/DataDump] 07paladox 0300ea014 - Update DataDumpPager.php [15:46:24] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [15:47:10] miraheze/DataDump - paladox the build has errored. [15:48:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.22, 3.76, 3.72 [15:50:51] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:54:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.36, 3.87, 3.84 [15:55:30] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:56:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.11, 3.95, 3.87 [15:58:33] [02mw-config] 07R4356th reviewed pull request 03#5290 commit - 13https://github.com/miraheze/mw-config/pull/5290#discussion_r1323258396 [15:59:24] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:02:38] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/00ea0149bb02...63d6baaebd5b [16:02:39] [02miraheze/DataDump] 07paladox 0363d6baa - Update SpecialDataDump.php [16:02:41] [02DataDump] 07paladox synchronize pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [16:05:50] miraheze/DataDump - paladox the build has errored. [16:10:30] [02mw-config] 07ugochimobi closed pull request 03#5307: T11148: added reviewer protection for infopediawiki - 13https://github.com/miraheze/mw-config/pull/5307 [16:19:11] [02mw-config] 07robkam synchronize pull request 03#5290: Add class extiw for extension LinkTarget - 13https://github.com/miraheze/mw-config/pull/5290 [16:19:37] [02mw-config] 07robkam reviewed pull request 03#5290 commit - 13https://github.com/miraheze/mw-config/pull/5290#discussion_r1323284383 [16:20:19] miraheze/mw-config - robkam the build passed. [16:28:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.58, 3.65, 4.00 [16:34:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.13, 3.88, 3.99 [16:36:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.63, 3.76, 3.93 [16:39:14] [02DataDump] 07paladox synchronize pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [16:39:16] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/63d6baaebd5b...0f621dc59e83 [16:39:18] [02miraheze/DataDump] 07paladox 030f621dc - Update DataDumpPager.php [16:40:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.69, 4.16, 4.05 [16:42:06] miraheze/DataDump - paladox the build has errored. [16:46:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.42, 3.69, 3.89 [16:50:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:54:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.97, 2.82, 3.34 [17:03:13] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5007351339 secs [17:27:23] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [17:42:58] PROBLEM - cp22 NTP time on cp22 is CRITICAL: NTP CRITICAL: Offset 0.5003926158 secs [17:57:23] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [18:09:14] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.01, 3.40, 3.03 [18:13:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.86, 3.84, 3.30 [18:19:10] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.13, 3.91, 3.49 [18:21:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.78, 3.86, 3.52 [18:29:06] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.91, 4.18, 3.75 [18:29:23] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:35:16] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [18:58:16] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [19:02:54] !log [paladox@mwtask141] starting deploy of {'world': True} to all [19:03:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:03:52] RECOVERY - wikipedia.yogfront.ooo - reverse DNS on sslhost is OK: SSL OK - wikipedia.yogfront.ooo reverse DNS resolves to cp22.miraheze.org - CNAME OK [19:08:22] !log [paladox@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 327s [19:08:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:20:43] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.37, 3.03, 3.84 [19:27:23] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [19:28:39] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.71, 2.81, 3.39 [19:33:30] !log [paladox@mwtask141] starting deploy of {'world': True} to all [19:33:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:36] !log [paladox@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 245s [19:37:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:44:30] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.54, 3.43, 3.38 [19:46:30] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.32, 3.38, 3.37 [19:52:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.05, 6.57, 5.94 [19:54:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.72, 6.21, 5.88 [19:55:25] PROBLEM - cp33 Current Load on cp33 is WARNING: LOAD WARNING - total load average: 2.20, 3.50, 2.51 [19:57:25] RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 1.16, 2.66, 2.32 [20:11:32] [02DataDump] 07paladox synchronize pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [20:11:33] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/0f621dc59e83...1f56f53585bd [20:11:35] [02miraheze/DataDump] 07paladox 031f56f53 - Update DataDumpGenerateJob.php [20:13:44] [02DataDump] 07paladox synchronize pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [20:13:45] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/1f56f53585bd...e69d4336a9a7 [20:13:47] [02miraheze/DataDump] 07paladox 03e69d433 - Update DataDumpGenerateJob.php [20:22:34] miraheze/DataDump - paladox the build passed. [20:30:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [20:53:28] !log [paladox@mwtask141] starting deploy of {'world': True} to all [20:53:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:54:04] PROBLEM - cloud14 Puppet on cloud14 is UNKNOWN: NRPE: Unable to read output [20:57:19] !log [paladox@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 230s [20:57:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:59:23] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [21:06:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.88, 3.50, 3.15 [21:10:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.80, 3.33, 3.17 [21:17:43] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/e69d4336a9a7...125381cfb017 [21:17:46] [02miraheze/DataDump] 07paladox 03125381c - Update DataDumpGenerateJob.php [21:17:49] [02DataDump] 07paladox synchronize pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [21:22:04] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:24:37] miraheze/DataDump - paladox the build passed. [21:47:03] !log [paladox@mwtask141] starting deploy of {'world': True} to all [21:47:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:16] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.75, 3.70, 3.19 [21:53:08] !log [paladox@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 364s [21:53:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:14] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.15, 2.96, 3.00 [21:54:52] !log [paladox@mwtask141] starting deploy of {'world': True} to all [21:54:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:57:23] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [21:58:28] !log [paladox@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [21:58:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:58:51] !log [paladox@mwtask141] starting deploy of {'world': True} to all [21:58:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:02:39] !log [paladox@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 228s [22:02:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:11:05] !log [paladox@mwtask141] starting deploy of {'world': True} to all [22:11:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:15:03] !log [paladox@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 238s [22:15:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:16:45] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/125381cfb017...1d850054127d [22:16:47] [02miraheze/DataDump] 07paladox 031d85005 - Update DataDumpGenerateJob.php [22:16:49] [02DataDump] 07paladox synchronize pull request 03#64: Prevent form submitting without target - 13https://github.com/miraheze/DataDump/pull/64 [22:17:39] [02DataDump] 07paladox edited pull request 03#64: Improvements to DataDump - 13https://github.com/miraheze/DataDump/pull/64 [22:18:48] [02DataDump] 07paladox edited pull request 03#64: Improvements to DataDump - 13https://github.com/miraheze/DataDump/pull/64 [22:22:28] miraheze/DataDump - paladox the build passed. [22:23:35] [02DataDump] 07paladox closed pull request 03#64: Improvements to DataDump - 13https://github.com/miraheze/DataDump/pull/64 [22:23:37] [02miraheze/DataDump] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/DataDump/compare/5c8d34dbde5e...8f95f2b681f6 [22:23:40] [02miraheze/DataDump] 07paladox 038f95f2b - Improvements to DataDump (#64) [22:24:10] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/9d9980d9a02a...463086e672cd [22:24:11] [02miraheze/mediawiki] 07paladox 03463086e - Update DataDump [22:25:26] !log [paladox@mwtask141] starting deploy of {'world': True} to all [22:25:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:25:41] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_40 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/fa01faa04ceb...a70590a40e6c [22:25:44] [02miraheze/mediawiki] 07paladox 03a70590a - Update DataDump [22:25:59] !log [paladox@test131] starting deploy of {'world': True, 'l10n': True} to all [22:26:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:29:18] !log [paladox@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 232s [22:29:23] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [22:29:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:30:25] miraheze/DataDump - paladox the build passed. [22:32:52] !log [paladox@test131] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 413s [22:32:56] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.59, 3.47, 2.96 [22:32:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:34:56] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.68, 3.21, 2.93 [22:42:45] PROBLEM - cp32 Current Load on cp32 is CRITICAL: LOAD CRITICAL - total load average: 6.80, 4.08, 2.64 [22:44:45] RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 1.63, 3.01, 2.42 [22:47:48] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.47, 3.60, 3.17 [22:53:46] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.74, 3.91, 3.46 [22:57:44] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.45, 3.26, 3.30 [22:58:16] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [23:04:24] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 12.63, 8.39, 3.97 [23:10:09] PROBLEM - cp33 Current Load on cp33 is WARNING: LOAD WARNING - total load average: 2.41, 3.83, 3.17 [23:10:24] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.05, 5.97, 4.68 [23:12:03] RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 1.14, 2.86, 2.89 [23:16:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: LOAD CRITICAL - total load average: 4.06, 4.59, 2.82 [23:18:50] PROBLEM - cp32 Current Load on cp32 is WARNING: LOAD WARNING - total load average: 1.40, 3.41, 2.60 [23:20:50] RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 2.67, 3.13, 2.59 [23:23:35] PROBLEM - cp33 Current Load on cp33 is CRITICAL: LOAD CRITICAL - total load average: 28.86, 9.57, 4.99 [23:24:01] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 32.02, 10.87, 4.37 [23:24:33] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 13.17, 7.89, 3.46 [23:24:50] PROBLEM - cp32 Current Load on cp32 is CRITICAL: LOAD CRITICAL - total load average: 14.98, 12.60, 6.58 [23:25:06] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:25:36] PROBLEM - cp22 Varnish Backends on cp22 is WARNING: No backends detected. If this is an error, see readme.txt [23:25:58] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 91% [23:26:11] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 16.25, 9.52, 5.86 [23:26:14] PROBLEM - closinglogosgroup.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:26:14] PROBLEM - wiki.conworlds.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:26:40] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: CRITICAL - load average: 5.15, 3.71, 2.69 [23:27:00] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 11 Nov 2023 23:59:59 GMT +0000. [23:27:06] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 21.54, 12.76, 7.74 [23:27:36] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [23:27:54] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [23:28:21] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 17.18, 12.67, 7.82 [23:28:27] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 17.88, 13.48, 8.34 [23:28:34] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 18.63, 14.29, 9.01 [23:28:47] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 5 backends are down. mw131 mw132 mw141 mw142 mw133 [23:28:49] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 15.80, 13.06, 7.85 [23:28:56] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [23:29:00] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 17.78, 12.14, 7.07 [23:29:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.68, 7.93, 6.23 [23:29:36] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 15 backends are healthy [23:29:49] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 15 backends are healthy [23:29:58] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 1% [23:30:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.42, 9.33, 6.86 [23:31:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.31, 7.97, 6.50 [23:32:13] PROBLEM - gtfo-wiki.cn - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:32:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 3.99, 7.30, 6.42 [23:32:28] PROBLEM - cp32 HTTPS on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:32:35] PROBLEM - istpcomputing.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:32:35] PROBLEM - wiki.graalmilitary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:32:39] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:32:42] PROBLEM - wiki.anglish.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:32:48] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 91% [23:32:49] PROBLEM - dragonquestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:32:51] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 15 backends are healthy [23:32:57] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 15 backends are healthy [23:33:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 18.89, 9.96, 7.28 [23:34:09] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 6.03, 10.63, 8.30 [23:34:19] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 34.29, 16.85, 9.94 [23:34:26] RECOVERY - cp32 HTTPS on cp32 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3734 bytes in 0.532 second response time [23:34:34] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Sun 05 Nov 2023 16:43:26 GMT +0000. [23:34:44] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 6% [23:35:01] PROBLEM - mw143 Current Load on mw143 is WARNING: WARNING - load average: 4.73, 10.76, 8.40 [23:36:09] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 3.97, 8.30, 7.72 [23:36:11]