[00:02:52] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.91, 4.20, 3.68 [00:07:24] RECOVERY - cp33 Disk Space on cp33 is OK: DISK OK - free space: / 14137 MB (18% inode=98%); [00:08:22] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.49, 3.24, 2.14 [00:10:22] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.26, 3.23, 2.28 [00:10:40] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.59, 3.84, 3.82 [00:20:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.94, 4.62, 4.07 [00:24:40] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [00:30:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.93, 3.66, 4.00 [00:34:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.03, 3.92, 3.98 [00:35:45] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:36:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.70, 3.77, 3.91 [00:38:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.34, 4.49, 4.14 [00:42:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.33, 3.51, 3.83 [00:50:39] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.39, 2.68, 3.33 [00:54:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.26, 3.57, 3.58 [00:56:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.34, 3.59, 3.57 [01:00:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.81, 3.25, 3.52 [01:04:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.22, 4.47, 3.91 [01:04:50] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [01:10:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.37, 3.62, 3.76 [01:12:22] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.32, 3.60, 3.27 [01:14:22] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 1.84, 2.94, 3.06 [01:16:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.06, 4.06, 3.78 [01:18:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.25, 3.48, 3.62 [01:22:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.96, 4.16, 3.85 [01:22:40] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [01:36:42] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [01:44:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.32, 3.41, 3.87 [01:46:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.34, 3.98, 4.01 [01:48:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.07, 3.13, 3.69 [01:50:39] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.22, 2.52, 3.40 [01:54:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.63, 3.45, 3.54 [01:56:39] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.04, 2.88, 3.32 [01:57:50] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [02:06:33] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:20:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.11, 3.54, 3.13 [02:22:39] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.93, 3.36, 3.12 [02:26:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.52, 4.68, 3.80 [02:30:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.48, 3.54, 3.54 [02:36:39] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.73, 3.02, 3.32 [02:54:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.05, 5.03, 3.66 [02:57:49] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [03:03:53] PROBLEM - mon141 Backups Grafana on mon141 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 864214 seconds old and 1910 bytes [03:07:53] RECOVERY - mon141 Backups Grafana on mon141 is OK: FILE_AGE OK: /var/log/grafana-backup.log is 14 seconds old and 0 bytes [03:12:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/074c5bb3f5a7 [03:12:38] [02miraheze/puppet] 07paladox 03074c5bb - db: Add option to disable sql backups [03:12:39] [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [03:12:42] [02puppet] 07paladox opened pull request 03#3352: db: Add option to disable sql backups - 13https://github.com/miraheze/puppet/pull/3352 [03:12:51] [02puppet] 07paladox synchronize pull request 03#3352: db: Add option to disable sql backups - 13https://github.com/miraheze/puppet/pull/3352 [03:12:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/074c5bb3f5a7...6c64eda65da3 [03:12:54] [02miraheze/puppet] 07paladox 036c64eda - Update db.pp [03:13:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6c64eda65da3...4a2d377a647b [03:13:07] [02miraheze/puppet] 07paladox 034a2d377 - Update db112.yaml [03:13:09] [02puppet] 07paladox synchronize pull request 03#3352: db: Add option to disable sql backups - 13https://github.com/miraheze/puppet/pull/3352 [03:13:16] PROBLEM - cloud11 NTP time on cloud11 is WARNING: NTP WARNING: Offset 0.1290666461 secs [03:13:27] [02puppet] 07paladox closed pull request 03#3352: db: Add option to disable sql backups - 13https://github.com/miraheze/puppet/pull/3352 [03:13:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/234202f95a2e...b0c6134a5845 [03:13:29] [02miraheze/puppet] 07paladox 03b0c6134 - db: Add option to disable sql backups (#3352) [03:13:32] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [03:13:35] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [03:13:53] PROBLEM - db112 Backups SQL icinga on db112 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-icinga-backup-fortnightly.log [03:24:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/09bb0be8ceb4 [03:24:09] [02miraheze/puppet] 07paladox 0309bb0be - miraheze-backup: if an error occurs with mediawiki-xml, continue [03:24:11] [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [03:24:14] [02puppet] 07paladox opened pull request 03#3353: miraheze-backup: if an error occurs with mediawiki-xml, continue - 13https://github.com/miraheze/puppet/pull/3353 [03:24:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.76, 3.36, 3.89 [03:27:49] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [03:27:53] RECOVERY - db112 Backups SQL icinga on db112 is OK: FILE_AGE OK: /var/log/sql-icinga-backup-fortnightly.log is 2 seconds old and 120 bytes [03:28:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.00, 3.33, 3.70 [03:30:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.57, 2.88, 3.48 [03:32:11] [02puppet] 07paladox synchronize pull request 03#3353: miraheze-backup: if an error occurs with mediawiki-xml, continue - 13https://github.com/miraheze/puppet/pull/3353 [03:32:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/09bb0be8ceb4...a1da70449065 [03:32:15] [02miraheze/puppet] 07paladox 03a1da704 - Update miraheze-backup.py.erb [03:32:39] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.96, 2.49, 3.26 [03:38:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.58, 3.37, 3.45 [03:39:16] [02puppet] 07paladox closed pull request 03#3353: miraheze-backup: if an error occurs with mediawiki-xml, continue - 13https://github.com/miraheze/puppet/pull/3353 [03:39:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b0c6134a5845...c316dde8dcdb [03:39:19] [02miraheze/puppet] 07paladox 03c316dde - miraheze-backup: if an error occurs with mediawiki-xml, continue (#3353) [03:39:21] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [03:39:24] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [03:53:07] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.95, 3.22, 3.38 [03:53:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/039167d23fb2...be5f19f5bfe6 [03:53:41] [02miraheze/puppet] 07paladox 03be5f19f - Update init.pp [03:53:42] [02puppet] 07paladox synchronize pull request 03#3314: puppetdb: symlinks /usr/bin/puppetdb to correct location - 13https://github.com/miraheze/puppet/pull/3314 [03:55:20] RECOVERY - cloud11 NTP time on cloud11 is OK: NTP OK: Offset 0.01118403673 secs [03:55:20] [02miraheze/puppet] 07paladox pushed 0358 commits to 03paladox-patch-4 [+6/-8/±501] 13https://github.com/miraheze/puppet/compare/be5f19f5bfe6...76fed4eb4553 [03:55:21] [02miraheze/puppet] 07paladox 0376fed4e - Merge branch 'master' into paladox-patch-4 [03:55:24] [02puppet] 07paladox synchronize pull request 03#3314: puppetdb: symlinks /usr/bin/puppetdb to correct location - 13https://github.com/miraheze/puppet/pull/3314 [03:56:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/76fed4eb4553...f4da30633174 [03:56:25] [02miraheze/puppet] 07paladox 03f4da306 - Update puppetdb.pp [03:56:27] [02puppet] 07paladox synchronize pull request 03#3314: puppetdb: symlinks /usr/bin/puppetdb to correct location - 13https://github.com/miraheze/puppet/pull/3314 [03:56:56] [02puppet] 07paladox closed pull request 03#3314: puppetdb: symlinks /usr/bin/puppetdb to correct location - 13https://github.com/miraheze/puppet/pull/3314 [03:56:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/c316dde8dcdb...ac745b508891 [03:57:01] [02miraheze/puppet] 07paladox 03ac745b5 - puppetdb: symlinks /usr/bin/puppetdb to correct location (#3314) [03:57:03] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [03:57:05] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [03:57:11] [02miraheze/puppet] 07paladox deleted branch 03The-Voidwalker-patch-1 [03:57:14] [02puppet] 07paladox deleted branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/puppet [04:00:52] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.80, 4.70, 3.89 [04:06:40] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.52, 3.63, 3.69 [04:10:31] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.38, 2.94, 3.39 [04:10:37] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:11:10] PROBLEM - puppet141 puppetdb on puppet141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:13:46] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.71, 3.00, 2.29 [04:14:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ac745b508891...ec41aa40f1f1 [04:14:04] [02miraheze/puppet] 07paladox 03ec41aa4 - Fix [04:14:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ec41aa40f1f1...a75d63484adf [04:14:38] [02miraheze/puppet] 07paladox 03a75d634 - Fix [04:15:43] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.32, 3.35, 2.50 [04:15:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a486d229e022 [04:15:54] [02miraheze/puppet] 07paladox 03a486d22 - Use puppetdb role on puppet141 [04:15:55] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [04:15:56] [02puppet] 07paladox opened pull request 03#3354: Use puppetdb role on puppet141 - 13https://github.com/miraheze/puppet/pull/3354 [04:16:31] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:16:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a486d229e022...f61edfe934ed [04:16:38] [02miraheze/puppet] 07paladox 03f61edfe - Update puppetserver.pp [04:16:40] [02puppet] 07paladox synchronize pull request 03#3354: Use puppetdb role on puppet141 - 13https://github.com/miraheze/puppet/pull/3354 [04:17:40] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.81, 3.41, 2.62 [04:18:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f61edfe934ed...0edc751687e1 [04:18:09] [02miraheze/puppet] 07paladox 030edc751 - Update init.pp [04:18:11] [02puppet] 07paladox synchronize pull request 03#3354: Use puppetdb role on puppet141 - 13https://github.com/miraheze/puppet/pull/3354 [04:18:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0edc751687e1...0d39f8e40c31 [04:18:41] [02miraheze/puppet] 07paladox 030d39f8e - Update puppetdb.pp [04:18:43] [02puppet] 07paladox synchronize pull request 03#3354: Use puppetdb role on puppet141 - 13https://github.com/miraheze/puppet/pull/3354 [04:18:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/0d39f8e40c31...6d2cad1f9537 [04:18:53] [02miraheze/puppet] 07paladox 036d2cad1 - Delete puppetdb_logback.xml [04:18:54] [02puppet] 07paladox synchronize pull request 03#3354: Use puppetdb role on puppet141 - 13https://github.com/miraheze/puppet/pull/3354 [04:19:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-1/±0] 13https://github.com/miraheze/puppet/compare/6d2cad1f9537...660fa780bc7a [04:19:09] [02miraheze/puppet] 07paladox 03660fa78 - Rename puppetdb_logback.xml to puppetdb_logback.xml [04:19:11] [02puppet] 07paladox synchronize pull request 03#3354: Use puppetdb role on puppet141 - 13https://github.com/miraheze/puppet/pull/3354 [04:19:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/660fa780bc7a...4a9fb472bbf1 [04:19:54] [02miraheze/puppet] 07paladox 034a9fb47 - Update client.pp [04:19:56] [02puppet] 07paladox synchronize pull request 03#3354: Use puppetdb role on puppet141 - 13https://github.com/miraheze/puppet/pull/3354 [04:20:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4a9fb472bbf1...3a1a44c44c40 [04:20:12] [02miraheze/puppet] 07paladox 033a1a44c - Update init.pp [04:20:14] [02puppet] 07paladox synchronize pull request 03#3354: Use puppetdb role on puppet141 - 13https://github.com/miraheze/puppet/pull/3354 [04:21:22] [02puppet] 07paladox closed pull request 03#3354: Use puppetdb role on puppet141 - 13https://github.com/miraheze/puppet/pull/3354 [04:21:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-2/±5] 13https://github.com/miraheze/puppet/compare/a75d63484adf...a49cbebe01ce [04:21:25] [02miraheze/puppet] 07paladox 03a49cbeb - Use puppetdb role on puppet141 (#3354) [04:21:26] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [04:21:29] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [04:23:27] RECOVERY - puppet141 puppetdb on puppet141 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:507 port 8081 [04:27:20] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [04:27:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.20, 3.32, 2.99 [04:36:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.06, 3.77, 3.33 [04:38:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.64, 3.32, 3.22 [04:38:36] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.28, 3.70, 3.35 [04:40:32] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.20, 3.18, 3.20 [04:50:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.12, 4.12, 3.47 [04:56:51] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [04:58:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.93, 3.72, 3.69 [05:00:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.44, 4.19, 3.87 [05:02:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.04, 3.34, 3.59 [05:06:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.84, 3.48, 3.56 [05:12:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.17, 3.71, 3.65 [05:16:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.90, 3.05, 3.40 [05:26:22] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [05:54:26] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cpt-ra.bid. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:55:54] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [06:08:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.89, 4.05, 3.39 [06:24:08] RECOVERY - with.cpt-ra.bid - reverse DNS on sslhost is OK: SSL OK - with.cpt-ra.bid reverse DNS resolves to cp22.miraheze.org - CNAME OK [06:28:13] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.18, 4.51, 2.55 [06:30:09] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.33, 3.31, 2.34 [06:36:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.10, 3.34, 2.93 [06:38:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.95, 3.21, 2.93 [06:54:55] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [07:25:38] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cpt-ra.bid. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:44:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.23, 3.42, 3.95 [07:54:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.54, 2.64, 3.31 [07:55:21] RECOVERY - with.cpt-ra.bid - reverse DNS on sslhost is OK: SSL OK - with.cpt-ra.bid reverse DNS resolves to cp22.miraheze.org - CNAME OK [08:22:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.02, 3.40, 3.17 [08:23:29] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [08:24:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.27, 3.40, 3.20 [08:52:59] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [09:12:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.64, 3.51, 3.21 [09:14:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.60, 3.20, 3.13 [09:22:30] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [09:52:00] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [10:08:25] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [10:24:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.48, 3.07, 2.14 [10:26:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.95, 2.67, 2.11 [10:34:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.35, 3.48, 2.95 [10:36:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.61, 3.54, 3.04 [10:38:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.13, 3.72, 3.17 [10:38:23] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [10:50:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.47, 3.95, 3.72 [11:06:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.75, 3.14, 3.40 [11:11:10] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.314853847 secs [11:11:40] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:33:26] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.39, 3.45, 2.53 [11:35:22] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.90, 2.54, 2.30 [11:37:10] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.004896610975 secs [11:38:19] PROBLEM - wikiru.wildterra2.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wildterra2.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:42:16] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:50:08] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [12:07:13] RECOVERY - wikiru.wildterra2.com - reverse DNS on sslhost is OK: SSL OK - wikiru.wildterra2.com reverse DNS resolves to cp22.miraheze.org - CNAME OK [12:07:57] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.74, 3.62, 3.09 [12:09:35] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [12:35:16] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.82, 3.90, 4.00 [12:37:13] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 3.96, 4.03, 4.04 [12:39:10] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.19, 3.69, 3.91 [12:41:07] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.63, 3.96, 3.97 [12:42:14] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [12:45:01] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.26, 3.85, 3.96 [12:49:09] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [12:54:47] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.49, 4.07, 3.96 [13:07:29] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [13:13:18] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [13:34:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.17, 3.02, 3.78 [13:42:20] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [13:42:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.70, 2.64, 3.34 [14:38:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.61, 3.42, 2.87 [14:40:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.73, 3.85, 3.09 [14:42:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.96, 3.71, 3.15 [14:47:12] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [14:50:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.65, 3.34, 3.23 [14:54:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.57, 3.53, 3.33 [14:57:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.20, 3.85, 2.97 [14:59:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.06, 3.34, 2.90 [15:00:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.47, 3.90, 3.55 [15:02:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.28, 3.64, 3.49 [15:04:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.31, 3.12, 3.32 [15:08:51] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.09, 3.78, 3.06 [15:12:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.29, 3.32, 3.04 [15:18:32] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.25, 5.25, 3.82 [15:26:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.09, 3.91, 3.85 [15:28:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.27, 5.07, 4.28 [15:32:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.03, 3.94, 3.99 [15:34:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.43, 4.37, 4.12 [15:36:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.23, 3.88, 3.97 [15:38:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.66, 4.92, 4.31 [15:39:33] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [15:40:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.39, 3.80, 3.97 [15:46:14] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [15:50:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.25, 3.70, 3.61 [15:54:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.71, 3.81, 3.68 [16:02:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.01, 3.44, 3.44 [16:08:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.90, 2.83, 3.19 [16:14:31] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.77, 3.42, 2.38 [16:15:22] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.30, 3.54, 3.01 [16:15:45] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [16:17:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.53, 3.48, 3.05 [16:20:31] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.11, 3.67, 2.89 [16:21:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.98, 3.32, 3.08 [16:22:31] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.66, 3.26, 2.83 [16:31:13] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.07, 3.50, 3.10 [16:32:14] PROBLEM - www.vanillaanarchy.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.vanillaanarchy.net' expires in 15 day(s) (Sun 20 Aug 2023 16:17:58 GMT +0000). [16:35:06] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.94, 3.27, 3.09 [16:36:19] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8c50f6c58edb...9789ffc25968 [16:36:21] [02miraheze/ssl] 07MirahezeSSLBot 039789ffc - Bot: Update SSL cert for www.vanillaanarchy.net [16:38:31] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.71, 3.60, 3.17 [16:39:33] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [16:40:31] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.48, 3.95, 3.35 [16:42:31] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.86, 3.90, 3.41 [16:42:35] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.59, 3.32, 3.11 [16:42:53] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.04, 3.45, 3.27 [16:44:31] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.22, 3.28, 3.24 [16:44:32] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.15, 3.28, 3.12 [16:44:49] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.58, 2.74, 3.03 [16:54:30] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.40, 3.48, 3.14 [16:56:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.67, 3.33, 3.17 [16:56:26] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.31, 3.92, 3.33 [16:58:22] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.70, 3.37, 3.19 [17:00:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.18, 3.21, 3.15 [17:01:41] RECOVERY - www.vanillaanarchy.net - LetsEncrypt on sslhost is OK: OK - Certificate 'www.vanillaanarchy.net' will expire on Thu 02 Nov 2023 15:36:10 GMT +0000. [17:02:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.35, 3.52, 3.31 [17:10:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.75, 3.35, 3.38 [17:12:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.81, 3.38, 3.19 [17:14:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.52, 3.72, 3.33 [17:16:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.02, 3.39, 3.25 [17:23:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.56, 3.49, 3.22 [17:27:05] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.09, 3.12, 3.15 [17:44:18] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [18:07:30] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/0493e2c312ca [18:07:34] [02miraheze/mw-config] 07paladox 030493e2c - Fix wgSharedTables for srewiki [18:07:34] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [18:07:35] [02mw-config] 07paladox opened pull request 03#5291: Fix wgSharedTables for srewiki - 13https://github.com/miraheze/mw-config/pull/5291 [18:08:36] miraheze/mw-config - paladox the build passed. [18:09:01] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 6.12, 4.18, 3.31 [18:09:29] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2d25d2bcff7b...84206c1bd1db [18:09:30] [02miraheze/mw-config] 07paladox 0384206c1 - Add missing options to wgSharedTables for srewiki [18:09:46] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [18:09:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:10:00] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 13s [18:10:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:10:08] !log [paladox@mwtask141] starting deploy of {'config': True} to all [18:10:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:10:16] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [18:10:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:10:30] miraheze/mw-config - paladox the build passed. [18:15:25] PROBLEM - wiki.theluxuryelevator.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.theluxuryelevator.com' expires in 15 day(s) (Sun 20 Aug 2023 18:11:19 GMT +0000). [18:17:26] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.84, 3.54, 2.05 [18:19:26] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.49, 2.81, 1.97 [18:21:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9789ffc25968...1063b220395b [18:21:07] [02miraheze/ssl] 07MirahezeSSLBot 031063b22 - Bot: Update SSL cert for wiki.theluxuryelevator.com [18:27:19] !log [@test131] starting deploy of {'config': True} to all [18:27:20] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [18:27:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:27:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:35] PROBLEM - eurocom.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'eurocom.wiki' expires in 15 day(s) (Sun 20 Aug 2023 18:12:39 GMT +0000). [18:29:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1063b220395b...083216a6eb58 [18:29:24] [02miraheze/ssl] 07MirahezeSSLBot 03083216a - Bot: Update SSL cert for eurocom.wiki [18:29:40] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/1b633c87e710 [18:29:43] [02miraheze/mw-config] 07paladox 031b633c8 - Add missing sql files to PropertySuggester [18:29:46] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [18:29:48] [02mw-config] 07paladox opened pull request 03#5292: Add missing sql files to PropertySuggester - 13https://github.com/miraheze/mw-config/pull/5292 [18:29:56] [02mw-config] 07paladox closed pull request 03#5292: Add missing sql files to PropertySuggester - 13https://github.com/miraheze/mw-config/pull/5292 [18:29:58] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/84206c1bd1db...dbcffd919140 [18:29:59] [02miraheze/mw-config] 07paladox 03dbcffd9 - Add missing sql files to PropertySuggester (#5292) [18:30:01] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [18:30:04] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [18:30:46] miraheze/mw-config - paladox the build passed. [18:30:56] miraheze/mw-config - paladox the build passed. [18:32:42] PROBLEM - mirawww.glitchcity.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mirawww.glitchcity.wiki' expires in 15 day(s) (Sun 20 Aug 2023 18:12:01 GMT +0000). [18:33:30] PROBLEM - swiftobject111 APT on swiftobject111 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [18:35:29] PROBLEM - swiftobject111 APT on swiftobject111 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (6 critical updates). [18:41:23] !log [@mwtask141] starting deploy of {'config': True} to all [18:41:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:41:32] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [18:41:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:45:02] RECOVERY - wiki.theluxuryelevator.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.theluxuryelevator.com' will expire on Thu 02 Nov 2023 17:20:58 GMT +0000. [18:45:55] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [18:49:55] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [18:50:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.59, 4.14, 3.19 [18:57:03] !log [@test131] starting deploy of {'config': True} to all [18:57:04] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [18:57:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:22] RECOVERY - eurocom.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'eurocom.wiki' will expire on Thu 02 Nov 2023 17:29:14 GMT +0000. [19:02:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.37, 3.72, 3.63 [19:08:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.10, 2.93, 3.33 [19:12:51] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [19:26:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.70, 3.65, 3.41 [19:28:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.02, 3.39, 3.34 [19:36:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.63, 3.89, 3.54 [19:38:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.41, 3.58, 3.50 [19:40:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.61, 3.83, 3.58 [19:42:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.95, 3.58, 3.53 [19:42:22] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [19:44:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.92, 3.02, 3.33 [19:48:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.28, 3.25, 3.88 [19:48:52] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [20:02:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.79, 2.88, 3.35 [20:05:56] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 8.99, 12.90, 8.08 [20:07:56] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 4.07, 9.73, 7.50 [20:11:53] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [20:15:50] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=typicalcolors2wiki --no-updates --username-prefix=wikia:typicalcolors2 /home/paladox/ImportDump/typicalcolors2wiki-20230804043625.xml (START) [20:15:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:51] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [20:20:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.60, 3.22, 3.13 [20:24:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.24, 3.77, 3.34 [20:24:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.15, 3.23, 3.16 [20:26:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.35, 3.17, 3.17 [20:32:03] PROBLEM - cp33 Disk Space on cp33 is WARNING: DISK WARNING - free space: / 8221 MB (10% inode=98%); [20:35:08] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/2eaa577ccbec...4c9aaf4099a8 [20:35:09] [02miraheze/mediawiki] 07paladox 034c9aaf4 - Update SocialProfile [20:35:55] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_40 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/0653f64857f0...9b5aa96d657c [20:35:58] [02miraheze/mediawiki] 07paladox 039b5aa96 - Update SocialProfile [20:36:12] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [20:36:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:36:24] !log [paladox@test131] starting deploy of {'world': True, 'l10n': True} to all [20:36:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:39:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.08, 3.43, 3.22 [20:40:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.80, 3.83, 3.46 [20:43:05] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.34, 3.02, 3.12 [20:43:11] !log [paladox@mwtask141] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 418s [20:43:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:43:18] !log [paladox@test131] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 413s [20:43:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:44:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.17, 3.45, 3.44 [20:48:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.05, 3.66, 3.52 [20:52:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.61, 3.74, 3.59 [20:54:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.44, 4.02, 3.72 [20:58:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.65, 3.94, 3.76 [21:00:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.17, 4.02, 3.82 [21:06:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.58, 3.76, 3.27 [21:08:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.19, 3.11, 3.10 [21:15:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.49, 3.65, 3.33 [21:17:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.24, 3.12, 3.18 [21:20:47] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [21:40:26] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [21:42:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.22, 3.60, 3.19 [21:44:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.72, 2.88, 2.97 [21:50:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.43, 3.60, 3.95 [21:50:46] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [21:52:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.85, 3.89, 3.37 [21:54:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.03, 3.99, 3.47 [21:56:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.48, 3.53, 3.37 [21:56:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.32, 3.95, 4.00 [22:00:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.15, 3.26, 3.38 [22:00:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.51, 3.81, 3.94 [22:02:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/dbe3b8e2ac0e [22:02:32] [02miraheze/puppet] 07paladox 03dbe3b8e - miraheze-backup: Backup mediawiki xml dumps to archive.org [22:02:33] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [22:02:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4-1 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/97cadb28b682 [22:02:45] [02miraheze/puppet] 07paladox 0397cadb2 - miraheze-backup: Backup mediawiki xml dumps to archive.org [22:02:47] [02puppet] 07paladox created branch 03paladox-patch-4-1 - 13https://github.com/miraheze/puppet [22:02:49] [02puppet] 07paladox opened pull request 03#3355: miraheze-backup: Backup mediawiki xml dumps to archive.org - 13https://github.com/miraheze/puppet/pull/3355 [22:02:54] [02puppet] 07paladox opened pull request 03#3356: miraheze-backup: Backup mediawiki xml dumps to archive.org - 13https://github.com/miraheze/puppet/pull/3356 [22:03:04] [02puppet] 07paladox closed pull request 03#3355: miraheze-backup: Backup mediawiki xml dumps to archive.org - 13https://github.com/miraheze/puppet/pull/3355 [22:03:25]