[00:25:17] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [00:36:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.11, 3.90, 3.00 [00:38:30] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.91, 4.62, 3.39 [00:40:53] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.31, 3.57, 2.90 [00:42:48] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.05, 2.75, 2.68 [00:46:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.99, 3.37, 3.42 [00:48:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.71, 3.10, 3.31 [00:48:17] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.77, 6.52, 4.92 [00:50:14] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.19, 7.82, 5.59 [00:52:12] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.62, 7.57, 5.78 [00:54:09] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.63, 5.40, 5.20 [00:59:24] PROBLEM - mem141 NTP time on mem141 is WARNING: NTP WARNING: Offset 0.1063225567 secs [01:07:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.00, 4.46, 3.45 [01:09:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.08, 3.84, 3.34 [01:11:51] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.79, 4.08, 3.50 [01:13:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.93, 3.24, 3.26 [01:38:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.11, 3.44, 2.89 [01:40:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.96, 3.14, 2.85 [01:52:57] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.37, 5.11, 3.57 [01:55:20] RECOVERY - mem141 NTP time on mem141 is OK: NTP OK: Offset 0.09786185622 secs [01:56:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.39, 3.83, 3.41 [01:58:56] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.40, 3.27, 3.25 [02:11:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.28, 4.19, 3.16 [02:13:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.90, 3.35, 2.98 [02:22:30] PROBLEM - readerswhoknow.paneidoverse.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'readerswhoknow.paneidoverse.com' expires in 15 day(s) (Wed 12 Jul 2023 01:53:50 GMT +0000). [02:28:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/52ae6c885c67...68e153ddbe36 [02:29:00] [02miraheze/ssl] 07MirahezeSSLBot 0368e153d - Bot: Update SSL cert for readerswhoknow.paneidoverse.com [02:40:52] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.41, 4.08, 2.88 [02:44:52] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.46, 3.26, 2.88 [02:48:52] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.00, 4.47, 3.45 [02:50:51] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.99, 3.54, 3.23 [02:52:51] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.05, 3.03, 3.08 [03:01:18] RECOVERY - db101 Backups SQL on db101 is OK: FILE_AGE OK: /var/log/sql-backup.log is 76 seconds old and 275 bytes [03:02:51] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.29, 4.28, 3.43 [03:04:50] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.03, 3.53, 3.27 [03:06:50] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.62, 2.85, 3.04 [03:11:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.34, 4.47, 3.34 [03:13:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.00, 3.55, 3.14 [03:14:13] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.86, 2.94, 1.46 [03:15:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.76, 2.84, 2.92 [03:16:13] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 0.89, 2.12, 1.33 [03:21:28] RECOVERY - readerswhoknow.paneidoverse.com - LetsEncrypt on sslhost is OK: OK - Certificate 'readerswhoknow.paneidoverse.com' will expire on Sun 24 Sep 2023 01:28:49 GMT +0000. [03:36:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.65, 4.40, 3.42 [03:44:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.23, 4.00, 3.92 [03:50:28] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.84, 2.60, 3.35 [03:52:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.85, 4.70, 3.26 [03:56:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.08, 3.39, 3.05 [04:03:42] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1158978939 secs [04:22:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.53, 3.18, 2.87 [04:24:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.06, 3.13, 2.89 [04:33:42] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.0911911428 secs [05:32:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.31, 6.31, 5.04 [05:34:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.50, 5.50, 4.89 [05:37:12] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.30, 3.45, 2.60 [05:39:09] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.77, 3.33, 2.66 [06:12:31] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.61, 3.46, 2.90 [06:16:31] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.97, 2.90, 2.83 [07:38:13] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.64, 3.53, 2.43 [07:40:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.84, 2.95, 2.35 [10:07:54] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.90, 4.19, 2.34 [10:09:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.50, 3.60, 2.37 [10:11:52] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.67, 3.35, 2.43 [10:26:24] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [10:27:59] PROBLEM - wiki.helsinginpoytaroolipelaajat.fi - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.helsinginpoytaroolipelaajat.fi All nameservers failed to answer the query. [10:45:28] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [10:55:17] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [10:56:58] RECOVERY - wiki.helsinginpoytaroolipelaajat.fi - reverse DNS on sslhost is OK: SSL OK - wiki.helsinginpoytaroolipelaajat.fi reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:07:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.04, 3.84, 2.22 [11:11:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.26, 3.49, 2.50 [11:13:52] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.07, 3.35, 2.57 [11:14:51] [02miraheze/RemovePII] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/544945d3eff9...16c8cd552261 [11:14:52] [02miraheze/RemovePII] 07translatewiki 0316c8cd5 - Localisation updates from https://translatewiki.net. [11:15:17] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:37:45] miraheze/RemovePII - translatewiki the build passed. [11:39:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/cd36e00341f5 [11:39:48] [02miraheze/puppet] 07paladox 03cd36e00 - mw: increase php childs to 30 [11:39:49] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [11:39:51] [02puppet] 07paladox opened pull request 03#3246: mw: increase php childs to 30 - 13https://github.com/miraheze/puppet/pull/3246 [11:40:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cd36e00341f5...791422e167cb [11:40:08] [02miraheze/puppet] 07paladox 03791422e - Update mw132.yaml [11:40:10] [02puppet] 07paladox synchronize pull request 03#3246: mw: increase php childs to 30 - 13https://github.com/miraheze/puppet/pull/3246 [11:40:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/791422e167cb...85ce93ab6035 [11:40:18] [02miraheze/puppet] 07paladox 0385ce93a - Update mw133.yaml [11:40:19] [02puppet] 07paladox synchronize pull request 03#3246: mw: increase php childs to 30 - 13https://github.com/miraheze/puppet/pull/3246 [11:40:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/85ce93ab6035...a8426ecd0d47 [11:40:29] [02miraheze/puppet] 07paladox 03a8426ec - Update mw134.yaml [11:40:31] [02puppet] 07paladox synchronize pull request 03#3246: mw: increase php childs to 30 - 13https://github.com/miraheze/puppet/pull/3246 [11:40:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a8426ecd0d47...90ec42e63ec9 [11:40:40] [02miraheze/puppet] 07paladox 0390ec42e - Update mw141.yaml [11:40:43] [02puppet] 07paladox synchronize pull request 03#3246: mw: increase php childs to 30 - 13https://github.com/miraheze/puppet/pull/3246 [11:40:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/90ec42e63ec9...9e637722549c [11:40:52] [02miraheze/puppet] 07paladox 039e63772 - Update mw142.yaml [11:40:53] [02puppet] 07paladox synchronize pull request 03#3246: mw: increase php childs to 30 - 13https://github.com/miraheze/puppet/pull/3246 [11:40:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9e637722549c...518dfbc4db39 [11:41:00] [02miraheze/puppet] 07paladox 03518dfbc - Update mw143.yaml [11:41:01] [02puppet] 07paladox synchronize pull request 03#3246: mw: increase php childs to 30 - 13https://github.com/miraheze/puppet/pull/3246 [11:55:37] [02puppet] 07paladox closed pull request 03#3246: mw: increase php childs to 30 - 13https://github.com/miraheze/puppet/pull/3246 [11:55:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±7] 13https://github.com/miraheze/puppet/compare/84cbfe57cb59...fc1ac722ff18 [11:55:42] [02miraheze/puppet] 07paladox 03fc1ac72 - mw: increase php childs to 30 (#3246) [11:55:45] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [11:55:47] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [12:07:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.92, 3.97, 2.19 [12:11:47] PROBLEM - cp32 PowerDNS Recursor on cp32 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:12:35] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:49] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:13:56] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 45% [12:14:20] Hmm [12:14:23] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:15:20] PROBLEM - wiki.thunis.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.thunis.eu All nameservers failed to answer the query. [12:16:01] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 6% [12:16:30] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 15 backends are healthy [12:16:31] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 37 minutes ago with 0 failures [12:16:41] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:18:08] RECOVERY - cp32 PowerDNS Recursor on cp32 is OK: DNS OK: 0.179 seconds response time. miraheze.org returns 108.175.15.182,2607:f1c0:1800:26f::1,2607:f1c0:1800:8100::1,74.208.203.152 [12:27:11] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:32:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fc1ac722ff18...36dde6727deb [12:32:48] [02miraheze/puppet] 07paladox 0336dde67 - prometheus::php-fpm: upgrade exporter to 2.2.0 [12:45:09] RECOVERY - wiki.thunis.eu - reverse DNS on sslhost is OK: SSL OK - wiki.thunis.eu reverse DNS resolves to cp23.miraheze.org - CNAME OK [12:54:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/8c53de1049e2 [12:54:29] [02miraheze/puppet] 07paladox 038c53de1 - mediawiki: cleanup php* and shellbox-* temp files [12:54:32] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [12:54:35] [02puppet] 07paladox opened pull request 03#3247: mediawiki: cleanup php* and shellbox-* temp files - 13https://github.com/miraheze/puppet/pull/3247 [12:54:40] [02puppet] 07paladox closed pull request 03#3247: mediawiki: cleanup php* and shellbox-* temp files - 13https://github.com/miraheze/puppet/pull/3247 [12:54:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/36dde6727deb...2d6c7ab737fc [12:54:44] [02miraheze/puppet] 07paladox 032d6c7ab - mediawiki: cleanup php* and shellbox-* temp files (#3247) [12:54:47] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [12:54:49] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [12:56:05] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [12:59:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.27, 1.60, 3.87 [13:05:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 16.71, 6.02, 4.68 [13:13:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.66, 2.92, 3.76 [13:17:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.46, 1.99, 3.17 [13:21:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2d6c7ab737fc...79002119ebbe [13:21:10] [02miraheze/puppet] 07paladox 037900211 - graylog: upgrade to 5.1.2 [13:24:59] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [13:47:19] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [14:00:26] [02miraheze/mw-config] 07paladox pushed 032 commits to 03OOJSPlus [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/2c7222da837e^...d18ae68188bc [14:00:28] [02miraheze/mw-config] 07paladox 032c7222d - Revert "Remove OOJSPlus from ManageWiki extensions" [14:00:29] [02miraheze/mw-config] 07paladox 03d18ae68 - Install sql files when enabling OOJSPlus [14:00:30] [02mw-config] 07paladox created branch 03OOJSPlus - 13https://github.com/miraheze/mw-config [14:00:44] [02mw-config] 07paladox opened pull request 03#5260: Install sql files when enabling OOJSPlus - 13https://github.com/miraheze/mw-config/pull/5260 [14:01:56] miraheze/mw-config - paladox the build passed. [14:06:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.42, 4.13, 2.63 [14:08:50] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.86, 3.19, 2.46 [14:15:13] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/7e760e3f33eb [14:15:14] [02miraheze/mw-config] 07paladox 037e760e3 - Install sql when enabling standarddialogs [14:15:15] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [14:15:16] [02mw-config] 07paladox opened pull request 03#5261: Install sql when enabling standarddialogs - 13https://github.com/miraheze/mw-config/pull/5261 [14:15:21] [02mw-config] 07paladox closed pull request 03#5261: Install sql when enabling standarddialogs - 13https://github.com/miraheze/mw-config/pull/5261 [14:15:24] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4a2d8941d2e9...dc144fe69225 [14:15:27] [02miraheze/mw-config] 07paladox 03dc144fe - Install sql when enabling standarddialogs (#5261) [14:15:29] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [14:15:30] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [14:16:21] miraheze/mw-config - paladox the build passed. [14:16:27] miraheze/mw-config - paladox the build passed. [14:17:08] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [14:19:31] [02mw-config] 07paladox closed pull request 03#5260: Install sql files when enabling OOJSPlus - 13https://github.com/miraheze/mw-config/pull/5260 [14:19:33] [02miraheze/mw-config] 07paladox deleted branch 03OOJSPlus [14:19:34] [02mw-config] 07paladox deleted branch 03OOJSPlus - 13https://github.com/miraheze/mw-config [14:26:52] PROBLEM - mw141 PowerDNS Recursor on mw141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:27:11] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.25, 8.52, 6.07 [14:27:17] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 45.02, 25.57, 12.71 [14:28:09] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 17.07, 13.10, 7.05 [14:28:14] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.63, 2.78, 1.45 [14:28:34] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 10.50, 12.52, 8.56 [14:28:47] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 1 backends are down. mw141 [14:28:54] RECOVERY - mw141 PowerDNS Recursor on mw141 is OK: DNS OK: 2.219 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [14:29:11] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 6.77, 7.88, 6.14 [14:29:33] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw141 [14:30:09] PROBLEM - mw143 Current Load on mw143 is WARNING: WARNING - load average: 7.64, 10.68, 6.89 [14:30:34] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.95, 10.13, 8.16 [14:30:48] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 15 backends are healthy [14:31:33] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 15 backends are healthy [14:32:09] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 3.18, 8.08, 6.40 [14:32:13] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.88, 3.12, 1.96 [14:35:13] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 4.58, 9.82, 10.32 [14:36:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.65, 4.40, 3.10 [14:37:13] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.00, 8.26, 9.69 [14:40:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.56, 3.76, 3.14 [14:42:12] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.63, 3.07, 2.96 [14:48:52] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [14:56:24] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [15:06:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.87, 4.95, 3.39 [15:10:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.61, 3.80, 3.28 [15:12:39] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.63, 4.40, 3.52 [15:14:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.75, 3.86, 3.42 [15:16:33] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.30, 3.02, 3.17 [15:18:40] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [15:25:18] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [15:26:30] PROBLEM - graylog131 NTP time on graylog131 is WARNING: NTP WARNING: Offset -0.1099532843 secs [15:37:54] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 108.175.15.182/cpweb, 2607:f1c0:1800:8100::1/cpweb [15:38:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 31.27, 16.10, 7.71 [15:38:28] PROBLEM - cp32 PowerDNS Recursor on cp32 is CRITICAL: CRITICAL - Plugin timed out while executing system call [15:38:35] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:28] RECOVERY - cp32 PowerDNS Recursor on cp32 is OK: DNS OK: 1.651 second response time. miraheze.org returns 108.175.15.182,2607:f1c0:1800:26f::1,2607:f1c0:1800:8100::1,74.208.203.152 [15:40:33] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:41:54] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:53:09] RECOVERY - graylog131 NTP time on graylog131 is OK: NTP OK: Offset -0.08816286922 secs [16:01:54] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.85, 1.94, 3.73 [16:05:51] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.14, 3.50, 3.93 [16:13:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.81, 3.12, 3.75 [16:19:51] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.17, 3.94, 3.77 [16:21:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.74, 2.98, 3.44 [16:23:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.49, 2.44, 3.18 [16:37:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.02, 4.00, 3.26 [16:39:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.55, 3.07, 3.01 [17:34:44] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.78, 2.48, 2.07 [17:36:41] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.34, 2.71, 2.23 [17:52:36] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [17:52:51] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.39, 2.80, 1.72 [17:54:46] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.28, 2.77, 1.86 [18:22:25] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:37:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.25, 4.45, 3.05 [18:39:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.51, 3.43, 2.85 [18:41:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.24, 2.69, 2.65 [18:43:07] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [18:52:19] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [19:08:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 28.12, 14.06, 6.56 [19:09:26] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.76, 3.70, 2.79 [19:13:26] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.80, 3.37, 2.87 [19:19:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.98, 2.54, 3.90 [19:25:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.62, 2.07, 3.25 [19:27:24] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.57, 3.09, 2.79 [19:28:34] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 14.62, 14.86, 9.47 [19:29:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.20, 2.65, 2.65 [19:32:34] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.63, 10.09, 8.75 [19:36:13] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.21, 3.09, 1.64 [19:37:11] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 12.12, 11.93, 8.21 [19:37:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.63, 4.59, 3.59 [19:38:13] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 3.17, 3.20, 1.86 [19:39:11] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 6.20, 9.82, 7.89 [19:39:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.61, 3.70, 3.37 [19:41:22] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [19:41:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.66, 3.12, 3.21 [19:59:46] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 34.67, 32.86, 14.46 [20:02:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.44, 7.21, 5.65 [20:04:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.22, 7.62, 6.00 [20:12:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.69, 7.79, 6.86 [20:12:46] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/cc68c8c1598a [20:12:49] [02miraheze/CreateWiki] 07paladox 03cc68c8c - Fix using CreateWikiAIThreshold in approve/decline functions [20:12:51] [02CreateWiki] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/CreateWiki [20:12:53] [02CreateWiki] 07paladox opened pull request 03#414: Fix using CreateWikiAIThreshold in approve/decline functions - 13https://github.com/miraheze/CreateWiki/pull/414 [20:16:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.84, 7.73, 6.98 [20:17:15] [02CreateWiki] 07paladox closed pull request 03#414: Fix using CreateWikiAIThreshold in approve/decline functions - 13https://github.com/miraheze/CreateWiki/pull/414 [20:17:17] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-1 [20:17:20] [02CreateWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/CreateWiki [20:18:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.72, 6.95, 6.78 [20:19:02] [02CreateWiki] 07paladox commented on pull request 03#408: Bump eslint-config-wikimedia from 0.25.0 to 0.25.1 - 13https://github.com/miraheze/CreateWiki/pull/408#issuecomment-1608187011 [20:19:05] [02CreateWiki] 07dependabot[bot] edited pull request 03#408: Bump eslint-config-wikimedia from 0.25.0 to 0.25.1 - 13https://github.com/miraheze/CreateWiki/pull/408 [20:19:20] [02CreateWiki] 07dependabot[bot] edited pull request 03#408: Bump eslint-config-wikimedia from 0.25.0 to 0.25.1 - 13https://github.com/miraheze/CreateWiki/pull/408 [20:19:22] [02miraheze/CreateWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.25.1 [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/682f32f2d961...5926ad80b033 [20:19:25] [02miraheze/CreateWiki] 07dependabot[bot] 035926ad8 - Bump eslint-config-wikimedia from 0.25.0 to 0.25.1 [20:19:26] [02CreateWiki] 07dependabot[bot] synchronize pull request 03#408: Bump eslint-config-wikimedia from 0.25.0 to 0.25.1 - 13https://github.com/miraheze/CreateWiki/pull/408 [20:24:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.75, 6.18, 6.56 [20:25:20] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.93, 1.33, 3.64 [20:27:18] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.54, 1.29, 3.34 [20:30:23] miraheze/CreateWiki - dependabot[bot] the build has errored. [20:37:53] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.34, 3.47, 2.63 [20:38:39] miraheze/CreateWiki - paladox the build passed. [20:39:04] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.25.1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/5926ad80b033...a5550706757b [20:39:06] [02miraheze/CreateWiki] 07paladox 03a555070 - Update ext.createwiki.oouiform.ooui.js [20:39:07] [02CreateWiki] 07paladox synchronize pull request 03#408: Bump eslint-config-wikimedia from 0.25.0 to 0.25.1 - 13https://github.com/miraheze/CreateWiki/pull/408 [20:39:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.19, 3.06, 2.59 [20:48:53] PROBLEM - vise.dayid.org - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.405097961425781 seconds [20:53:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.08, 7.16, 6.40 [20:53:59] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14601 MB (10% inode=99%); [20:54:56] miraheze/CreateWiki - paladox the build passed. [20:55:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.78, 6.10, 6.09 [20:55:18] [02CreateWiki] 07paladox closed pull request 03#408: Bump eslint-config-wikimedia from 0.25.0 to 0.25.1 - 13https://github.com/miraheze/CreateWiki/pull/408 [20:55:21] [02miraheze/CreateWiki] 07paladox deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.25.1 [20:55:23] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/CreateWiki/compare/d31ba34d924d...4d7b3216005f [20:55:25] [02miraheze/CreateWiki] 07dependabot[bot] 034d7b321 - Bump eslint-config-wikimedia from 0.25.0 to 0.25.1 (#408) [20:55:26] [02CreateWiki] 07paladox deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.25.1 - 13https://github.com/miraheze/CreateWiki [21:00:14] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 6.20, 4.30, 2.49 [21:02:15] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.24, 3.79, 2.54 [21:08:13] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.50, 3.20, 2.70 [21:10:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.19, 3.78, 3.07 [21:12:17] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.80, 3.68, 3.14 [21:14:14] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.09, 3.29, 3.05 [21:17:33] RECOVERY - vise.dayid.org - reverse DNS on sslhost is OK: SSL OK - vise.dayid.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [21:37:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.17, 4.42, 3.60 [21:45:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.98, 3.64, 3.60 [21:47:52] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.43, 2.85, 3.31 [21:51:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.62, 3.18, 3.33 [21:52:11] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.13, 4.16, 2.92 [21:53:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.76, 2.63, 3.11 [21:56:11] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.14, 3.81, 3.09 [21:56:26] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:58:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.49, 2.99, 2.87 [22:07:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.65, 3.49, 2.98 [22:09:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.96, 2.85, 2.80 [22:26:15] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [22:35:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.63, 3.41, 2.75 [22:37:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.49, 4.52, 3.22 [22:49:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.53, 3.11, 3.73 [22:53:05] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.18, 3.49, 2.88 [22:53:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.39, 2.23, 3.23 [22:55:05] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.00, 3.22, 2.85 [23:07:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 11.19, 8.98, 5.35 [23:17:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.00, 3.15, 3.87 [23:21:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 16.40, 6.34, 4.73 [23:24:02] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.15, 4.01, 3.15 [23:26:02] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.81, 3.56, 3.09 [23:26:24] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [23:28:02] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.76, 2.83, 2.87 [23:31:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.59, 2.74, 3.85 [23:32:01] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.43, 4.17, 3.45 [23:34:01] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.55, 3.76, 3.40 [23:35:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 13.63, 6.25, 4.83 [23:38:01] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.16, 3.58, 3.39 [23:40:01] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.13, 3.61, 3.44 [23:42:01] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.83, 3.01, 3.24 [23:45:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.77, 3.20, 3.98 [23:46:00] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.64, 3.49, 3.46 [23:48:00] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.79, 2.80, 3.20 [23:51:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.91, 2.43, 3.37 [23:55:17] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL