[00:03:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.48, 3.76, 2.67 [00:05:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.12, 3.39, 2.66 [00:05:33] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 24.66, 18.65, 10.25 [00:07:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.07, 3.69, 3.07 [00:09:29] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 4.52, 10.85, 8.93 [00:09:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.65, 3.23, 2.97 [00:11:27] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 3.54, 8.34, 8.24 [00:21:30] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:21:41] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.46, 3.36, 2.11 [00:23:32] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 2.196 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [00:23:40] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 3.30, 3.38, 2.28 [00:36:46] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.58, 2.96, 2.60 [00:38:45] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.03, 3.25, 2.77 [00:43:11] RECOVERY - nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Fri 15 Sep 2023 19:33:27 GMT +0000. [01:05:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.61, 5.28, 3.46 [01:11:38] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.73, 3.69, 3.45 [01:13:37] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.34, 2.84, 3.16 [01:13:56] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sun 23 Jul 2023 15:38:14 GMT +0000. [01:36:29] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.96, 4.00, 3.12 [01:38:28] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.94, 3.23, 2.95 [01:42:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.82, 3.91, 3.33 [01:44:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.69, 2.78, 2.98 [01:45:56] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address mrlove.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:06:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.27, 4.01, 3.14 [02:08:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.50, 3.71, 3.14 [02:12:14] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.61, 6.72, 4.40 [02:18:12] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.38, 3.44, 3.63 [02:22:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.31, 3.07, 3.40 [02:23:25] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:38:04] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.40, 4.03, 3.24 [02:38:06] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:40:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.20, 3.57, 3.16 [02:42:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.96, 4.99, 3.79 [02:46:01] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.81, 3.59, 3.51 [02:48:00] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.73, 3.01, 3.30 [02:52:17] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [02:54:58] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1164191961 secs [03:00:11] RECOVERY - db121 Backups SQL on db121 is OK: FILE_AGE OK: /var/log/sql-backup.log is 8 seconds old and 273 bytes [03:03:01] PROBLEM - mon141 Backups Grafana on mon141 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 864136 seconds old and 119 bytes [03:04:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.48, 7.84, 4.59 [03:07:10] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [03:08:50] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.85, 5.27, 3.89 [03:10:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.77, 3.90, 3.83 [03:12:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.07, 3.72, 3.58 [03:13:58] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sun 23 Jul 2023 15:38:14 GMT +0000. [03:14:48] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.96, 3.13, 3.37 [03:14:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08651313186 secs [03:16:05] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.64, 2.91, 3.37 [03:20:45] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.70, 3.14, 3.25 [03:22:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.17, 2.42, 2.98 [03:23:30] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.42, 3.47, 2.65 [03:29:30] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.50, 3.85, 3.12 [03:33:30] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.25, 3.30, 3.08 [03:36:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.51, 3.91, 3.07 [03:37:31] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.07, 3.65, 3.26 [03:38:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.71, 3.39, 2.98 [03:39:30] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.44, 3.68, 3.33 [03:41:30] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.12, 3.18, 3.19 [03:45:56] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address mrlove.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:46:55] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.47, 6.12, 3.09 [03:48:48] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.21, 6.87, 4.96 [03:50:12] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.43, 3.51, 3.27 [03:50:48] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.94, 6.71, 5.13 [03:50:49] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.38, 3.03, 2.50 [03:51:23] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:52:06] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.25, 3.42, 3.27 [03:54:00] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.25, 2.87, 3.07 [03:54:48] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.77, 6.65, 5.49 [04:01:48] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 65.78, 47.08, 22.04 [04:02:24] PROBLEM - db121 MariaDB Connections on db121 is CRITICAL: PHP Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Connection failed: Connection refused [04:03:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.74, 4.13, 2.89 [04:03:41] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: CRITICAL - load average: 5.30, 3.57, 2.07 [04:04:18] RECOVERY - db121 MariaDB Connections on db121 is OK: OK connection usage: 1.6%Current connections: 8 [04:05:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.82, 3.34, 2.74 [04:05:30] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.25, 3.40, 3.13 [04:05:41] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 2.17, 2.96, 2.03 [04:07:30] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.42, 3.28, 3.11 [04:09:30] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.96, 3.12, 3.07 [04:19:43] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 1.23, 2.42, 7.83 [04:23:42] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.58, 1.60, 6.30 [04:34:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.60, 3.56, 2.60 [04:35:06] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:40:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.53, 3.69, 3.06 [04:42:13] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.83, 3.87, 3.18 [04:51:22] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [04:54:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.99, 3.31, 3.77 [04:58:07] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.95, 2.07, 3.16 [05:07:30] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.28, 3.47, 2.85 [05:08:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.42, 4.08, 3.45 [05:09:30] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.36, 3.44, 2.92 [05:10:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.36, 3.07, 3.15 [05:11:30] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.31, 3.04, 2.84 [05:13:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.83, 3.14, 2.19 [05:15:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.60, 2.75, 2.16 [05:35:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.67, 3.57, 2.74 [05:38:28] That's not a lot and what do you think isn't useful [05:39:48] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.92, 3.78, 3.01 [05:43:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.74, 3.10, 2.95 [05:51:24] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [06:05:04] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [06:09:00] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.93, 3.93, 3.29 [06:23:23] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:31:25] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.24, 4.53, 2.94 [06:47:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.55, 3.64, 3.77 [06:51:18] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.91, 2.22, 3.16 [06:52:15] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [07:05:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.20, 3.71, 3.95 [07:06:01] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1297318637 secs [07:11:00] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.09, 4.16, 4.03 [07:12:01] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.08574908972 secs [07:13:58] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sun 23 Jul 2023 15:38:14 GMT +0000. [07:14:49] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.61, 2.39, 1.48 [07:16:49] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.72, 2.07, 1.47 [07:21:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.55, 3.91, 3.98 [07:23:25] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:25:00] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.02, 3.97, 3.99 [07:27:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.85, 3.93, 3.97 [07:37:12] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.96, 3.95, 2.48 [07:39:00] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.95, 2.93, 3.37 [07:43:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.39, 3.87, 3.13 [07:45:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.92, 2.82, 2.83 [07:45:56] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address mrlove.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:04:01] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1305780709 secs [08:14:52] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sun 23 Jul 2023 15:38:14 GMT +0000. [08:20:01] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.09164011478 secs [08:21:22] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [08:33:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.19, 3.44, 2.21 [08:37:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.57, 3.58, 2.54 [08:39:12] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.64, 4.32, 2.94 [08:43:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.69, 3.82, 3.14 [08:45:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.09, 2.90, 2.88 [08:45:56] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address mrlove.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:48:01] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset -0.1110490263 secs [08:56:01] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -0.09176743031 secs [09:07:12] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.75, 2.50, 1.93 [09:09:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 15.28, 6.07, 3.24 [09:11:56] PROBLEM - cp32 PowerDNS Recursor on cp32 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:13:51] RECOVERY - cp32 PowerDNS Recursor on cp32 is OK: DNS OK: 0.249 seconds response time. miraheze.org returns 108.175.15.182,2607:f1c0:1800:26f::1,2607:f1c0:1800:8100::1,74.208.203.152 [09:23:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.51, 2.17, 3.65 [09:25:15] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.72, 1.72, 3.30 [09:35:12] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.33, 5.16, 4.10 [09:41:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.48, 3.12, 3.58 [09:43:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.14, 2.44, 3.27 [10:07:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.54, 3.01, 2.46 [10:09:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.51, 2.70, 2.40 [10:35:09] PROBLEM - db142 Backups SQL on db142 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209661 seconds old and 1879 bytes [10:38:06] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [10:53:23] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:07:10] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:09:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.55, 5.82, 4.92 [11:11:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.53, 5.00, 2.95 [11:11:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.07, 5.10, 4.77 [11:15:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.48, 3.27, 2.72 [11:27:40] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.83, 2.46, 1.33 [11:29:40] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.87, 2.74, 1.58 [11:32:58] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1356450021 secs [11:33:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/857298244ec8 [11:33:01] [02miraheze/puppet] 07paladox 038572982 - mariadb: set open table cache to 100k [11:33:02] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [11:33:05] [02puppet] 07paladox opened pull request 03#3270: mariadb: set open table cache to 100k - 13https://github.com/miraheze/puppet/pull/3270 [11:34:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/857298244ec8...bbc365bac8ba [11:34:35] [02miraheze/puppet] 07paladox 03bbc365b - Update mariadb-systemd-override.conf.erb [11:34:36] [02puppet] 07paladox synchronize pull request 03#3270: mariadb: set open table cache to 100k - 13https://github.com/miraheze/puppet/pull/3270 [11:36:14] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [11:37:55] [02puppet] 07paladox synchronize pull request 03#3270: mariadb: set open table cache to 100k - 13https://github.com/miraheze/puppet/pull/3270 [11:37:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bbc365bac8ba...aab6229aea7c [11:37:59] [02miraheze/puppet] 07paladox 03aab6229 - Update mw.cnf.erb [11:38:02] [02puppet] 07paladox edited pull request 03#3270: mariadb: set open table cache to 150k - 13https://github.com/miraheze/puppet/pull/3270 [11:38:09] [02puppet] 07paladox edited pull request 03#3270: mariadb: set open table cache to 150k - 13https://github.com/miraheze/puppet/pull/3270 [11:39:36] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 16.23, 11.16, 7.18 [11:39:40] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.83, 3.05, 1.99 [11:41:34] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.51, 8.67, 6.74 [11:41:40] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 0.60, 2.10, 1.77 [11:48:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.0776925981 secs [11:51:23] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [12:05:18] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:07:12] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.20, 3.50, 2.26 [12:19:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.24, 3.39, 3.30 [12:35:04] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [12:39:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.11, 4.44, 3.07 [12:41:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.77, 3.16, 2.80 [12:43:12] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.64, 3.78, 3.12 [12:45:00] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.84, 3.15, 2.89 [12:45:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.65, 4.69, 3.54 [12:50:02] PROBLEM - cloud11 NTP time on cloud11 is WARNING: NTP WARNING: Offset -0.1077442765 secs [12:52:22] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 18.44, 15.20, 9.24 [12:56:13] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 3.91, 9.38, 8.20 [12:57:18] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.71, 2.67, 3.88 [13:01:17] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.03, 1.83, 3.25 [13:03:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.51, 3.34, 3.08 [13:05:00] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.67, 3.08, 3.02 [13:09:13] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.14, 3.75, 3.48 [13:11:12] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.23, 3.19, 3.31 [13:12:24] RECOVERY - cloud11 NTP time on cloud11 is OK: NTP OK: Offset -0.08603289723 secs [13:23:26] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:37:13] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 15.40, 6.33, 3.70 [13:38:58] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1064745188 secs [13:47:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.84, 2.93, 3.50 [13:49:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.21, 2.40, 3.24 [13:52:22] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [14:02:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09250557423 secs [14:09:07] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:13:22] RECOVERY - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rebirthofthenight.com' will expire on Sun 23 Jul 2023 22:58:18 GMT +0000. [14:21:24] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:27:44] RECOVERY - wiki.dunmoore.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dunmoore.net' will expire on Mon 24 Jul 2023 15:43:21 GMT +0000. [14:37:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.76, 3.27, 2.20 [14:39:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.79, 2.66, 2.11 [14:45:13] PROBLEM - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.rebirthofthenight.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:50:46] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 8.48, 11.55, 8.35 [14:51:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/8aae19925d47 [14:51:03] [02miraheze/puppet] 07paladox 038aae199 - matomo: create 2 more concurrent archivers for a total of 5 [14:51:04] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [14:51:06] [02puppet] 07paladox opened pull request 03#3271: matomo: create 2 more concurrent archivers for a total of 5 - 13https://github.com/miraheze/puppet/pull/3271 [14:51:27] [02puppet] 07paladox synchronize pull request 03#3271: matomo: create 2 more concurrent archivers for a total of 5 - 13https://github.com/miraheze/puppet/pull/3271 [14:51:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8aae19925d47...71057d55c97d [14:51:32] [02miraheze/puppet] 07paladox 0371057d5 - Update init.pp [14:51:46] [02puppet] 07paladox closed pull request 03#3271: matomo: create 2 more concurrent archivers for a total of 5 - 13https://github.com/miraheze/puppet/pull/3271 [14:51:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8e46846329fb...4572dfb9e18b [14:51:49] [02miraheze/puppet] 07paladox 034572dfb - matomo: create 2 more concurrent archivers for a total of 5 (#3271) [14:51:51] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [14:51:52] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [14:52:46] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.33, 9.61, 8.02 [14:59:41] PROBLEM - wiki.dunmoore.net - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.dunmoore.net and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:59:53] [02puppet] 07paladox closed pull request 03#3270: mariadb: set open table cache to 150k - 13https://github.com/miraheze/puppet/pull/3270 [14:59:55] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [14:59:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/4572dfb9e18b...02dda2dc1681 [14:59:58] [02miraheze/puppet] 07paladox 0302dda2d - mariadb: set open table cache to 150k (#3270) [14:59:59] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [15:06:58] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1199959815 secs [15:07:12] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.76, 4.15, 2.67 [15:12:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07595133781 secs [15:15:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.25, 3.22, 3.02 [15:21:23] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [15:36:19] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [15:38:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.62, 4.44, 3.45 [15:40:07] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.73, 3.93, 3.38 [15:42:06] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.61, 3.15, 3.17 [15:49:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a017840dfbca [15:49:45] [02miraheze/puppet] 07paladox 03a017840 - varnish: increase rate limit threshold for static to 1000 over 1s [15:49:46] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [15:49:49] [02puppet] 07paladox opened pull request 03#3272: varnish: increase rate limit threshold for static to 1000 over 1s - 13https://github.com/miraheze/puppet/pull/3272 [15:51:27] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:51:40] [02puppet] 07paladox closed pull request 03#3272: varnish: increase rate limit threshold for static to 1000 over 1s - 13https://github.com/miraheze/puppet/pull/3272 [15:51:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/02dda2dc1681...ac1815447d98 [15:51:42] [02miraheze/puppet] 07paladox 03ac18154 - varnish: increase rate limit threshold for static to 1000 over 1s (#3272) [15:51:45] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [15:51:48] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [15:53:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/455e8af8498a [15:53:05] [02miraheze/puppet] 07paladox 03455e8af - varnish: increase rate limit threshold for non static and math to 20r over 2s [15:53:07] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [15:53:08] [02puppet] 07paladox opened pull request 03#3273: varnish: increase rate limit threshold for non static and math to 20r… - 13https://github.com/miraheze/puppet/pull/3273 [15:53:25] [02puppet] 07paladox closed pull request 03#3273: varnish: increase rate limit threshold for non static and math to 20r… - 13https://github.com/miraheze/puppet/pull/3273 [15:53:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ac1815447d98...71f13854c526 [15:53:30] [02miraheze/puppet] 07paladox 0371f1385 - varnish: increase rate limit threshold for non static and math to 20r… (#3273) [15:53:33] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [15:53:35] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [16:01:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.71, 3.27, 2.27 [16:03:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.57, 2.65, 2.17 [16:05:23] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [16:06:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.39, 4.61, 3.46 [16:07:30] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.82, 2.94, 2.02 [16:08:54] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.92, 3.50, 3.19 [16:09:30] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.46, 2.89, 2.13 [16:10:53] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.81, 2.99, 3.04 [16:14:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ec21bbe99cad [16:14:01] [02miraheze/puppet] 07paladox 03ec21bbe - varnish: overhaul rate limits [16:14:02] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [16:14:05] [02puppet] 07paladox opened pull request 03#3274: varnish: overhaul rate limits - 13https://github.com/miraheze/puppet/pull/3274 [16:15:52] [02puppet] 07paladox edited pull request 03#3274: varnish: overhaul rate limits - 13https://github.com/miraheze/puppet/pull/3274 [16:17:27] [02puppet] 07paladox synchronize pull request 03#3274: varnish: overhaul rate limits - 13https://github.com/miraheze/puppet/pull/3274 [16:17:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ec21bbe99cad...74be72ae787b [16:17:31] [02miraheze/puppet] 07paladox 0374be72a - Update default.vcl [16:21:22] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [16:36:43] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.06, 3.45, 2.61 [16:40:41] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.77, 3.03, 2.67 [16:48:00] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.92, 3.58, 3.14 [16:49:54] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.14, 3.34, 3.10 [16:51:22] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:51:30] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1106225848 secs [16:53:42] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.53, 3.96, 3.40 [16:55:35] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.35, 3.73, 3.38 [16:57:30] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.45, 3.40, 3.31 [17:05:30] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.26, 3.55, 3.32 [17:05:43] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [17:06:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.15, 3.82, 2.88 [17:07:30] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.02, 3.29, 3.25 [17:08:30] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.96, 3.09, 2.74 [17:09:43] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [17:15:30] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09803006053 secs [17:21:23] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [17:47:30] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1016599536 secs [17:49:30] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08890107274 secs [17:51:24] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:08:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.72, 5.08, 3.22 [18:13:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.84, 3.13, 2.18 [18:14:04] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.15, 3.92, 3.40 [18:16:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.35, 3.31, 3.23 [18:17:00] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.97, 3.21, 2.45 [18:19:30] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1164028645 secs [18:21:22] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [18:27:30] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.08607956767 secs [18:32:25] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [18:37:54] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.63, 3.99, 2.92 [18:47:50] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.10, 3.31, 3.56 [18:51:24] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [18:51:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.67, 2.68, 3.27 [18:55:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.19, 3.42, 2.37 [18:57:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.08, 2.53, 2.17 [19:01:21] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query zulaclub.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:09:42] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.07, 4.81, 3.70 [19:11:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.21, 3.99, 3.54 [19:13:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.04, 2.95, 3.22 [19:23:23] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:30:14] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:25] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [19:31:26] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 39.97, 27.62, 13.90 [19:32:09] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.358 second response time [19:32:48] PROBLEM - mw143 Current Load on mw143 is WARNING: WARNING - load average: 10.76, 11.51, 6.92 [19:32:58] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw142 [19:33:35] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw142 [19:33:51] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 1 backends are down. mw142 [19:34:19] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw142 [19:34:48] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 4.42, 8.87, 6.51 [19:34:58] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 15 backends are healthy [19:35:35] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 15 backends are healthy [19:35:50] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 15 backends are healthy [19:36:19] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 15 backends are healthy [19:37:14] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 4.35, 11.18, 10.62 [19:39:10] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 3.97, 8.79, 9.81 [19:52:15] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [19:52:52] PROBLEM - wiki.leapit.no - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406840562820435 seconds [19:57:42] [02puppet] 07paladox closed pull request 03#3264: mariadb: increase innodb_ft_cache_size to 150MB - 13https://github.com/miraheze/puppet/pull/3264 [19:57:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/71f13854c526...7f190fab8043 [19:57:48] [02miraheze/puppet] 07paladox 037f190fa - mariadb: increase innodb_ft_cache_size to 150MB (#3264) [19:57:51] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [19:57:54] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [20:00:25] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [20:03:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.19, 4.13, 3.19 [20:05:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.98, 3.73, 3.14 [20:09:18] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.89, 4.77, 3.71 [20:13:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.57, 3.64, 3.49 [20:15:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.80, 2.91, 3.23 [20:22:24] RECOVERY - wiki.leapit.no - reverse DNS on sslhost is OK: SSL OK - wiki.leapit.no reverse DNS resolves to cp22.miraheze.org - CNAME OK [20:23:22] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:37:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.23, 4.75, 3.63 [20:41:24] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 10.93, 11.26, 7.44 [20:43:22] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.32, 9.06, 7.08 [20:47:01] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.18, 3.15, 3.65 [20:49:01] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.89, 2.39, 3.30 [20:52:15] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [21:06:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.43, 4.07, 3.27 [21:13:56] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sun 23 Jul 2023 15:38:14 GMT +0000. [21:14:50] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.54, 3.24, 3.37 [21:21:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.80, 3.39, 2.35 [21:23:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.46, 2.71, 2.23 [21:37:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.23, 3.99, 3.38 [21:39:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.96, 5.01, 3.79 [21:41:39] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.20, 3.84, 3.50 [21:43:38] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.35, 2.93, 3.20 [21:45:56] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address mrlove.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [21:51:25] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [21:54:52] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.30, 3.50, 2.56 [21:56:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.16, 2.63, 2.35 [22:10:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.46, 3.23, 2.79 [22:12:25] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.19, 2.76, 2.66 [22:18:48] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 15.26, 22.22, 12.05 [22:19:00] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:20:57] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.137 second response time [22:30:48] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.12, 5.70, 7.73 [22:34:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.37, 4.35, 3.22 [22:34:48] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.60, 4.35, 6.69 [22:37:05] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [22:40:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.56, 3.72, 3.30 [22:42:14] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.82, 3.09, 3.13 [23:05:40] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.97, 3.18, 2.00 [23:08:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/303f197ef070 [23:08:46] [02miraheze/puppet] 07paladox 03303f197 - miraheze-backup: use mysqldump to backup [23:08:48] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [23:08:49] [02puppet] 07paladox opened pull request 03#3275: miraheze-backup: use mysqldump to backup - 13https://github.com/miraheze/puppet/pull/3275 [23:11:40] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.77, 3.95, 2.75 [23:12:58] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1280289292 secs [23:16:52] [02puppet] 07paladox synchronize pull request 03#3275: miraheze-backup: use mysqldump to backup - 13https://github.com/miraheze/puppet/pull/3275 [23:16:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/303f197ef070...ca9e33a2c10c [23:16:58] [02miraheze/puppet] 07paladox 03ca9e33a - Update miraheze-backup.py.erb [23:17:40] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.52, 3.18, 2.79 [23:23:23] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:27:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ca9e33a2c10c...c295a889244d [23:27:35] [02miraheze/puppet] 07paladox 03c295a88 - Update miraheze-backup.py.erb [23:27:36] [02puppet] 07paladox synchronize pull request 03#3275: miraheze-backup: use mysqldump to backup - 13https://github.com/miraheze/puppet/pull/3275 [23:34:50] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.36, 3.88, 2.99 [23:36:50] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.36, 3.39, 2.92 [23:44:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08753657341 secs [23:52:58] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1024409831 secs [23:54:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08924350142 secs [23:55:44] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.91, 3.66, 3.20 [23:59:42] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.11, 3.15, 3.14