[00:01:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.11, 3.73, 2.70 [00:01:53] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.44, 6.61, 5.08 [00:03:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.18, 3.48, 2.74 [00:03:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.33, 3.70, 3.38 [00:03:54] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.34, 6.17, 5.12 [00:05:04] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:05:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.16, 3.05, 2.67 [00:05:39] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.68, 5.35, 4.05 [00:07:40] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.45, 2.76, 1.75 [00:09:40] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.69, 2.66, 1.83 [00:12:58] PROBLEM - cp23 NTP time on cp23 is CRITICAL: NTP CRITICAL: Offset 0.6051000059 secs [00:13:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.87, 6.00, 3.97 [00:16:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.001542568207 secs [00:21:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.82, 3.91, 3.81 [00:25:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.63, 3.72, 3.72 [00:27:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.80, 2.96, 3.44 [00:27:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.41, 2.51, 3.79 [00:29:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.61, 2.50, 3.21 [00:31:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.88, 7.29, 5.86 [00:33:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.78, 4.61, 4.14 [00:33:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.03, 7.18, 5.99 [00:35:04] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [00:39:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.92, 6.68, 6.13 [00:43:32] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.68, 7.22, 5.48 [00:45:32] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 5.46, 6.44, 5.40 [00:46:31] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.94, 6.19, 5.23 [00:46:33] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.39, 6.63, 5.93 [00:47:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.25, 7.81, 6.81 [00:47:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.27, 2.75, 3.73 [00:48:31] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.96, 6.15, 5.33 [00:48:32] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.34, 6.38, 5.92 [00:52:25] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.98, 7.99, 6.49 [00:53:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.29, 7.84, 7.29 [00:53:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.63, 2.27, 3.21 [00:54:21] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.28, 7.04, 6.30 [01:00:08] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 6.15, 6.38, 6.25 [01:05:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.81, 8.19, 7.56 [01:09:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.77, 7.25, 7.35 [01:10:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.01, 3.54, 3.21 [01:12:25] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.95, 2.95, 3.04 [01:12:43] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 9.76, 8.08, 7.05 [01:13:22] RECOVERY - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rebirthofthenight.com' will expire on Sun 23 Jul 2023 22:58:18 GMT +0000. [01:13:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.28, 7.73, 7.50 [01:13:50] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.66, 3.41, 3.08 [01:14:04] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sun 23 Jul 2023 15:38:14 GMT +0000. [01:15:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.93, 7.30, 7.36 [01:15:44] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.26, 3.19, 3.03 [01:21:23] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [01:21:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.28, 7.87, 7.61 [01:22:21] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.77, 7.50, 7.38 [01:23:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.80, 7.10, 7.36 [01:24:17] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.12, 7.74, 7.49 [01:26:12] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.02, 7.34, 7.39 [01:27:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.23, 7.36, 7.37 [01:29:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.30, 6.85, 7.18 [01:31:59] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.70, 7.52, 7.43 [01:34:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 10.06, 7.04, 4.29 [01:35:50] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.46, 7.78, 7.64 [01:37:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.54, 7.33, 7.18 [01:37:46] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.81, 8.27, 7.84 [01:38:58] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.1138916612 secs [01:39:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.54, 6.86, 7.01 [01:39:41] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.16, 7.53, 7.62 [01:41:38] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.97, 7.99, 7.77 [01:42:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.09915056825 secs [01:45:13] PROBLEM - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.rebirthofthenight.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:45:56] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address mrlove.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:49:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.95, 7.47, 7.15 [01:50:13] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.49, 2.56, 3.78 [01:51:22] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:51:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.34, 7.14, 7.08 [01:54:12] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.98, 2.28, 3.38 [01:59:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.12, 6.52, 6.80 [02:01:32] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 5.43, 7.47, 7.98 [02:03:32] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.45, 7.93, 8.09 [02:04:09] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 14.33, 7.53, 5.00 [02:05:32] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.29, 7.52, 7.93 [02:07:32] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 9.71, 7.97, 8.02 [02:10:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7f190fab8043...e2cddc791aa0 [02:10:53] [02miraheze/puppet] 07paladox 03e2cddc7 - db112: remove empty string from role::db::monthly_misc [02:11:02] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14581 MB (10% inode=99%); [02:11:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.50, 8.13, 7.30 [02:12:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e2cddc791aa0...031a1e8e9c64 [02:12:04] [02miraheze/puppet] 07paladox 03031a1e8 - mariadb: Remove DB_backups cron [02:13:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.94, 7.57, 7.19 [02:14:05] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.15, 2.95, 3.93 [02:15:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.87, 8.37, 7.52 [02:19:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.47, 7.79, 7.49 [02:20:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.20, 4.17, 4.00 [02:21:24] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:25:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.14, 7.35, 7.34 [02:27:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.39, 7.24, 7.32 [02:29:59] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.40, 3.28, 3.89 [02:31:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.11, 8.42, 7.78 [02:33:32] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 5.82, 7.15, 7.99 [02:33:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 15.11, 5.81, 4.51 [02:35:05] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:37:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.07, 7.30, 7.56 [02:38:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c295a889244d...6daa299d4fec [02:38:41] [02miraheze/puppet] 07paladox 036daa299 - Update miraheze-backup.py.erb [02:38:44] [02puppet] 07paladox synchronize pull request 03#3275: miraheze-backup: use mysqldump to backup - 13https://github.com/miraheze/puppet/pull/3275 [02:41:32] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.68, 7.71, 7.85 [02:41:35] [02puppet] 07paladox closed pull request 03#3275: miraheze-backup: use mysqldump to backup - 13https://github.com/miraheze/puppet/pull/3275 [02:41:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/031a1e8e9c64...67a57a8bf112 [02:41:38] [02miraheze/puppet] 07paladox 0367a57a8 - miraheze-backup: use mysqldump to backup (#3275) [02:41:41] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [02:41:44] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [02:41:47] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [02:41:50] [02puppet] 07paladox closed pull request 03#3274: varnish: overhaul rate limits - 13https://github.com/miraheze/puppet/pull/3274 [02:41:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/67a57a8bf112...17609ef2cae3 [02:41:54] [02miraheze/puppet] 07paladox 0317609ef - varnish: overhaul rate limits (#3274) [02:41:56] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [02:43:32] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.62, 7.45, 7.75 [02:44:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/17609ef2cae3...6c9b22c4207a [02:44:45] [02miraheze/puppet] 07paladox 036c9b22c - varnish: use X-Real-IP not x-client-ip for now [02:45:32] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 9.37, 8.19, 7.98 [02:47:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6c9b22c4207a...cc64e0ab1ac2 [02:47:19] [02miraheze/puppet] 07paladox 03cc64e0a - varnish: Fix syntax error [02:47:32] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.96, 8.00, 7.94 [02:47:36] PROBLEM - cp23 Puppet on cp23 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [02:48:09] PROBLEM - cp22 Puppet on cp22 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[retry-load-new-vcl-file] [02:49:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.60, 8.27, 7.80 [02:49:32] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 9.31, 8.33, 8.05 [02:49:53] PROBLEM - cp33 Puppet on cp33 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [02:50:46] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [02:51:36] RECOVERY - cp23 Puppet on cp23 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [02:52:08] RECOVERY - cp22 Puppet on cp22 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [02:53:53] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [02:55:34] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.89, 3.09, 2.43 [02:57:31] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.11, 3.00, 2.48 [02:57:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.76, 7.28, 7.72 [02:58:42] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:59:29] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.87, 2.47, 3.67 [03:01:06] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.60, 7.24, 7.79 [03:01:18] PROBLEM - db112 Backups SQL on db112 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209672 seconds old and 362 bytes [03:01:23] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.57, 1.87, 3.31 [03:03:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.78, 7.97, 7.81 [03:05:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.16, 3.85, 2.91 [03:05:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.83, 7.72, 7.73 [03:06:58] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.83, 7.32, 7.59 [03:07:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.42, 2.98, 2.70 [03:10:52] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.09, 7.95, 7.88 [03:12:51] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 11.12, 8.87, 8.20 [03:26:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.97, 5.99, 6.66 [03:34:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.91, 4.41, 3.04 [03:42:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.56, 3.01, 3.33 [04:00:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.13, 6.80, 6.52 [04:02:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.64, 6.35, 6.38 [04:29:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.49, 6.76, 6.51 [04:31:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.07, 7.26, 6.72 [04:33:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.07, 6.78, 6.61 [04:50:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.53, 6.84, 6.10 [04:52:20] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.06, 6.46, 6.05 [05:01:18] PROBLEM - swiftac111 PowerDNS Recursor on swiftac111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:03:22] RECOVERY - swiftac111 PowerDNS Recursor on swiftac111 is OK: DNS OK: 6.149 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [05:07:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.73, 3.36, 2.56 [05:08:03] PROBLEM - swiftac111 PowerDNS Recursor on swiftac111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:08:12] PROBLEM - swiftac111 SSH on swiftac111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:09:39] PROBLEM - swiftac111 Backups Swift Account Container on swiftac111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:09:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.48, 3.24, 2.62 [05:09:59] PROBLEM - swiftac111 Puppet on swiftac111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:10:08] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [05:10:09] PROBLEM - swiftac111 ferm_active on swiftac111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:10:12] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [05:11:16] PROBLEM - drowiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for drowiki.com could not be found [05:12:14] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized [05:12:25] PROBLEM - swiftac111 NTP time on swiftac111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:12:57] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 401 Unauthorized [05:13:39] PROBLEM - swiftac111 conntrack_table_size on swiftac111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:14:47] PROBLEM - swiftac111 Swift Account Service on swiftac111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:15:50] PROBLEM - swiftac111 Swift Container Service on swiftac111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:17:49] RECOVERY - swiftac111 Swift Container Service on swiftac111 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:202 port 6001 [05:21:37] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address drowiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:21:37] PROBLEM - mercy.theking1543.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'mercy.theking1543.eu.org' expires in 7 day(s) (Fri 14 Jul 2023 05:13:02 GMT +0000). [05:22:15] PROBLEM - swiftac111 Swift Container Service on swiftac111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:23:00] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.030 second response time [05:23:07] RECOVERY - swiftac111 Swift Account Service on swiftac111 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:202 port 6002 [05:24:00] RECOVERY - swiftac111 NTP time on swiftac111 is OK: NTP OK: Offset -0.002524882555 secs [05:24:01] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.080 second response time [05:24:04] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.019 second response time [05:24:11] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.045 second response time [05:24:13] RECOVERY - swiftac111 Swift Container Service on swiftac111 is OK: TCP OK - 0.005 second response time on 2a10:6740::6:202 port 6001 [05:28:12] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [05:28:13] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [05:29:01] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [05:30:08] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [05:30:17] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 4.319 second response time [05:30:59] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.057 second response time [05:32:03] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.024 second response time [05:32:04] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.019 second response time [05:32:23] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 6.73, 6.91, 4.35 [05:34:23] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 12.33, 8.80, 5.33 [05:39:42] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 6956 MB (5% inode=99%); [05:40:03] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [05:40:04] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [05:40:18] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [05:40:32] RECOVERY - drowiki.com - reverse DNS on sslhost is OK: SSL OK - drowiki.com reverse DNS resolves to cp22.miraheze.org - CNAME FLAT [05:40:51] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [05:42:17] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.065 second response time [05:42:49] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.030 second response time [05:44:01] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.028 second response time [05:44:02] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.017 second response time [05:50:29] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized [05:50:59] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 401 Unauthorized [05:51:01] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized [05:51:07] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 401 Unauthorized [05:52:08] PROBLEM - swiftac111 NTP time on swiftac111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:54:05] PROBLEM - swiftac111 Swift Account Service on swiftac111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:54:15] PROBLEM - swiftac111 Swift Container Service on swiftac111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:54:23] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.81, 5.54, 7.60 [05:54:50] PROBLEM - swiftac111 Disk Space on swiftac111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:56:23] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 9.57, 6.25, 7.56 [05:58:23] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.67, 5.56, 7.14 [06:00:23] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 7.74, 8.06, 8.00 [06:02:23] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.31, 6.65, 7.48 [06:05:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 22.82, 10.21, 4.91 [06:06:23] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.87, 5.34, 6.77 [06:09:06] RECOVERY - swiftac111 SSH on swiftac111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:09:06] RECOVERY - swiftac111 Swift Container Service on swiftac111 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:202 port 6001 [06:09:12] RECOVERY - swiftac111 Disk Space on swiftac111 is OK: DISK OK - free space: / 115085 MB (93% inode=95%); [06:09:18] RECOVERY - swiftac111 NTP time on swiftac111 is OK: NTP OK: Offset -0.004033505917 secs [06:09:42] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 12502 MB (9% inode=99%); [06:09:51] RECOVERY - swiftac111 Backups Swift Account Container on swiftac111 is OK: FILE_AGE OK: /var/log/swift-account-container-backup.log is 345634 seconds old and 107 bytes [06:10:24] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.124 second response time [06:10:32] RECOVERY - swiftac111 conntrack_table_size on swiftac111 is OK: OK: nf_conntrack is 6 % full [06:10:43] RECOVERY - swiftac111 ferm_active on swiftac111 is OK: OK ferm input default policy is set [06:10:47] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.041 second response time [06:10:51] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.018 second response time [06:10:51] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.019 second response time [06:10:55] RECOVERY - swiftac111 Swift Account Service on swiftac111 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:202 port 6002 [06:10:57] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:11:22] RECOVERY - swiftac111 PowerDNS Recursor on swiftac111 is OK: DNS OK: 0.246 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [06:11:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.95, 3.45, 2.75 [06:13:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.34, 3.73, 2.93 [06:15:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.70, 3.71, 3.02 [06:16:23] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 11.86, 9.24, 7.77 [06:21:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.68, 3.29, 3.08 [06:21:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.26, 7.26, 5.82 [06:22:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.03, 2.63, 3.82 [06:23:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.64, 7.02, 5.90 [06:25:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 7.36, 6.15 [06:25:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.25, 3.80, 3.34 [06:26:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.99, 1.74, 3.17 [06:27:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.32, 6.81, 6.08 [06:27:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.91, 3.74, 3.37 [06:29:10] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.69, 6.49, 6.06 [06:31:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.34, 4.00, 3.56 [06:33:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.07, 6.85, 6.34 [06:33:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.87, 3.98, 3.62 [06:34:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.08, 6.72, 6.35 [06:36:23] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 0.57, 4.48, 7.80 [06:38:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.31, 7.73, 6.92 [06:39:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.14, 3.86, 3.67 [06:40:22] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.27, 2.82, 6.31 [06:40:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.24, 7.99, 7.09 [06:41:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.54, 5.33, 3.73 [06:42:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.95, 7.50, 7.02 [06:43:02] PROBLEM - drowiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for drowiki.com could not be found [06:43:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.36, 3.83, 3.73 [06:47:29] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.93, 3.80, 3.58 [06:49:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.34, 2.95, 3.30 [06:55:26] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.16, 3.89, 3.76 [06:56:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.52, 7.51, 7.15 [06:58:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.56, 7.44, 7.16 [06:59:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.48, 3.81, 3.76 [07:05:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.59, 3.07, 2.69 [07:05:50] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 4.96, 6.14, 7.68 [07:07:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.86, 3.02, 3.40 [07:07:36] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.56, 2.80, 2.63 [07:11:42] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 3.23, 4.94, 6.68 [07:12:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.17, 7.70, 7.23 [07:14:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.59, 7.09, 7.05 [07:19:28] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.66, 6.82, 6.85 [07:21:27] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 5.75, 6.36, 6.68 [07:26:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.09, 7.60, 7.28 [07:28:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.89, 7.23, 7.17 [07:30:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.99, 7.83, 7.40 [07:32:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.02, 7.70, 7.42 [07:38:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.34, 4.07, 2.69 [07:42:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.50, 3.85, 2.94 [07:44:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.70, 3.25, 2.84 [07:49:55] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.85, 6.73, 6.35 [07:50:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.28, 7.86, 7.41 [07:51:53] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 6.07, 6.61, 6.36 [07:52:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.85, 7.70, 7.40 [07:55:44] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 5.60, 6.93, 6.60 [07:56:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.52, 7.60, 7.39 [07:57:41] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 6.41, 6.61, 6.51 [07:58:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.75, 7.31, 7.31 [08:01:32] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.98, 7.01, 6.73 [08:03:30] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.24, 7.40, 6.90 [08:05:27] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 3.91, 6.21, 6.54 [08:06:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.04, 8.17, 7.63 [08:08:19] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.64, 6.81, 6.48 [08:08:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 8.00, 7.82, 7.56 [08:10:34] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.56, 2.62, 1.73 [08:10:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.03, 7.81, 7.58 [08:12:17] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.20, 6.46, 6.39 [08:12:34] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.19, 2.47, 1.78 [08:12:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.36, 3.52, 2.63 [08:12:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.69, 7.38, 7.45 [08:14:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.73, 3.31, 2.67 [08:16:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.75, 8.03, 7.72 [08:18:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.38, 7.73, 7.64 [08:21:10] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.68, 7.17, 6.73 [08:23:09] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.87, 6.69, 6.61 [08:26:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.69, 7.57, 7.52 [08:28:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.31, 6.80, 7.25 [08:32:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.92, 7.70, 7.46 [08:38:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.56, 7.90, 7.63 [08:43:04] RECOVERY - drowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'drowiki.com' will expire on Tue 25 Jul 2023 15:53:31 GMT +0000. [08:45:39] RECOVERY - wiki.dunmoore.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dunmoore.net' will expire on Mon 24 Jul 2023 15:43:21 GMT +0000. [08:46:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:51:48] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.15, 6.67, 6.43 [08:52:47] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.12, 7.09, 6.34 [08:53:47] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.99, 6.55, 6.42 [08:54:44] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.09, 7.17, 6.46 [08:54:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.12, 7.62, 7.48 [08:56:41] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 5.41, 6.38, 6.25 [08:58:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.55, 7.64, 7.53 [09:00:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.32, 7.66, 7.53 [09:04:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.34, 7.71, 7.62 [09:06:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.50, 8.84, 8.04 [09:08:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.32, 7.31, 6.74 [09:10:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.08, 7.12, 6.74 [09:10:50] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.27, 3.72, 2.31 [09:13:46] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address drowiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:14:41] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.75, 6.48, 6.59 [09:14:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.56, 4.00, 2.95 [09:14:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.58, 7.63, 7.93 [09:15:55] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [09:16:37] PROBLEM - wiki.dunmoore.net - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.dunmoore.net and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:16:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.75, 3.17, 2.76 [09:16:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.49, 8.38, 8.19 [09:25:35] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.84, 7.39, 7.00 [09:25:46] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.16, 7.05, 6.31 [09:27:43] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.44, 7.40, 6.54 [09:29:41] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 4.52, 6.40, 6.28 [09:33:30] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.63, 7.82, 7.24 [09:34:33] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.99, 6.59, 6.17 [09:35:29] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.30, 7.44, 7.18 [09:36:32] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.68, 6.49, 6.18 [09:40:18] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.407325506210327 seconds [09:44:36] PROBLEM - db101 Backups SQL on db101 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864230 seconds old and 1753 bytes [09:59:17] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.32, 5.83, 6.57 [10:00:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.12, 6.77, 7.83 [10:11:21] RECOVERY - drowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'drowiki.com' will expire on Tue 25 Jul 2023 15:53:31 GMT +0000. [10:11:39] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.38, 4.09, 2.57 [10:12:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.49, 8.43, 8.00 [10:13:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.04, 6.82, 6.62 [10:14:37] RECOVERY - wiki.dunmoore.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dunmoore.net' will expire on Mon 24 Jul 2023 15:43:21 GMT +0000. [10:17:05] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.77, 6.79, 6.65 [10:18:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.21, 7.80, 7.93 [10:23:00] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.12, 6.91, 6.71 [10:28:56] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.73, 6.21, 6.55 [10:29:01] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.81, 7.47, 6.42 [10:30:50] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.81, 2.34, 3.94 [10:30:58] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 4.14, 6.03, 6.02 [10:34:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.00, 1.56, 3.24 [10:40:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.07, 7.95, 7.71 [10:41:45] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.97, 7.27, 6.93 [10:42:03] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address drowiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:42:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.59, 7.76, 7.68 [10:43:28] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.89, 2.37, 1.36 [10:45:28] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.25, 1.89, 1.30 [10:45:35] PROBLEM - wiki.dunmoore.net - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.dunmoore.net and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:45:43] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.57, 6.71, 6.78 [10:48:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.13, 7.50, 7.53 [10:50:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.97, 7.71, 7.60 [10:54:18] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.73, 7.01, 6.08 [10:54:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.61, 7.14, 6.94 [10:54:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.31, 7.66, 7.57 [10:56:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.96, 7.81, 7.64 [10:58:12] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 4.38, 6.05, 5.94 [11:00:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.52, 7.54, 7.53 [11:02:41] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.21, 6.46, 6.73 [11:04:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.37, 7.45, 7.54 [11:06:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.86, 2.91, 1.95 [11:06:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.17, 7.47, 7.52 [11:08:07] RECOVERY - wikipedia.yogfront.ooo - reverse DNS on sslhost is OK: SSL OK - wikipedia.yogfront.ooo reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:08:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.05, 3.46, 2.26 [11:08:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.30, 7.72, 7.63 [11:10:49] RECOVERY - drowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'drowiki.com' will expire on Tue 25 Jul 2023 15:53:31 GMT +0000. [11:16:32] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/38c0f3c4bf19...233f1143e498 [11:16:33] [02miraheze/CreateWiki] 07translatewiki 03233f114 - Localisation updates from https://translatewiki.net. [11:17:40] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.51, 6.49, 5.79 [11:18:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.57, 7.47, 7.42 [11:19:37] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 5.43, 5.97, 5.67 [11:23:38] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:24:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.69, 3.16, 3.59 [11:26:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.13, 2.47, 3.28 [11:32:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.40, 7.48, 7.71 [11:38:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.10, 7.37, 7.57 [11:41:26] miraheze/CreateWiki - translatewiki the build passed. [11:41:30] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address drowiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [11:42:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.86, 7.74, 7.68 [11:46:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.32, 8.42, 7.93 [11:52:45] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [11:55:41] PROBLEM - ie.unigon.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ie.unigon.net' expires in 15 day(s) (Sat 22 Jul 2023 11:40:36 GMT +0000). [11:56:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.16, 7.55, 7.89 [11:57:30] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/885cc0c3cad8...5d2f48a5e5e0 [11:57:32] [02miraheze/ssl] 07MirahezeSSLBot 035d2f48a - Bot: Update SSL cert for ie.unigon.net [11:58:50] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.06, 6.96, 6.08 [12:00:47] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 3.98, 5.84, 5.78 [12:02:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.16, 8.02, 7.96 [12:04:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.76, 7.72, 7.84 [12:05:11] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.01, 7.10, 6.54 [12:06:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.35, 4.88, 2.86 [12:06:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.26, 8.59, 8.14 [12:07:10] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.21, 6.81, 6.51 [12:09:09] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.11, 6.73, 6.52 [12:10:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.34, 7.86, 7.95 [12:12:43] RECOVERY - wiki.dunmoore.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dunmoore.net' will expire on Mon 24 Jul 2023 15:43:21 GMT +0000. [12:12:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.04, 3.88, 3.17 [12:14:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.62, 3.16, 2.99 [12:16:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.82, 7.91, 7.88 [12:18:18] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.63, 6.77, 5.89 [12:18:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.05, 7.13, 7.60 [12:20:15] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 5.73, 6.26, 5.81 [12:25:22] RECOVERY - ie.unigon.net - LetsEncrypt on sslhost is OK: OK - Certificate 'ie.unigon.net' will expire on Wed 04 Oct 2023 10:57:22 GMT +0000. [12:26:04] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.55, 7.31, 6.43 [12:26:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.02, 7.92, 7.73 [12:28:01] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 6.01, 6.67, 6.29 [12:28:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.61, 6.86, 6.45 [12:30:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.83, 7.87, 7.75 [12:30:57] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.20, 6.18, 6.25 [12:36:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.25, 4.66, 3.00 [12:38:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.20, 3.83, 2.92 [12:40:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.27, 3.26, 2.81 [12:40:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.43, 7.77, 7.58 [12:44:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.43, 7.82, 7.72 [12:46:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.46, 9.00, 8.18 [12:53:38] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:55:27] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.13, 6.80, 6.00 [12:57:27] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 4.50, 5.97, 5.80 [12:58:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.38, 7.51, 7.91 [13:00:41] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.64, 7.06, 6.54 [13:00:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.91, 8.10, 8.07 [13:02:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.05, 7.59, 7.88 [13:06:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.55, 8.46, 8.16 [13:08:04] RECOVERY - drowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'drowiki.com' will expire on Tue 25 Jul 2023 15:53:31 GMT +0000. [13:10:37] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.40, 6.46, 6.59 [13:14:29] PROBLEM - wiki.dunmoore.net - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.dunmoore.net and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:21:27] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.45, 6.42, 5.65 [13:22:45] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [13:23:27] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 4.82, 5.94, 5.58 [13:28:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.52, 7.67, 7.97 [13:30:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.15, 7.97, 8.03 [13:36:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.45, 5.05, 2.90 [13:42:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.57, 3.30, 2.90 [13:44:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.46, 7.19, 7.79 [13:46:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.65, 7.93, 8.00 [13:51:27] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.08, 6.16, 5.44 [13:51:52] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:53:27] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 6.18, 6.14, 5.52 [14:00:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.44, 7.56, 7.96 [14:02:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.46, 8.00, 8.05 [14:04:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.33, 7.50, 7.85 [14:06:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.15, 4.13, 2.75 [14:06:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.92, 8.59, 8.20 [14:08:02] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.18, 7.33, 6.80 [14:09:46] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address drowiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:10:01] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.31, 6.97, 6.74 [14:12:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.40, 3.58, 3.13 [14:14:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.89, 2.65, 2.84 [14:16:54] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.80, 6.94, 6.54 [14:18:53] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.21, 6.37, 6.39 [14:19:56] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.34, 6.53, 6.71 [14:22:52] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 11.32, 9.66, 6.16 [14:24:52] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 7.38, 8.69, 6.23 [14:27:35] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 9.53, 7.27, 6.08 [14:29:32] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 4.25, 6.06, 5.78 [14:34:13] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1691254079 secs [14:36:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 10.88, 4.31, 2.61 [14:39:42] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sun 23 Jul 2023 15:38:14 GMT +0000. [14:40:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.65, 3.57, 2.81 [14:42:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.42, 2.83, 2.63 [14:43:36] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.19, 7.45, 6.51 [14:43:44] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.11, 8.01, 7.12 [14:50:16] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.17, 6.45, 5.76 [14:52:13] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 4.90, 5.80, 5.60 [14:54:13] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08853197098 secs [14:59:41] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.44, 7.53, 7.85 [15:01:26] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.05, 7.37, 7.93 [15:07:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.81, 4.22, 2.77 [15:09:41] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 3.72, 5.25, 6.62 [15:10:53] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.49, 3.53, 2.83 [15:11:25] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address mrlove.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:14:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.15, 2.72, 2.72 [15:15:18] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.44, 6.95, 7.07 [15:17:17] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.25, 6.94, 7.05 [15:21:14] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.21, 6.43, 6.80 [15:26:19] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.77, 6.74, 5.78 [15:30:13] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 4.38, 6.16, 5.82 [15:36:21] PROBLEM - cp32 PowerDNS Recursor on cp32 is CRITICAL: CRITICAL - Plugin timed out while executing system call [15:36:59] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 26.70, 12.59, 5.98 [15:37:25] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:59] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:40:39] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:41:13] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sun 23 Jul 2023 15:38:14 GMT +0000. [15:42:32] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 30 minutes ago with 0 failures [15:42:36] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 15 backends are healthy [15:42:42] RECOVERY - cp32 PowerDNS Recursor on cp32 is OK: DNS OK: 0.444 seconds response time. miraheze.org returns 108.175.15.182,2607:f1c0:1800:26f::1,2607:f1c0:1800:8100::1,74.208.203.152 [15:43:29] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:44:00] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.64, 7.25, 6.73 [15:45:47] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.24, 6.41, 5.90 [15:47:44] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 5.97, 6.12, 5.84 [15:49:57] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.50, 6.45, 6.55 [15:56:27] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.77, 7.37, 6.56 [15:57:29] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:58:25] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 5.85, 6.61, 6.37 [16:10:27] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 7.62, 10.47, 6.81 [16:10:37] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 8.90, 11.07, 7.10 [16:12:27] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.93, 8.46, 6.52 [16:12:32] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 4.67, 8.89, 6.75 [16:12:32] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.70, 3.16, 2.60 [16:14:28] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.76, 2.58, 2.45 [16:14:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.84, 6.76, 6.51 [16:16:41] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.43, 6.74, 6.54 [16:16:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.12, 2.12, 3.96 [16:20:48] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.61, 7.29, 6.33 [16:22:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.37, 7.40, 6.92 [16:22:45] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 5.01, 6.50, 6.16 [16:22:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.87, 1.61, 3.16 [16:26:36] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.87, 7.37, 6.60 [16:26:41] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.95, 6.60, 6.70 [16:27:03] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [16:28:33] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 4.47, 6.44, 6.36 [16:34:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.59, 7.36, 6.95 [16:36:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 20.14, 8.71, 4.71 [16:37:25] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:24] PROBLEM - cp32 PowerDNS Recursor on cp32 is CRITICAL: CRITICAL - Plugin timed out while executing system call [16:38:41] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.02, 6.69, 6.77 [16:39:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 108.175.15.182/cpweb, 2607:f1c0:1800:8100::1/cpweb [16:41:29] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:42:39] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:42:50] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address mrlove.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:43:34] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:43:34] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:43:35] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 31 minutes ago with 0 failures [16:44:38] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 15 backends are healthy [16:44:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.43, 6.95, 6.88 [16:44:49] RECOVERY - cp32 PowerDNS Recursor on cp32 is OK: DNS OK: 7.142 seconds response time. miraheze.org returns 108.175.15.182,2607:f1c0:1800:26f::1,2607:f1c0:1800:8100::1,74.208.203.152 [16:48:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 11.18, 7.87, 7.20 [16:50:13] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1052105725 secs [16:51:58] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.83, 6.02, 5.62 [16:52:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.66, 7.12, 7.03 [16:53:55] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 6.51, 6.09, 5.69 [16:57:47] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 9.58, 7.93, 6.54 [16:58:41] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.53, 6.33, 6.71 [16:59:44] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.40, 7.57, 6.59 [17:01:41] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 2.98, 5.98, 6.14 [17:10:13] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.07936921716 secs [17:12:38] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sun 23 Jul 2023 15:38:14 GMT +0000. [17:21:06] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.51, 6.81, 6.23 [17:24:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.94, 1.89, 3.77 [17:25:00] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.72, 7.48, 6.59 [17:26:57] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.45, 7.24, 6.62 [17:28:54] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 3.86, 6.18, 6.32 [17:30:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.30, 1.86, 3.17 [17:44:21] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address mrlove.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [17:49:27] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.97, 6.79, 6.20 [17:50:11] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.46, 7.21, 6.86 [17:53:27] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 5.40, 6.39, 6.20 [17:54:09] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.72, 6.70, 6.75 [18:04:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.40, 6.47, 7.82 [18:06:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.96, 4.08, 2.84 [18:08:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.17, 3.57, 2.79 [18:12:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.23, 2.68, 2.64 [18:22:16] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.98, 6.71, 6.05 [18:22:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.15, 7.14, 7.12 [18:24:13] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 4.81, 6.04, 5.89 [18:24:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.91, 6.75, 6.99 [18:28:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 13.55, 8.93, 7.71 [18:31:21] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.13, 7.18, 5.80 [18:31:49] !log (re)started script on swiftobject* to resync swift metadata [18:31:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:33:20] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.62, 6.85, 5.85 [18:39:17] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.18, 6.32, 5.98 [18:42:30] RECOVERY - wiki.dunmoore.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dunmoore.net' will expire on Mon 24 Jul 2023 15:43:21 GMT +0000. [18:53:27] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.89, 6.39, 5.36 [18:53:37] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [18:55:27] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 4.85, 5.69, 5.23 [18:59:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.37, 6.97, 6.50 [18:59:25] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.33, 6.86, 6.51 [19:02:13] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1285328269 secs [19:03:22] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.86, 6.66, 6.53 [19:05:00] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.06, 7.46, 6.83 [19:06:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.05, 6.91, 6.71 [19:07:46] RECOVERY - drowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'drowiki.com' will expire on Tue 25 Jul 2023 15:53:31 GMT +0000. [19:08:58] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.45, 6.70, 6.65 [19:14:30] PROBLEM - wiki.dunmoore.net - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.dunmoore.net and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:22:09] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 12.57, 8.69, 7.28 [19:22:45] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:22:50] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.97, 7.19, 6.79 [19:24:08] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.74, 7.78, 7.12 [19:24:49] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.20, 7.40, 6.90 [19:26:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.37, 8.89, 7.62 [19:26:48] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.60, 7.50, 7.00 [19:28:13] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08232820034 secs [19:34:02] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.56, 7.45, 7.61 [19:34:44] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.07, 7.29, 7.06 [19:36:01] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.76, 8.08, 7.80 [19:38:42] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.83, 7.69, 7.33 [19:38:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.26, 5.23, 3.38 [19:39:46] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address drowiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:39:59] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.29, 7.53, 7.67 [19:40:13] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [19:42:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.84, 3.82, 3.28 [19:46:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.88, 3.28, 3.18 [19:50:36] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.05, 6.35, 6.74 [19:51:52] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [19:53:50] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.30, 6.12, 6.77 [19:54:13] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1446484327 secs [19:54:32] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.04, 6.90, 6.85 [19:56:31] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.73, 6.71, 6.78 [19:57:46] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.83, 6.62, 6.83 [19:59:45] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.82, 6.30, 6.69 [20:02:13] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08056032658 secs [20:06:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.16, 4.14, 3.05 [20:08:24] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.47, 6.63, 6.50 [20:09:34] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [20:09:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.38, 6.60, 6.58 [20:10:23] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.69, 6.63, 6.52 [20:13:38] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.43, 6.25, 6.47 [20:13:42] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7960 MB (5% inode=99%); [20:14:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.17, 3.17, 3.19 [20:23:30] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.08, 7.11, 6.79 [20:28:13] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1108420491 secs [20:31:10] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.12, 7.40, 6.92 [20:31:25] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.13, 7.51, 7.06 [20:33:24] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.16, 7.19, 6.99 [20:34:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.18, 3.63, 2.92 [20:35:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.08, 7.56, 7.09 [20:36:13] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.09747755527 secs [20:36:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.03, 2.97, 2.77 [20:37:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.00, 7.39, 7.08 [20:39:20] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.48, 7.93, 7.30 [20:43:17] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.81, 7.20, 7.16 [20:45:03] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 3.81, 5.73, 6.54 [20:57:09] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.64, 6.10, 6.67 [21:00:52] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.82, 7.15, 6.72 [21:01:05] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.07, 7.43, 7.07 [21:02:51] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.00, 7.11, 6.76 [21:03:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.68, 7.00, 6.95 [21:04:50] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.17, 7.52, 6.96 [21:05:03] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.37, 6.51, 6.78 [21:05:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.76, 3.79, 2.85 [21:06:49] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.91, 7.18, 6.89 [21:10:47] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.91, 6.70, 6.78 [21:12:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.52, 3.33, 3.30 [21:16:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/3e89a2bd8934 [21:16:57] [02miraheze/mw-config] 07paladox 033e89a2b - Switch on read-only [21:16:59] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [21:17:02] [02mw-config] 07paladox opened pull request 03#5274: Switch on read-only - 13https://github.com/miraheze/mw-config/pull/5274 [21:18:04] miraheze/mw-config - paladox the build passed. [21:18:42] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.70, 6.83, 6.82 [21:20:41] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.82, 6.47, 6.68 [21:36:33] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.44, 6.73, 6.49 [21:36:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.16, 4.66, 3.32 [21:36:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.16, 4.66, 3.32 [21:38:32] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.33, 6.62, 6.48 [21:40:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.21, 6.81, 6.43 [21:40:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.19, 3.39, 3.12 [21:42:29] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.80, 7.24, 6.77 [21:42:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.97, 6.90, 6.52 [21:44:28] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.83, 6.68, 6.62 [21:44:41] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.76, 6.52, 6.43 [21:48:24] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.89, 7.28, 6.90 [21:48:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.03, 7.71, 6.94 [21:50:08] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.06, 3.55, 2.81 [21:50:23] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.77, 6.70, 6.74 [21:52:04] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.45, 2.84, 2.64 [21:54:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.95, 7.28, 7.01 [22:00:23] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/606e7dc03291...bfe593452d50 [22:00:24] [02miraheze/mw-config] 07paladox 03bfe5934 - Switch on read-only (#5274) [22:00:27] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [22:00:30] [02mw-config] 07paladox closed pull request 03#5274: Switch on read-only - 13https://github.com/miraheze/mw-config/pull/5274 [22:00:31] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [22:00:34] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [22:00:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:00:44] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [22:01:01] !log doing maintenance on cloud14 [22:01:06] that was a fast deploy [22:01:30] miraheze/mw-config - paladox the build passed. [22:02:41] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.83, 6.45, 6.72 [22:03:26] PROBLEM - db142 APT on db142 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [22:05:01] PROBLEM - db142 MariaDB Connections on db142 is CRITICAL: PHP Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Connection failed: Connection refused [22:05:42] PROBLEM - db142 MariaDB on db142 is CRITICAL: Can't connect to MySQL server on 'db142.miraheze.org' (115) [22:05:49] PROBLEM - db101 APT on db101 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [22:06:11] PROBLEM - db101 MariaDB Connections on db101 is CRITICAL: PHP Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Connection failed: Connection refused [22:06:13] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.2171367407 secs [22:06:28] PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 3.92, 2.70, 1.88 [22:06:45] PROBLEM - db121 APT on db121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [22:06:45] PROBLEM - db131 APT on db131 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [22:07:22] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.04, 2.71, 1.40 [22:07:50] PROBLEM - db101 APT on db101 is CRITICAL: APT CRITICAL: 53 packages available for upgrade (5 critical updates). [22:08:11] RECOVERY - db101 MariaDB Connections on db101 is OK: OK connection usage: 3%Current connections: 15 [22:08:26] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: CRITICAL - load average: 4.95, 3.58, 2.31 [22:08:46] PROBLEM - db131 APT on db131 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (5 critical updates). [22:08:46] PROBLEM - db121 APT on db121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (5 critical updates). [22:08:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.26, 3.60, 2.91 [22:10:26] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 3.17, 3.32, 2.36 [22:14:14] PROBLEM - db142 Puppet on db142 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[mariadb-server-10.5],Package[mariadb-backup] [22:16:13] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.008558869362 secs [22:16:26] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: CRITICAL - load average: 5.78, 4.32, 3.04 [22:18:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.79, 3.28, 3.81 [22:19:22] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.41, 3.87, 3.32 [22:19:35] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.94, 6.62, 6.35 [22:21:22] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.70, 3.12, 3.11 [22:21:33] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.76, 6.32, 6.28 [22:22:26] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 1.79, 3.39, 3.18 [22:23:28] PROBLEM - db142 APT on db142 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (5 critical updates). [22:24:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.15, 7.23, 7.83 [22:25:01] RECOVERY - db142 MariaDB Connections on db142 is OK: OK connection usage: 9.2%Current connections: 46 [22:25:22] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.11, 3.92, 3.46 [22:42:54] PROBLEM - reports121 Puppet on reports121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:42:54] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:42:56] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:42:57] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [22:43:01] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [23:03:06] PROBLEM - prometheus131 Puppet on prometheus131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:07] PROBLEM - mw143 Puppet on mw143 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:07] PROBLEM - mon141 Puppet on mon141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:07] PROBLEM - os131 Puppet on os131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:08] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 51% [23:03:08] PROBLEM - bast141 Puppet on bast141 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 20 minutes ago with 7 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wildcard.miraheze.org-2020-2.crt],File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ferm/conf.d/02_main],File[/etc/sudoers] [23:03:08] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:09] PROBLEM - swiftobject121 Puppet on swiftobject121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:11] PROBLEM - swiftobject112 Puppet on swiftobject112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:11] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:15] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Puppet has 41 failures. Last run 19 minutes ago with 41 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.d],File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [23:03:19] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.305 second response time [23:03:20] PROBLEM - swiftobject113 Puppet on swiftobject113 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:22] RECOVERY - mw143 MediaWiki Rendering on mw143 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.040 second response time [23:03:26] PROBLEM - cp22 Puppet on cp22 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:27] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:31] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 50% [23:03:32] PROBLEM - mem131 Puppet on mem131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:38] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:40] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:58] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:58] PROBLEM - mon141 Current Load on mon141 is WARNING: WARNING - load average: 3.17, 3.97, 1.78 [23:03:58] PROBLEM - db121 Puppet on db121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:59] PROBLEM - swiftac111 Puppet on swiftac111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:04:01] PROBLEM - cp33 Puppet on cp33 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:04:05] PROBLEM - swiftobject122 Puppet on swiftobject122 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:04:07] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 1% [23:04:20] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:04:21] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:05:02] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:05:04] RECOVERY - bast141 Puppet on bast141 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [23:05:04] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:05:06] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 6% [23:05:18] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 2% [23:05:22] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 15 backends are healthy [23:05:24] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:05:28] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:05:29] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 6% [23:05:38] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 15 backends are healthy [23:05:50] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 15 backends are healthy [23:05:52] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 0.87, 2.85, 1.62 [23:06:01] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bfe593452d50...f1dff5b121d0 [23:06:02] [02miraheze/mw-config] 07paladox 03f1dff5b - Revert "Switch on read-only (#5274)" [23:06:15] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [23:06:17] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 15 backends are healthy [23:06:45] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 30s [23:06:58] miraheze/mw-config - paladox the build passed. [23:07:00] RECOVERY - swiftproxy131 Puppet on swiftproxy131 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:07:02] RECOVERY - matomo121 Puppet on matomo121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:07:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:07:29] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [23:07:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:07:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:07:37] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [23:07:43] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 7.18, 3.79, 1.62 [23:07:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:07:52] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:08:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.06, 7.19, 5.87 [23:09:09] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: CRITICAL - load average: 4.35, 3.43, 2.65 [23:09:14] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:09:18] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:09:22] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:09:34] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:09:38] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.78, 4.78, 3.10 [23:09:49] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:10:12] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [23:10:15] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:10:55] RECOVERY - puppet141 Puppet on puppet141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:10:58] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:11:16] RECOVERY - db142 Puppet on db142 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [23:11:33] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:13:14] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:13:16] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:13:26] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:13:31] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:13:40] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 0.64, 2.96, 2.13 [23:14:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.50, 6.76, 6.09 [23:14:59] RECOVERY - os131 Puppet on os131 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:14:59] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 1.73, 3.29, 2.98 [23:15:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f1dff5b121d0...b993f0e9a4ae [23:15:15] RECOVERY - cp22 Puppet on cp22 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:15:17] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [23:15:17] [02miraheze/mw-config] 07paladox 03b993f0e - Remove db142 from being a parsercacher [23:15:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:15:22] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:15:25] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [23:15:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:16:15] miraheze/mw-config - paladox the build passed. [23:16:34] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.25, 5.99, 4.96 [23:17:10] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:17:37] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.14, 6.41, 5.66 [23:18:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.02, 8.24, 6.88 [23:18:31] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 4.88, 5.55, 4.93 [23:19:24] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:19:33] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.34, 6.39, 5.74 [23:19:47] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:20:43] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:21:13] RECOVERY - swiftobject121 Puppet on swiftobject121 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:21:17] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:21:28] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:22:03] RECOVERY - swiftobject122 Puppet on swiftobject122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:22:12] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:22:50] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:23:09] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:23:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 7.91, 7.22 [23:24:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cc64e0ab1ac2...fbfe9cd31bdb [23:24:55] [02miraheze/puppet] 07paladox 03fbfe9cd - db131: increase innodb buffer pool to 8g [23:24:57] RECOVERY - prometheus131 Puppet on prometheus131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:25:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fbfe9cd31bdb...cd3f17afed08 [23:25:14] [02miraheze/puppet] 07paladox 03cd3f17a - db142: increase innodb buffer pool to 8g [23:25:32] RECOVERY - mem131 Puppet on mem131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:26:53] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:26:58] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:27:08] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:50] !log [@test131] starting deploy of {'config': True} to all [23:27:51] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [23:27:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:50] RECOVERY - cp23 Puppet on cp23 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:29:20] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:29:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.96, 8.53, 7.60 [23:30:13] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/102964f80505 [23:30:14] [02miraheze/mw-config] 07paladox 03102964f - Remove site notice for maintenance [23:30:16] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [23:30:19] [02mw-config] 07paladox opened pull request 03#5275: Remove site notice for maintenance - 13https://github.com/miraheze/mw-config/pull/5275 [23:30:25] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b993f0e9a4ae...fc60179637c2 [23:30:28] [02miraheze/mw-config] 07paladox 03fc60179 - Remove site notice for maintenance (#5275) [23:30:30] [02mw-config] 07paladox closed pull request 03#5275: Remove site notice for maintenance - 13https://github.com/miraheze/mw-config/pull/5275 [23:30:31] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [23:30:33] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [23:30:33] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [23:30:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:41] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [23:30:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:00] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:31:20] miraheze/mw-config - paladox the build passed. [23:31:24] miraheze/mw-config - paladox the build passed. [23:40:22] RECOVERY - wiki.dunmoore.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dunmoore.net' will expire on Mon 24 Jul 2023 15:43:21 GMT +0000. [23:42:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.71, 7.58, 6.66 [23:42:23] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [23:44:00] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.47, 7.22, 6.64 [23:45:57] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.36, 6.58, 6.48 [23:48:17] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.99, 3.06, 3.85 [23:49:53] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.38, 7.42, 6.83 [23:52:07] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.58, 2.28, 3.36 [23:53:47] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.72, 7.21, 6.91 [23:55:44] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.30, 6.57, 6.71 [23:57:39] !log [@test131] starting deploy of {'config': True} to all [23:57:40] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [23:57:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:57:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:59:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [23:59:40] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.64, 7.24, 6.96