[00:01:35] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.91, 3.77, 3.83 [00:03:05] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.27, 9.17, 5.99 [00:03:27] RECOVERY - cp24 Disk Space on cp24 is OK: DISK OK - free space: / 19349MiB (25% inode=98%); [00:03:37] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 12.12, 6.98, 3.23 [00:03:59] RECOVERY - cp25 Disk Space on cp25 is OK: DISK OK - free space: / 19612MiB (25% inode=98%); [00:05:02] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 6.70, 8.07, 5.96 [00:05:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 52% [00:07:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 48% [00:07:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [00:09:17] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.37, 3.90, 2.38 [00:09:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [00:10:10] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 10.07, 6.59, 4.47 [00:10:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 56% [00:12:19] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 38.89, 18.22, 9.79 [00:12:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 46% [00:13:21] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [00:13:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 56% [00:14:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 90% [00:14:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [00:15:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [00:15:37] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.72, 6.44, 6.19 [00:18:01] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 1.70, 7.56, 7.73 [00:18:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 49% [00:20:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 57% [00:21:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 47% [00:22:00] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 1.37, 4.10, 6.25 [00:22:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [00:22:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [00:23:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [00:26:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 58% [00:27:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 59% [00:28:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [00:29:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 51% [00:31:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.64, 3.40, 2.91 [00:33:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [00:33:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [00:33:45] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.03, 2.84, 2.77 [00:34:39] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.fediverse.kr and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:34:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 40% [00:36:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [00:38:37] RECOVERY - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is OK: OK - NGINX Error Rate is 33% [00:38:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 54% [00:39:00] PROBLEM - db131 Disk Space on db131 is CRITICAL: DISK CRITICAL - free space: / 14280 MB (5% inode=96%); [00:40:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [00:41:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 58% [00:41:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 59% [00:42:01] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 50.34, 26.11, 12.58 [00:42:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [00:43:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [00:43:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [00:44:10] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 4.30, 6.25, 7.61 [00:44:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 59% [00:46:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [00:48:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 54% [00:49:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 56% [00:50:10] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 4.36, 5.22, 6.74 [00:50:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 81% [00:51:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 87% [00:54:00] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 1.00, 4.14, 7.67 [00:58:00] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 1.17, 2.50, 6.18 [00:58:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 55% [00:58:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 57% [00:59:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 59% [00:59:39] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5012994707 secs [01:00:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [01:01:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [01:04:10] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 13.05, 7.88, 6.75 [01:04:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [01:04:39] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fediverse.kr' expires in 14 day(s) (Mon 04 Dec 2023 20:50:46 GMT +0000). [01:05:49] PROBLEM - graylog131 PowerDNS Recursor on graylog131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:06:31] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 4 backends are down. mw131 mw132 mw141 mw143 [01:07:06] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.30, 3.67, 3.25 [01:07:07] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw131 mw134 [01:07:09] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.25, 8.24, 6.00 [01:07:16] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw131 mw134 [01:07:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 59% [01:08:10] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 13.41, 11.30, 7.90 [01:08:26] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [01:09:02] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.82, 3.31, 3.16 [01:09:07] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [01:09:07] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 3.89, 6.43, 5.61 [01:09:16] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [01:12:09] RECOVERY - graylog131 PowerDNS Recursor on graylog131 is OK: DNS OK: 0.326 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:12:21] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw134 [01:12:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 58% [01:12:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 47% [01:13:01] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.32, 3.84, 3.40 [01:13:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 50% [01:14:06] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 4.54, 10.20, 8.89 [01:14:10] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 5.05, 7.21, 7.33 [01:14:13] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 31.67, 16.70, 9.04 [01:14:21] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [01:14:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [01:14:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [01:14:57] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.87, 3.89, 3.47 [01:15:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [01:15:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [01:15:48] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:16:57] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.40, 4.12, 3.61 [01:18:53] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.59, 3.57, 3.47 [01:19:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 53% [01:20:10] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 10.18, 8.48, 7.77 [01:20:13] PROBLEM - swiftproxy111 ferm_active on swiftproxy111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:20:29] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [01:20:50] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.02, 3.12, 3.32 [01:20:51] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [01:20:57] PROBLEM - swiftproxy111 conntrack_table_size on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [01:21:18] PROBLEM - swiftproxy111 memcached on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 and port 11211: No route to host [01:21:25] PROBLEM - swiftproxy111 SSH on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 and port 22: No route to host [01:21:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 55% [01:21:51] PROBLEM - cloud11 NTP time on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host PROBLEM - Host cloud11 is DOWN: CRITICAL - Host Unreachable (2a10:6740::6:200) [01:22:07] PROBLEM - swiftproxy111 Current Load on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [01:22:08] PROBLEM - swiftproxy111 Disk Space on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [01:22:26] PROBLEM - swiftproxy111 NTP time on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [01:22:26] PROBLEM - Host swiftproxy111 is DOWN: CRITICAL - Host Unreachable (2a10:6740::6:201) [01:22:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 51% [01:23:00] PROBLEM - db131 Disk Space on db131 is WARNING: DISK WARNING - free space: / 14996 MB (6% inode=96%); [01:23:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [01:23:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [01:23:54] PROBLEM - graylog131 PowerDNS Recursor on graylog131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:23:56] PROBLEM - os131 PowerDNS Recursor on os131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:24:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 97% [01:26:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 52% [01:27:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 48% [01:28:05] RECOVERY - graylog131 PowerDNS Recursor on graylog131 is OK: DNS OK: 1.211 second response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:28:05] RECOVERY - os131 PowerDNS Recursor on os131 is OK: DNS OK: 1.452 second response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:28:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [01:29:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [01:30:00] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 1.10, 3.52, 7.79 [01:34:00] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 1.88, 2.46, 6.36 [01:35:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 58% [01:37:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [01:39:28] RECOVERY - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is OK: OK - NGINX Error Rate is 23% [01:39:51] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 11.15, 10.25, 9.36 [01:40:20] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.83, 3.24, 3.05 [01:41:49] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 6.53, 9.01, 9.02 [01:42:05] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 35.80, 17.71, 10.59 [01:43:21] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [01:43:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [01:44:12] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.86, 3.09, 3.04 [01:50:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 59% [01:52:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [01:54:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 52% [01:56:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [01:56:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 52% [01:57:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 51% [01:58:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 83% [01:59:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [01:59:57] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.91, 3.36, 3.08 [02:00:13] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 5.44, 6.66, 7.85 [02:00:36] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:01:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 59% [02:01:53] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.59, 3.00, 2.98 [02:02:00] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 1.26, 2.45, 7.26 [02:03:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [02:04:00] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 1.15, 2.01, 6.51 [02:04:39] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.fediverse.kr and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:07:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 49% [02:08:37] RECOVERY - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is OK: OK - NGINX Error Rate is 34% [02:08:43] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.62, 3.73, 3.30 [02:09:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [02:10:04] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 49.70, 23.08, 13.07 [02:10:11] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 4.97, 5.25, 6.60 [02:10:37] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:10:42] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.26, 3.48, 3.26 [02:12:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 55% [02:12:41] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.89, 4.04, 3.49 [02:13:21] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [02:13:48] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [02:14:12] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 6.53, 6.47, 6.81 [02:14:38] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.60, 3.54, 3.38 [02:16:10] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 5.42, 6.13, 6.65 [02:16:34] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.49, 3.30, 3.31 [02:16:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [02:16:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 56% [02:17:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 58% [02:18:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [02:19:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [02:19:42] I'll help the first 20 people interested in how to earn $50k within 72hours and hours but you will reimburse me 10% of your dividend when you collect it. Note: only interested people should send a friend request or send me a dm!👇 ask me HOW ! [02:20:10] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 9.22, 7.80, 7.18 [02:26:00] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 1.22, 2.89, 7.34 [02:26:11] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 6.41, 7.49, 7.40 [02:27:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 57% [02:28:00] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 1.13, 2.29, 6.57 [02:29:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 58% [02:29:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [02:31:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [02:32:10] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 3.04, 5.17, 6.44 [02:34:39] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fediverse.kr' expires in 14 day(s) (Mon 04 Dec 2023 20:50:46 GMT +0000). [02:34:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 44% [02:34:48] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 1.37, 2.62, 3.83 [02:35:00] PROBLEM - db131 Disk Space on db131 is CRITICAL: DISK CRITICAL - free space: / 14267 MB (5% inode=96%); [02:36:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [02:37:00] RECOVERY - db131 Disk Space on db131 is OK: DISK OK - free space: / 41896 MB (17% inode=96%); [02:38:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 41% [02:38:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 56% [02:38:48] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.59, 1.89, 3.23 [02:39:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 49% [02:40:01] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 45.93, 20.12, 10.73 [02:42:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [02:42:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [02:43:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [02:45:48] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:48:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 58% [02:49:52] RECOVERY - Host cloud11 is UP: PING OK - Packet loss = 0%, RTA = 0.85 ms [02:49:55] PROBLEM - cloud11 conntrack_table_size on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: Connection refusedconnect to host 2a10:6740::6:200 port 5666: Connection refused [02:50:00] PROBLEM - cloud11 Disk Space on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: Connection refusedconnect to host 2a10:6740::6:200 port 5666: Connection refused [02:50:30] PROBLEM - cloud11 ferm_active on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: Connection refusedconnect to host 2a10:6740::6:200 port 5666: Connection refused [02:50:35] PROBLEM - cloud11 PowerDNS Recursor on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: Connection refusedconnect to host 2a10:6740::6:200 port 5666: Connection refused [02:50:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 80% [02:50:40] PROBLEM - cloud11 SSH on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 and port 22: Connection refused [02:50:55] PROBLEM - cloud11 Current Load on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: Connection refusedconnect to host 2a10:6740::6:200 port 5666: Connection refused [02:52:46] RECOVERY - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is OK: OK - NGINX Error Rate is 38% [02:54:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 45% [02:55:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 47% [02:56:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [02:56:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [02:57:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [02:59:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 56% [03:00:00] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 1.02, 2.35, 7.35 [03:01:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [03:01:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 58% [03:02:00] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 1.00, 1.90, 6.57 [03:03:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [03:04:35] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.72, 3.23, 3.93 [03:04:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 54% [03:06:34] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.43, 3.61, 3.98 [03:06:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [03:07:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 59% [03:08:32] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.36, 3.62, 3.95 [03:09:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [03:10:00] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 49.18, 21.78, 12.14 [03:10:31] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.33, 3.84, 3.98 [03:12:30] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.26, 3.72, 3.93 [03:14:37] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [03:15:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 57% [03:17:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 87% [03:20:17] PROBLEM - ping6 on swiftobject101 is CRITICAL: CRITICAL - Host Unreachable (2a10:6740::6:103) [03:20:26] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.86, 3.96, 3.88 [03:20:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 53% [03:20:44] PROBLEM - swiftobject101 Puppet on swiftobject101 is CRITICAL: connect to address 2a10:6740::6:103 port 5666: No route to hostconnect to host 2a10:6740::6:103 port 5666: No route to host [03:20:59] PROBLEM - swiftobject101 conntrack_table_size on swiftobject101 is CRITICAL: connect to address 2a10:6740::6:103 port 5666: No route to hostconnect to host 2a10:6740::6:103 port 5666: No route to host [03:21:03] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: connect to address 2a10:6740::6:103 port 5666: No route to hostconnect to host 2a10:6740::6:103 port 5666: No route to host [03:21:19] PROBLEM - Host swiftobject101 is DOWN: CRITICAL - Host Unreachable (2a10:6740::6:103) [03:21:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 55% [03:22:25] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.57, 3.72, 3.79 [03:22:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [03:23:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [03:24:00] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 1.08, 3.28, 7.48 [03:26:00] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 1.03, 2.53, 6.69 [03:26:26] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [03:29:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 54% [03:31:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 89% [03:32:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 59% [03:34:19] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.57, 3.75, 3.66 [03:34:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [03:35:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 53% [03:36:18] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.96, 3.63, 3.61 [03:36:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 59% [03:36:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 51% [03:37:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 56% [03:37:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [03:38:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [03:39:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [03:40:04] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 43.86, 22.95, 11.95 [03:40:18] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.31, 4.36, 3.91 [03:40:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 80% [03:41:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 47% [03:45:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [03:45:48] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:46:16] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.57, 3.97, 3.94 [03:48:01] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/42bed7461c8a [03:48:03] [02miraheze/mw-config] 07paladox 0342bed74 - Disable file uploads temporarily [03:48:05] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [03:48:07] [02mw-config] 07paladox opened pull request 03#5377: Disable file uploads temporarily - 13https://github.com/miraheze/mw-config/pull/5377 [03:48:34] [02mw-config] 07paladox closed pull request 03#5377: Disable file uploads temporarily - 13https://github.com/miraheze/mw-config/pull/5377 [03:48:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d5a04fe4343d...a24b405aa9a0 [03:48:38] [02miraheze/mw-config] 07paladox 03a24b405 - Disable file uploads temporarily (#5377) [03:48:41] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [03:48:42] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [03:48:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 48% [03:48:54] miraheze/mw-config - paladox the build passed. [03:48:56] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [03:49:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:49:06] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [03:49:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:49:20] !log [paladox@mwtask141] starting deploy of {'config': True} to all [03:49:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:49:28] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [03:49:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 57% [03:49:29] miraheze/mw-config - paladox the build passed. [03:49:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:50:37] RECOVERY - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is OK: OK - NGINX Error Rate is 39% [03:51:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [03:52:18] PROBLEM - swiftproxy131 SSH on swiftproxy131 is CRITICAL: connect to address 2a10:6740::6:315 and port 22: No route to host [03:53:03] PROBLEM - swiftproxy131 Swift Proxy on swiftproxy131 is CRITICAL: connect to address 2a10:6740::6:315 and port 80: No route to host [03:53:10] PROBLEM - swiftproxy131 PowerDNS Recursor on swiftproxy131 is CRITICAL: connect to address 2a10:6740::6:315 port 5666: No route to hostconnect to host 2a10:6740::6:315 port 5666: No route to host [03:53:10] PROBLEM - swiftproxy131 NTP time on swiftproxy131 is CRITICAL: connect to address 2a10:6740::6:315 port 5666: No route to hostconnect to host 2a10:6740::6:315 port 5666: No route to host [03:53:20] PROBLEM - Host swiftproxy131 is DOWN: CRITICAL - Host Unreachable (2a10:6740::6:315) [03:54:11] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.22, 2.71, 3.34 [03:54:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 55% [03:54:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [03:55:59] !log [@test131] starting deploy of {'config': True} to all [03:56:00] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [03:56:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:56:07] PROBLEM - cp35 Current Load on cp35 is WARNING: LOAD WARNING - total load average: 5.87, 6.87, 3.50 [03:56:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:56:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [03:58:10] RECOVERY - cp35 Current Load on cp35 is OK: LOAD OK - total load average: 4.52, 6.07, 3.62 [03:58:25] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:59:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 57% [03:59:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 46% [04:01:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [04:01:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [04:01:51] PROBLEM - Host cloud11 is DOWN: CRITICAL - Host Unreachable (2a10:6740::6:200) [04:04:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 54% [04:05:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 54% [04:06:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 80% [04:07:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [04:10:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 48% [04:12:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 48% [04:12:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [04:13:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 56% [04:14:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [04:14:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 43% [04:15:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [04:15:38] PROBLEM - cp35 Current Load on cp35 is WARNING: LOAD WARNING - total load average: 1.97, 6.87, 5.60 [04:16:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [04:17:33] RECOVERY - cp35 Current Load on cp35 is OK: LOAD OK - total load average: 0.73, 4.82, 4.99 [04:18:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 47% [04:19:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 56% [04:20:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [04:20:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 55% [04:21:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [04:22:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 87% [04:25:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 48% [04:26:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 55% [04:34:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [04:35:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [04:42:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 44% [04:43:21] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [04:44:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [04:46:23] PROBLEM - cp35 Current Load on cp35 is CRITICAL: LOAD CRITICAL - total load average: 16.37, 11.89, 6.81 [04:46:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 55% [04:47:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 49% [04:49:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [04:50:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 50% [04:50:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [04:52:09] PROBLEM - cp35 Current Load on cp35 is WARNING: LOAD WARNING - total load average: 7.35, 7.92, 6.60 [04:52:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [04:54:07] RECOVERY - cp35 Current Load on cp35 is OK: LOAD OK - total load average: 3.88, 6.49, 6.24 [04:54:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 59% [04:56:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 55% [04:56:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [04:58:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [04:59:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 57% [05:00:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 45% [05:01:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [05:02:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 54% [05:02:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [05:04:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [05:04:39] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.fediverse.kr and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:06:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 58% [05:08:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 59% [05:08:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [05:10:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [05:10:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 53% [05:11:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 49% [05:11:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 55% [05:12:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [05:13:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [05:13:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [05:13:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [05:13:49] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [05:15:29] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 59% [05:16:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 59% [05:17:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 54% [05:18:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [05:19:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [05:19:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [05:21:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 59% [05:22:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 44% [05:23:29] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 49% [05:24:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [05:25:29] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [05:26:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 48% [05:27:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [05:31:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 55% [05:32:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [05:32:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 54% [05:34:39] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fediverse.kr' expires in 14 day(s) (Mon 04 Dec 2023 20:50:46 GMT +0000). [05:34:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [05:35:32] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [05:38:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 53% [05:38:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 49% [05:40:37] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [05:40:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 83% [05:42:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 57% [05:46:46] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [05:52:37]