[00:00:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.62, 8.03, 7.30 [00:02:34] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.59, 4.55, 3.31 [00:02:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.23, 6.91, 6.98 [00:04:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.37, 5.84, 4.21 [00:06:23] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.40, 3.16, 3.03 [00:06:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.34, 8.48, 7.54 [00:07:11] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15088 MB (11% inode=99%); [00:08:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.16, 7.83, 7.41 [00:10:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.48, 8.20, 7.61 [00:12:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.61, 7.91, 7.57 [00:14:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.61, 8.15, 7.68 [00:16:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.61, 7.94, 7.69 [00:24:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.45, 8.22, 7.75 [00:25:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.42, 3.82, 3.31 [00:26:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.78, 6.58, 4.10 [00:27:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.90, 2.81, 3.90 [00:28:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.66, 7.35, 4.69 [00:28:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.27, 7.31, 7.55 [00:29:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.22, 2.83, 3.02 [00:30:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.82, 6.65, 4.75 [00:30:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.49, 7.54, 7.56 [00:32:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.91, 7.52, 7.54 [00:36:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.24, 6.11, 4.99 [00:37:23] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.02, 3.47, 3.65 [00:38:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.69, 5.79, 5.03 [00:39:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.88, 3.00, 3.46 [00:41:18] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.20, 2.34, 3.16 [00:42:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.71, 6.83, 5.60 [00:42:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.81, 5.98, 6.73 [00:44:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.94, 7.73, 6.10 [00:46:08] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 4.00, 3.30, 2.69 [00:46:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.62, 7.72, 7.35 [00:50:07] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.09, 3.63, 2.96 [00:50:18] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.15, 3.65, 2.99 [00:50:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.05, 8.03, 7.55 [00:52:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.22, 2.98, 2.82 [00:52:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.16, 7.25, 7.31 [00:54:07] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.01, 3.52, 3.08 [00:54:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.44, 7.36 [00:56:07] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.75, 3.34, 3.08 [00:56:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.54, 7.91, 7.49 [00:56:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.79, 7.42, 7.35 [01:02:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.79, 6.19, 6.78 [01:03:16] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.278334558 secs [01:10:08] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.47, 3.24, 3.02 [01:12:08] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 3.39, 3.27, 3.06 [01:12:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.71, 3.15, 2.91 [01:12:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.24, 7.67, 7.40 [01:14:16] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.47, 2.64, 2.77 [01:15:16] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.03929013014 secs [01:16:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.67, 7.96, 7.68 [01:16:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.57, 4.61, 3.80 [01:20:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.30, 7.85, 7.67 [01:20:50] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.78, 3.63, 3.58 [01:22:48] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.14, 3.10, 3.39 [01:23:11] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14595 MB (10% inode=99%); [01:23:20] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.59, 8.75, 7.23 [01:24:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.93, 7.68, 7.63 [01:26:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.70, 8.51, 7.89 [01:27:11] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.13, 6.49, 6.79 [01:30:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.33, 7.72, 7.70 [01:37:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.24, 3.43, 3.34 [01:39:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.69, 2.80, 3.12 [01:42:03] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.90, 3.38, 2.84 [01:42:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.54, 7.67, 7.44 [01:44:03] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.68, 3.60, 2.97 [01:44:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.16, 7.10, 7.26 [01:46:03] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.92, 3.30, 2.94 [01:50:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.18, 7.86, 7.42 [01:52:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.98, 7.50, 7.34 [01:56:04] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.58, 4.02, 3.40 [01:59:59] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.88, 3.36, 3.32 [02:03:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.56, 3.88, 3.59 [02:09:45] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.39, 3.15, 3.37 [02:12:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.49, 7.35, 7.10 [02:13:36] PROBLEM - eoa-wiki.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for eoa-wiki.ga could not be found [02:14:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.66, 7.45, 7.18 [02:19:10] PROBLEM - miraheze.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.ga could not be found [02:22:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.77, 8.36, 7.63 [02:26:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.82, 7.79, 7.59 [02:32:10] PROBLEM - petparents.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for petparents.ga could not be found [02:32:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.52, 3.43, 3.30 [02:34:13] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.70, 3.16, 3.22 [02:46:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.56, 7.52, 7.20 [02:54:34] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.90, 4.64, 3.58 [02:54:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.77, 7.82, 7.53 [02:56:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.62, 3.49, 3.29 [02:58:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.40, 2.92, 3.11 [03:01:36] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query zulaclub.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:01:54] RECOVERY - db131 Backups SQL on db131 is OK: FILE_AGE OK: /var/log/sql-backup.log is 113 seconds old and 275 bytes [03:03:46] PROBLEM - az.wikiretet.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for az.wikiretet.ga could not be found [03:04:09] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.84, 4.30, 3.68 [03:06:04] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.60, 3.54, 3.47 [03:07:59] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.58, 2.87, 3.22 [03:10:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.68, 7.50, 7.24 [03:12:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.20, 6.87, 7.04 [03:14:09] PROBLEM - www.wikiretet.ga - reverse DNS on sslhost is WARNING: NXDOMAIN: The DNS query name does not exist: www.wikiretet.ga. [03:18:34] PROBLEM - tr.wikiretet.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tr.wikiretet.ga could not be found [03:18:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.49, 7.75, 7.32 [03:22:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.60, 7.74, 7.49 [03:23:11] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 14864 MB (11% inode=99%); [03:32:06] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.41, 3.50, 3.13 [03:32:32] RECOVERY - az.wikiretet.ga - reverse DNS on sslhost is OK: SSL OK - az.wikiretet.ga reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:34:04] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.93, 3.27, 3.08 [03:37:26] PROBLEM - wiki.scpsharp.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.scpsharp.ga could not be found [03:37:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.27, 4.67, 3.70 [03:41:53] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.14, 3.80, 3.61 [03:47:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.40, 3.04, 3.36 [03:49:56] PROBLEM - en.wikiretet.ga - reverse DNS on sslhost is WARNING: NXDOMAIN: The DNS query name does not exist: wikiretet.ga. [03:50:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.34, 7.15, 6.94 [03:55:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.63, 11.00, 11.91 [03:58:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.99, 7.45, 7.29 [03:59:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.96, 12.33, 12.23 [04:00:48] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [04:03:46] PROBLEM - az.wikiretet.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for az.wikiretet.ga could not be found [04:04:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.05, 6.16, 6.75 [04:05:11] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14591 MB (10% inode=99%); [04:05:17] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.75, 3.60, 3.39 [04:07:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.07, 3.72, 3.46 [04:08:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.70, 7.44, 7.28 [04:09:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.94, 3.03, 3.23 [04:16:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.13, 6.23, 6.78 [04:20:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.50, 8.34, 7.49 [04:24:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.27, 7.51, 7.38 [04:27:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.95, 11.23, 11.89 [04:29:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.68, 11.48, 11.87 [04:36:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.02, 11.50, 11.95 [04:36:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.79, 7.18, 7.13 [04:38:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.61, 6.44, 6.86 [04:39:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.17, 11.41, 11.86 [04:40:54] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1759274304 secs [04:41:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.23, 11.87, 11.98 [04:42:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.81, 12.12, 11.97 [04:42:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.08, 7.45, 7.15 [04:44:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.29, 11.92, 11.93 [04:44:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.05, 7.20, 7.09 [04:45:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.32, 11.03, 11.62 [04:46:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.23, 11.78, 11.86 [04:48:54] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.08868387341 secs [04:49:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.60, 12.95, 12.20 [04:56:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.31, 7.14 [04:57:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.44, 11.38, 11.76 [04:58:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 7.25, 7.14 [05:00:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.15, 11.53, 11.93 [05:04:43] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.29, 6.06, 6.66 [05:06:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.01, 10.83, 11.38 [05:06:12] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:06:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.13, 3.86, 3.22 [05:08:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.78, 10.39, 11.15 [05:08:54] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.74, 3.30, 3.10 [05:09:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.63, 11.55, 11.41 [05:10:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.91, 11.78, 11.54 [05:19:50] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset -0.1264284253 secs [05:21:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.33, 11.27, 11.76 [05:23:49] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -0.07844650745 secs [05:33:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.62, 11.48, 11.27 [05:35:36] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [05:37:06] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [05:43:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.03, 3.45, 3.14 [05:45:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.48, 2.62, 2.86 [05:48:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.11, 6.55, 6.42 [05:52:43] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.78, 6.59, 6.46 [05:59:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.02, 7.45, 6.90 [06:01:25] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.53, 6.63, 6.66 [06:05:00] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:07:07] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [06:10:55] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.20, 7.21, 6.82 [06:12:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.82, 6.70, 6.67 [06:16:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.87, 6.73, 6.69 [06:19:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.11, 3.48, 2.62 [06:21:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.03, 3.30, 2.65 [06:24:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.19, 6.37, 6.63 [06:29:16] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1993756294 secs [06:33:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:34:23] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [06:35:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.41, 6.87, 6.69 [06:39:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.51, 7.72, 7.06 [06:41:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.58, 3.43, 2.74 [06:43:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.70, 7.95, 7.39 [06:43:16] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.04881426692 secs [06:45:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.53, 3.39, 2.93 [06:48:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.33, 7.77, 7.46 [06:52:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.60, 7.13, 7.32 [06:58:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [07:00:49] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.96, 4.38, 2.38 [07:02:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.17, 5.97, 6.64 [07:02:47] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.91, 3.82, 2.42 [07:04:46] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.75, 3.06, 2.31 [07:07:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.46, 3.20, 2.52 [07:09:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.37, 2.94, 2.52 [07:11:57] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:21:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.92, 7.65, 7.08 [07:33:11] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [07:36:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.17, 6.55, 6.80 [07:40:08] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.94, 2.68, 1.34 [07:40:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.15, 6.70, 6.86 [07:41:50] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [07:42:07] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 3.37, 3.00, 1.63 [07:44:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.28, 6.39, 6.70 [08:05:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.22, 6.75 [08:07:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.07, 6.84, 6.67 [08:09:00] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.68, 6.31, 6.48 [08:12:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.90, 7.23, 6.90 [08:13:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.80, 10.05, 11.77 [08:18:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.16, 6.62, 6.78 [08:22:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.71, 10.79, 11.83 [08:26:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.54, 11.06, 11.64 [08:31:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.85, 11.98, 11.37 [08:33:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.77, 11.81, 11.39 [08:34:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.40, 11.22, 11.83 [08:36:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.96, 6.44, 6.40 [08:37:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.76, 12.08, 11.51 [08:38:03] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.51, 2.95, 2.41 [08:38:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.48, 6.54, 6.45 [08:40:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.28, 11.35, 11.60 [08:40:03] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.15, 2.62, 2.36 [08:41:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.88, 11.77, 11.50 [08:41:34] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:43:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.38, 11.95, 11.57 [08:44:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.92, 11.86, 11.85 [08:45:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.41, 11.38, 11.41 [08:46:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.28, 7.67, 6.96 [08:51:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.27, 11.49, 11.42 [08:52:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.39, 12.62, 12.13 [08:55:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.56, 11.81, 11.62 [08:56:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.43, 6.44, 6.72 [09:03:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.46, 12.05, 11.65 [09:04:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 6.76, 6.73 [09:06:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.05, 6.17, 6.53 [09:07:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.80, 11.94, 11.82 [09:08:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.71, 11.25, 11.86 [09:09:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.36, 12.12, 11.91 [09:10:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.09, 12.08, 12.07 [09:11:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.77, 11.55, 11.74 [09:18:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.51, 7.35, 6.95 [09:22:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.90, 10.32, 11.57 [09:22:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.96, 6.74, 6.80 [09:27:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.43, 8.72, 10.16 [09:33:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.26, 6.82, 6.84 [09:34:45] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:35:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.21, 7.85, 7.20 [09:36:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.00, 8.39, 9.99 [09:37:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.06, 7.15, 7.02 [09:41:07] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.97, 6.50, 6.79 [09:41:18] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [09:48:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.15, 6.95, 6.92 [09:52:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.44, 7.55, 7.16 [09:56:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.91, 9.89, 9.50 [09:56:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.04, 7.17, 7.12 [10:00:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.33, 9.25, 9.41 [10:00:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.29, 7.88, 7.41 [10:02:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.43, 7.64, 7.38 [10:04:08] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [10:10:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.86, 11.47, 10.27 [10:14:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.27, 11.78, 10.72 [10:15:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.18, 10.56, 9.90 [10:16:36] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=periloustimeswiki --no-updates --username-prefix=wikia:moon-guard /home/reception/ImportDump/periloustimeswiki-20230530215523.xml (START) [10:16:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:16:55] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=brukerforumwiki --no-updates --username-prefix=nowikipedia /home/reception/ImportDump/brukerforumwiki-20230604101310.xml (START) [10:16:56] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=brukerforumwiki --no-updates --username-prefix=nowikipedia /home/reception/ImportDump/brukerforumwiki-20230604101310.xml (END - exit=256) [10:16:57] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=brukerforumwiki (START) [10:17:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:17:03] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=brukerforumwiki (END - exit=0) [10:17:04] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=brukerforumwiki --active --update (END - exit=0) [10:17:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:17:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:17:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:17:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.96, 10.01, 9.77 [10:17:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:17:25] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=periloustimeswiki --no-updates --username-prefix=wikia:moon-guard /home/reception/ImportDump/periloustimeswiki-20230530215523.xml (END - exit=0) [10:17:26] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=periloustimeswiki (START) [10:17:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:17:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:18:31] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.49, 9.87, 10.12 [10:20:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.02, 8.87, 7.92 [10:21:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/00b50a92f67d...91d198c751e1 [10:21:10] [02miraheze/ssl] 07MirahezeSSLBot 0391d198c - Bot: Add SSL cert for wiki.sofurry.games [10:21:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.11, 10.60, 10.09 [10:22:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.75, 10.77, 10.52 [10:23:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.03, 5.74, 3.15 [10:24:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.55, 7.47, 7.59 [10:25:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.06, 9.23, 9.70 [10:29:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.28, 10.57, 10.16 [10:31:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.07, 11.03, 10.37 [10:31:25] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=periloustimeswiki (END - exit=0) [10:31:26] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=periloustimeswiki --active --update (END - exit=0) [10:31:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:31:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:32:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 7.37, 7.41 [10:33:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.58, 11.13, 10.48 [10:35:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.12, 12.68, 11.14 [10:35:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 1.48, 3.40, 3.83 [10:35:54] PROBLEM - db142 Backups SQL on db142 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209665 seconds old and 1753 bytes [10:36:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.44, 11.05, 10.65 [10:36:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.46, 6.94, 7.27 [10:39:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.35, 2.61, 3.38 [10:41:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.79, 11.40, 11.11 [10:42:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.80, 11.95, 11.21 [10:44:08] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.54, 3.02, 1.51 [10:44:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.01, 5.57, 2.98 [10:44:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.64, 6.99, 7.12 [10:46:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.98, 12.03, 11.36 [10:46:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.23, 5.58, 3.32 [10:48:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.38, 11.00, 11.06 [10:48:07] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 0.79, 2.54, 1.74 [10:48:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.75, 7.22, 7.20 [10:49:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.76, 11.26, 11.05 [10:50:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.23, 13.13, 11.81 [10:50:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.13, 8.43, 7.67 [10:51:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.68, 12.00, 11.39 [10:52:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.54, 7.86, 7.57 [10:53:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.37, 12.32, 11.59 [10:56:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.29, 8.11, 7.72 [10:57:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.45, 11.85, 11.57 [10:57:28] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.37, 3.06, 2.21 [10:57:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.16, 11.89, 11.77 [10:59:28] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.94, 3.10, 2.34 [11:02:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.99, 7.73, 7.73 [11:05:31] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.44, 8.56, 10.20 [11:11:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.58, 10.37, 10.45 [11:12:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.24, 7.72, 7.63 [11:15:07] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.15, 9.39, 10.02 [11:18:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.28, 7.67, 7.67 [11:20:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 12.23, 8.86, 8.07 [11:27:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.77, 11.40, 10.88 [11:29:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.50, 11.04, 10.42 [11:29:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.68, 11.04, 10.81 [11:35:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.65, 12.01, 11.14 [11:37:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.33, 11.93, 10.92 [11:37:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.62, 11.42, 11.04 [11:39:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.94, 12.10, 11.34 [11:41:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.71, 12.00, 11.43 [11:43:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.82, 12.46, 11.65 [11:53:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.34, 7.13, 7.75 [12:05:25] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [12:05:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.63, 7.54, 7.71 [12:07:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.80, 7.54, 7.69 [12:08:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.52, 6.63, 3.49 [12:10:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.88, 6.22, 3.74 [12:11:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.12, 7.46, 7.60 [12:15:04] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:29:07] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.99, 3.43, 2.65 [12:31:06] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.08, 2.55, 2.42 [12:34:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [12:44:08] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.78, 3.58, 2.23 [12:44:23] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.86, 2.64, 1.83 [12:44:33] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 13.69, 10.59, 7.16 [12:44:40] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [12:45:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.54, 3.05, 2.33 [12:45:19] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 14.89, 9.42, 5.79 [12:45:39] PROBLEM - mail121 Current Load on mail121 is WARNING: WARNING - load average: 1.83, 1.30, 0.70 [12:45:45] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 19.03, 13.36, 7.71 [12:45:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 45.00, 28.09, 14.88 [12:46:23] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.05, 2.46, 1.87 [12:47:37] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 0.52, 0.97, 0.65 [12:48:40] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:43] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.33, 3.83, 2.76 [12:50:08] !log [paladox@puppet141] "salt-ssh -E cp.* cmd.run varnishadm "ban req.http.Host == static.miraheze.org && req.url ~ "^/vanataswiki/thumb/6/6d/SouthGladomyrMap.png/300px-SouthGladomyrMap.png""" [12:50:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:50:38] paladox: help pls on discord [12:50:46] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 6.560 second response time [12:50:46] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 6.546 second response time [12:51:15] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.53, 3.29, 2.70 [12:58:02] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 10.01, 7.99, 4.86 [12:59:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:00:08] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.99, 3.42, 3.68 [13:04:08] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.49, 2.71, 3.35 [13:06:01] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.95, 3.04, 3.73 [13:06:51] !log lower swiftobject122 weight to 6000 (from 8000) [13:06:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:07:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.87, 3.64, 3.02 [13:08:00] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.14, 2.38, 3.40 [13:09:15] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.05, 3.14, 2.92 [13:14:08] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.04, 4.08, 3.46 [13:16:08] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.94, 3.27, 3.24 [13:16:15] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:23:57] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.56, 2.83, 2.61 [13:24:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:25:08] PROBLEM - wiki.closai.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.closai.com' expires in 15 day(s) (Tue 20 Jun 2023 13:16:17 GMT +0000). [13:25:55] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.04, 2.48, 2.51 [13:26:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/91d198c751e1...31f5e46080cf [13:26:35] [02miraheze/ssl] 07MirahezeSSLBot 0331f5e46 - Bot: Update SSL cert for wiki.closai.com [13:27:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.46, 3.83, 3.18 [13:29:01] PROBLEM - swiftobject122 PowerDNS Recursor on swiftobject122 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:29:55] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.83, 3.12, 3.00 [13:30:32] PROBLEM - swiftobject122 SSH on swiftobject122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:11] PROBLEM - swiftobject122 Puppet on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:33:03] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:33:13] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.014 second response time [13:35:19] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 5.01, 6.30, 7.69 [13:37:44] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.58, 6.36, 7.83 [13:39:01] RECOVERY - swiftobject122 SSH on swiftobject122 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:41:39] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.015 second response time [13:43:19] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.44, 5.73, 6.80 [13:43:26] PROBLEM - swiftobject122 SSH on swiftobject122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:44] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.78, 6.94, 7.47 [13:45:44] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.17, 6.19, 7.13 [13:47:30] RECOVERY - swiftobject122 SSH on swiftobject122 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:47:44] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.41, 7.64, 7.58 [13:48:08]