[00:00:28] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.46, 3.58, 3.12 [00:00:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.39, 8.60, 8.32 [00:01:44] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.02, 3.12, 2.85 [00:02:23] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.65, 3.62, 3.19 [00:03:42] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.38, 3.24, 2.94 [00:04:29] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [00:04:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.24, 7.38, 7.89 [00:06:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.70, 3.23, 3.13 [00:09:35] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1306689978 secs [00:12:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.55, 7.08, 7.26 [00:14:27] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 7.26, 3.79, 2.24 [00:14:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.50, 7.07, 7.23 [00:16:24] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.74, 3.27, 2.24 [00:17:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [00:18:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.14, 6.96, 7.10 [00:19:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.95, 3.28, 3.03 [00:20:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.86, 7.43, 7.26 [00:21:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.09, 3.46, 3.11 [00:23:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.08, 2.86, 2.93 [00:26:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.36, 5.35, 3.54 [00:26:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.88, 6.29, 6.77 [00:30:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.94, 6.25, 4.39 [00:32:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.46, 3.39, 3.10 [00:34:09] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.70, 3.46, 3.13 [00:34:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.35, 7.36, 7.01 [00:36:04] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.92, 3.95, 3.38 [00:38:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.98, 7.05, 7.01 [00:39:54] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.19, 3.10, 3.16 [00:40:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.12, 7.97, 7.36 [00:41:35] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.07599338889 secs [00:43:44] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.43, 3.56, 3.33 [00:43:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.46, 3.68, 3.21 [00:44:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.31, 7.02, 7.16 [00:45:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.88, 3.19, 3.25 [00:45:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.02, 3.06, 3.03 [00:48:07] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.63, 3.28, 2.86 [00:48:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.65, 8.92, 6.63 [00:48:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.53, 7.07, 7.11 [00:49:42] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.93, 5.56, 4.06 [00:52:08] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.28, 3.08, 2.91 [00:52:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.08, 7.95, 6.84 [00:52:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.82, 7.77, 7.50 [00:53:37] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.90, 3.61, 3.58 [00:53:43] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.08, 4.10, 3.49 [00:55:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.74, 2.94, 3.34 [00:56:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.87, 6.38, 6.46 [00:57:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.16, 3.68, 3.52 [01:01:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.83, 2.82, 3.22 [01:02:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.83, 7.28, 7.17 [01:04:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.39, 7.30, 6.66 [01:04:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.95, 6.62, 6.94 [01:06:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.99, 4.30, 3.70 [01:08:07] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.16, 3.69, 3.23 [01:12:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.66, 7.58, 7.32 [01:12:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.25, 6.49, 6.78 [01:14:07] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.11, 3.88, 3.54 [01:16:07] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 0.50, 2.62, 3.11 [01:16:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [01:16:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.68, 4.98, 6.39 [01:16:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.65, 7.88, 7.20 [01:24:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.43, 7.14, 7.39 [01:42:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.23, 3.50, 2.87 [01:43:58] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.98, 3.24, 2.84 [01:44:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.76, 8.26, 7.64 [01:45:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.28, 3.11, 3.89 [01:46:00] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [01:46:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.03, 7.02, 7.27 [01:53:23] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.47, 2.54, 3.35 [01:53:43] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.55, 3.97, 3.17 [01:55:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.23, 3.24, 3.00 [01:56:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.66, 6.17, 6.73 [02:02:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.88, 7.59, 7.15 [02:06:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.11, 7.27, 7.16 [02:07:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.18, 4.38, 3.46 [02:08:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.65, 7.76, 7.34 [02:09:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.82, 3.67, 3.32 [02:10:18] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.64, 3.49, 3.12 [02:10:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.89, 7.78, 7.40 [02:11:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.98, 4.44, 3.63 [02:12:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.36, 3.65, 3.24 [02:12:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.32, 8.09, 7.54 [02:14:08] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.73, 4.26, 3.53 [02:15:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.04, 4.00, 3.67 [02:16:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 12.72, 8.80, 4.96 [02:16:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.36, 7.37, 7.39 [02:17:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.11, 5.11, 4.07 [02:17:58] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.12, 3.84, 3.60 [02:19:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.74, 3.91, 3.74 [02:20:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 2.76, 7.23, 5.37 [02:24:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.44, 4.92, 4.91 [02:24:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.99, 6.03, 6.78 [02:25:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.11, 3.02, 3.39 [02:27:24] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:31:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.17, 3.16, 3.39 [02:35:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.89, 3.53, 3.48 [02:37:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.75, 4.75, 3.88 [02:43:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.09, 3.22, 3.57 [02:45:13] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [02:47:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.11, 3.62, 3.58 [02:49:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.24, 3.61, 3.60 [02:51:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.83, 3.09, 3.40 [02:51:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.11, 6.51, 6.23 [02:53:12] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.45, 5.99, 6.07 [02:56:48] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:59:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.79, 4.14, 3.68 [03:01:35] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [03:10:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.88, 3.87, 3.14 [03:12:51] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.80, 3.09, 2.94 [03:14:49] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [03:15:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.91, 7.84, 6.97 [03:20:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.80, 8.13, 7.28 [03:21:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.75, 2.73, 3.93 [03:22:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.22, 7.45, 7.13 [03:27:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.89, 3.76, 3.88 [03:29:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.92, 3.36, 3.72 [03:31:11] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:33:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.76, 2.61, 3.34 [03:34:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.89, 5.10, 3.07 [03:36:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.77, 4.41, 3.09 [03:36:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.24, 6.17, 6.68 [03:49:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.46, 7.78, 6.98 [03:51:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.65, 4.53, 3.67 [03:53:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.78, 7.50, 7.11 [03:55:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.48, 3.56, 3.49 [03:57:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.66, 3.08, 3.31 [03:58:24] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.31, 4.49, 3.45 [03:59:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.86, 9.32, 7.94 [04:02:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.77, 7.83, 7.63 [04:04:09] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.68, 3.93, 3.59 [04:07:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.24, 4.18, 3.54 [04:07:59] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.97, 4.11, 3.68 [04:10:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.31, 7.36, 7.38 [04:11:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.20, 3.61, 3.45 [04:13:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.23, 3.70, 3.48 [04:15:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.30, 3.55, 3.45 [04:15:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.98, 3.33, 3.59 [04:16:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.30, 7.55, 7.58 [04:17:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.79, 3.88, 3.57 [04:19:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.63, 3.37, 3.42 [04:19:43] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.51, 3.36, 3.47 [04:21:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.58, 2.90, 3.28 [04:23:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.01, 3.20, 3.39 [04:24:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.29, 7.81, 7.65 [04:26:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.48, 7.18, 7.44 [04:27:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.47, 3.40, 3.42 [04:29:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.59, 2.84, 3.21 [04:34:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.53, 6.24, 6.79 [04:37:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.61, 3.35, 3.14 [04:38:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.10, 7.50, 7.20 [04:40:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.07, 7.12, 7.11 [04:41:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.43, 3.63, 3.28 [04:41:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.78, 3.06, 2.30 [04:43:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.29, 2.83, 2.31 [04:44:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.50, 6.22, 6.75 [04:49:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.64, 3.62, 3.54 [04:53:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.90, 2.82, 3.24 [05:03:46] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.98, 5.75, 3.12 [05:07:41] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.27, 3.10, 2.62 [05:17:10] [02mediawiki] 07brandon-wm opened pull request 03#15373: Add OOJSPlus to REL1_40 - 13https://github.com/miraheze/mediawiki/pull/15373 [05:17:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.43, 11.04, 11.83 [05:17:15] [02mediawiki] 07brandon-wm closed pull request 03#15373: Add OOJSPlus to REL1_40 - 13https://github.com/miraheze/mediawiki/pull/15373 [05:23:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.11, 10.80, 11.37 [05:26:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.98, 7.29, 6.85 [05:28:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.72, 6.75, 6.71 [05:31:35] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1028482616 secs [05:41:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.08, 7.59, 6.84 [05:43:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.18, 7.46, 6.89 [05:45:19] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.20, 6.77, 6.69 [05:47:35] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08136555552 secs [05:53:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.06, 10.94, 11.75 [05:56:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.18, 7.13, 6.77 [05:59:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.51, 11.84, 11.84 [06:01:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.42, 11.60, 11.78 [06:02:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.58, 6.28, 6.54 [06:03:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.23, 13.11, 12.31 [06:05:35] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1066742837 secs [06:09:35] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.0965282321 secs [06:11:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.79, 4.10, 3.04 [06:19:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.79, 3.99, 3.45 [06:21:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.55, 4.18, 3.58 [06:27:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.58, 3.93, 3.67 [06:27:33] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [06:29:26] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.14, 3.92, 3.69 [06:33:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.49, 3.80, 3.70 [06:37:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.07, 3.85, 3.75 [06:39:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.71, 7.22, 6.89 [06:41:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.55, 6.70, 6.75 [06:45:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.39, 10.48, 11.88 [06:45:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.86, 7.68, 7.14 [06:49:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.25, 7.32, 7.15 [06:49:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.49, 3.97, 3.97 [06:50:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.30, 9.45, 11.40 [06:53:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.55, 4.06, 3.98 [06:54:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.65, 8.08, 7.43 [06:55:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.73, 11.51, 11.41 [06:56:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.43, 10.76, 11.23 [06:56:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.46, 7.39, 7.26 [06:57:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.25, 10.95, 11.22 [06:58:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.41, 10.97, 11.25 [07:00:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.13, 11.54, 11.41 [07:02:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.54, 6.02, 6.65 [07:04:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.13, 11.21, 11.33 [07:09:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.25, 11.83, 11.41 [07:10:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.28, 12.64, 11.78 [07:15:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.34, 11.50, 11.47 [07:17:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.23, 12.17, 11.71 [07:17:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 1.99, 3.30, 3.88 [07:21:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.29, 2.16, 3.27 [07:26:20] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [07:27:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.66, 11.73, 11.81 [07:27:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.09, 7.37, 6.85 [07:29:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.08, 11.73, 11.79 [07:29:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.28, 7.13, 6.81 [07:30:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.10, 11.46, 11.92 [07:32:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.78, 11.80, 11.98 [07:33:28] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.87, 6.76, 6.76 [07:34:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.29, 11.86, 12.00 [07:36:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.46, 12.54, 12.26 [07:38:24] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [07:41:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.70, 11.63, 11.94 [07:43:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.54, 12.06, 12.07 [07:44:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.37, 7.12, 6.78 [07:45:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.68, 11.68, 11.94 [07:48:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.64, 7.60, 7.06 [07:49:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.39, 12.38, 12.11 [07:50:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.30, 7.90, 7.22 [07:52:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.50, 6.86, 6.92 [07:54:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.04, 6.29, 6.70 [07:57:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.64, 11.56, 11.89 [08:00:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.80, 10.69, 11.93 [08:05:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.15, 12.63, 11.99 [08:07:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.14, 11.58, 11.68 [08:08:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.30, 12.91, 12.07 [08:09:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.45, 12.47, 12.00 [08:11:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.84, 11.99, 11.86 [08:13:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.68, 12.29, 11.99 [08:36:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.32, 10.43, 11.78 [08:37:05] PROBLEM - db121 Backups SQL on db121 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209695 seconds old and 44336409 bytes [08:38:08] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:39:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.11, 10.94, 11.96 [08:42:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.92, 12.20, 11.95 [08:48:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.07, 10.88, 11.60 [08:56:31] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:57:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.39, 11.92, 11.06 [08:59:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.11, 10.81, 10.75 [09:03:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.15, 12.48, 11.33 [09:08:00] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [09:08:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.37, 10.69, 10.55 [09:11:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.27, 11.78, 11.49 [09:13:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.96, 12.20, 11.68 [09:15:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.34, 11.78, 11.58 [09:17:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.83, 11.93, 11.65 [09:22:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.84, 11.78, 11.76 [09:23:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.53, 11.46, 11.66 [09:24:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.98, 12.29, 11.92 [09:25:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.75, 11.95, 11.81 [09:25:55] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [09:26:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.22, 11.46, 11.70 [09:27:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.04, 10.52, 11.31 [09:34:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.03, 10.28, 10.83 [09:35:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.43, 6.62, 6.14 [09:35:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.92, 11.33, 11.12 [09:36:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.88, 9.81, 10.59 [09:37:07] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.00, 6.00, 5.97 [09:37:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.33, 10.65, 10.88 [09:40:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.53, 10.71, 10.70 [09:40:55] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.59, 7.85, 6.78 [09:42:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.03, 8.26, 7.05 [09:44:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.96, 11.48, 11.19 [09:44:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.88, 7.30, 6.83 [09:47:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.78, 11.55, 11.05 [09:48:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.40, 11.41, 11.17 [09:49:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.50, 10.58, 10.78 [09:52:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.97, 11.58, 11.29 [09:53:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.40, 11.24, 10.89 [09:54:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.59, 6.37, 6.74 [09:55:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.83, 10.96, 10.83 [09:56:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.07, 12.22, 11.58 [09:57:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.05, 11.39, 10.97 [09:59:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.78, 10.92, 10.87 [10:02:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.88, 11.66, 11.61 [10:03:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.83, 11.80, 11.20 [10:05:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.18, 11.19, 11.07 [10:09:40] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [10:12:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.89, 11.99, 11.69 [10:14:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.97, 11.00, 11.38 [10:17:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.57, 9.42, 10.13 [10:20:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.13, 10.47, 10.86 [10:22:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.22, 10.92, 11.00 [10:23:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.10, 10.19, 10.26 [10:24:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.17, 11.52, 11.20 [10:26:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.12, 11.00, 11.04 [10:27:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.25, 9.60, 10.01 [10:34:48] [02mediawiki] 07Reception123 commented on pull request 03#15266: T10794: Add Extension:OOJSPlus - 13https://github.com/miraheze/mediawiki/pull/15266#issuecomment-1573517238 [10:36:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.80, 10.59, 10.31 [10:37:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.84, 12.05, 10.86 [10:38:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.82, 11.77, 11.16 [10:38:56] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [10:39:32] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [10:45:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.84, 11.98, 11.47 [10:49:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.91, 12.40, 11.70 [10:50:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.87, 7.10, 6.53 [10:52:11] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.98, 6.63, 6.42 [10:54:06] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [10:55:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.86, 6.90, 6.63 [10:57:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.40, 11.78, 11.96 [10:59:47] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.30, 6.18, 6.39 [11:00:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.35, 10.61, 11.82 [11:01:00] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mineland.eu. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:03:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.84, 11.70, 11.81 [11:05:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.99, 7.27, 6.81 [11:07:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.50, 11.33, 11.69 [11:07:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.74, 6.80, 6.70 [11:08:32] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [11:12:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.33, 12.32, 11.54 [11:14:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.81, 11.97, 11.51 [11:17:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.91, 11.82, 11.58 [11:18:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.72, 11.52, 11.37 [11:20:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.02, 7.10, 6.65 [11:22:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.12, 6.93, 6.63 [11:24:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.55, 6.46, 6.50 [11:29:59] RECOVERY - wiki.mineland.eu - reverse DNS on sslhost is OK: SSL OK - wiki.mineland.eu reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:33:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.23, 7.83, 6.94 [11:37:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.05, 7.92, 7.18 [11:48:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.20, 7.50, 7.16 [11:50:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.91, 7.38, 7.17 [11:58:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.16, 6.07, 6.65 [12:02:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.45, 7.10, 7.01 [12:04:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.37, 7.85, 7.28 [12:06:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.52, 7.13, 7.10 [12:08:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.57, 6.24, 6.78 [12:18:50] PROBLEM - wiki.hsdfz.org.cn - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['emily.ns.cloudflare.com.', 'lloyd.ns.cloudflare.com.'], 'CNAME': 'hsdfz.miraheze.org.'} [12:20:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.25, 7.60, 6.95 [12:22:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.44, 7.46, 6.98 [12:24:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.26, 6.66, 6.74 [12:25:16] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:28:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.12, 8.90, 7.68 [12:29:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.57, 3.23, 2.69 [12:31:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.51, 2.98, 2.68 [12:32:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.04, 7.95, 7.62 [12:34:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.04, 8.43, 7.82 [12:38:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.13, 7.87, 7.77 [12:54:40] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [12:58:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.17, 6.80, 6.93 [13:00:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.95, 6.89, 6.93 [13:02:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.27, 7.12, 7.00 [13:06:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.65, 7.25, 7.07 [13:10:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.21, 8.03, 7.35 [13:14:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.95, 7.96, 7.52 [13:20:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.78, 8.98, 8.01 [13:23:37] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.63, 3.47, 2.47 [13:24:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.79, 7.54, 7.67 [13:26:03] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [13:27:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.51, 3.45, 2.73 [13:31:28] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.57, 3.26, 2.83 [13:32:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.51, 7.68, 7.57 [13:40:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.18, 7.96, 7.89 [13:42:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.75, 8.13, 7.97 [13:44:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.71, 7.89, 7.91 [13:46:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.82, 8.66, 8.18 [13:52:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.82, 7.14, 3.92 [13:54:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.11, 6.36, 4.06 [13:55:26] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:56:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.63, 7.59, 7.94 [13:58:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.14, 7.74, 7.94 [14:07:06] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [14:08:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.74, 6.98, 7.68 [14:12:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.89, 7.39, 7.65 [14:21:31] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.4551858902 secs [14:24:50] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [14:37:07] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [14:39:31] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.001581579447 secs [14:53:43] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.88, 3.29, 2.37 [14:53:43] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.58, 3.58, 3.05 [14:55:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.99, 2.73, 2.28 [14:57:36] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.31, 3.50, 3.14 [14:59:32] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.21, 3.36, 3.13 [15:07:54] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.28, 3.90, 2.70 [15:12:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.26, 4.97, 2.86 [15:13:47] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.10, 3.77, 3.15 [15:14:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.06, 7.16, 7.90 [15:16:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.64, 7.86, 8.07 [15:17:42] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.63, 3.40, 3.15 [15:18:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.39, 5.63, 4.12 [15:18:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.81, 7.60, 7.94 [15:20:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.34, 8.30, 8.15 [15:22:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.29, 7.59, 7.89 [15:23:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.78, 2.63, 2.06 [15:25:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.07, 2.29, 2.00 [15:28:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.59, 8.31, 8.02 [15:31:35] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [15:32:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.11, 7.44, 7.74 [15:33:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a91692b147e0...be054bc7bf34 [15:33:14] [02miraheze/mw-config] 07paladox 03be054bc - Temporarily disable siwftTempUrlKey [15:33:33] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [15:33:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:33:59] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [15:34:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:34:09] !log [paladox@mwtask141] starting deploy of {'config': True} to all [15:34:18] miraheze/mw-config - paladox the build passed. [15:34:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:34:21] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 12s [15:34:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:34:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 14.05, 8.80, 8.13 [15:37:06] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [15:43:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.99, 4.01, 2.69 [15:45:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.88, 3.90, 2.82 [15:48:07] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.33, 7.43, 5.75 [15:49:01] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.69, 4.67, 3.32 [15:50:02] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.18, 5.58, 5.27 [15:51:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.79, 3.40, 2.98 [15:52:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.70, 3.85, 3.30 [15:54:54] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.44, 3.34, 3.17 [15:56:29] !log [@test131] starting deploy of {'config': True} to all [15:56:30] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [15:56:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.63, 7.40, 7.96 [15:56:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:00:03] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.76, 3.19, 2.88 [16:00:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.81, 8.65, 8.30 [16:01:10] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [16:02:03] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.91, 3.04, 2.86 [16:06:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.82, 7.80, 8.00 [16:07:07] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [16:08:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.33, 8.39, 8.19 [16:13:28] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.74, 3.99, 3.45 [16:15:52] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:17:23] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.21, 2.75, 3.08 [16:21:35] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1439752579 secs [16:24:25] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:37:35] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08274620771 secs [16:37:55] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.48, 2.97, 2.75 [16:39:52] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.93, 3.09, 2.83 [16:42:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.27, 5.49, 3.03 [16:44:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.55, 6.13, 3.56 [16:45:44] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [16:46:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.13, 7.61, 4.48 [16:48:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.43, 6.40, 4.44 [16:50:03] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.43, 3.23, 2.99 [16:52:03] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.00, 3.12, 2.98 [16:52:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.87, 4.53, 3.51 [16:53:50] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [16:56:03] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 5.53, 4.10, 3.39 [16:56:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.98, 3.92, 3.49 [16:58:03] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 4.00, 4.00, 3.44 [16:58:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.91, 3.19, 3.27 [17:02:02] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.21, 3.15, 3.24 [17:07:35] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.112141192 secs [17:08:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:11:35] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.08497220278 secs [17:44:24] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.09, 3.30, 2.67 [17:48:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.89, 3.45, 2.91 [17:50:08] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.50, 3.06, 2.82 [17:53:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [18:13:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.89, 3.40, 2.42 [18:15:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.33, 3.37, 2.53 [18:15:35] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.118606776 secs [18:19:35] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.07658758759 secs [18:20:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.14, 5.43, 2.83 [18:22:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 6.20, 6.02, 3.40 [18:22:36] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.70, 3.32, 2.83 [18:26:29] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.19, 3.53, 3.01 [18:26:48] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.14, 3.79, 3.06 [18:28:26] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.11, 3.39, 3.02 [18:31:35] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1800357401 secs [18:32:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.79, 3.45, 3.28 [18:34:38] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.99, 4.32, 3.62 [18:37:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.03, 3.41, 2.98 [18:43:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.10, 3.68, 3.30 [18:44:49] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.39, 6.55, 5.96 [18:46:46] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.19, 6.32, 5.94 [18:47:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.03, 3.40, 3.27 [18:48:23] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.64, 3.77, 3.93 [18:51:01] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [18:53:35] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.06979978085 secs [18:54:15] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.19, 2.62, 3.38 [19:00:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.10, 4.04, 3.67 [19:09:54] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.61, 3.68, 3.85 [19:11:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.55, 3.45, 3.20 [19:13:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.32, 2.38, 3.27 [19:15:35] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1161845922 secs [19:17:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.52, 2.77, 3.05 [19:20:53] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:25:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.88, 4.13, 3.71 [19:27:28] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.00, 3.32, 3.46 [19:27:35] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.08869761229 secs [19:27:51] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.45, 3.97, 3.14 [19:28:13] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [19:28:35] PROBLEM - swiftobject122 SSH on swiftobject122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:48] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.11, 3.14, 3.37 [19:30:09] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.287 second response time [19:30:31] RECOVERY - swiftobject122 SSH on swiftobject122 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:30:42] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.323 second response time [19:31:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.17, 2.64, 2.80 [19:35:56] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:38:40] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [19:38:57] PROBLEM - swiftobject122 SSH on swiftobject122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:39:17] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:39:26] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [19:40:03] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [19:40:49] PROBLEM - swiftobject122 conntrack_table_size on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:41:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:41:00] PROBLEM - swiftobject122 Puppet on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:41:30] PROBLEM - swiftobject122 PowerDNS Recursor on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:42:07] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.327 second response time [19:42:20] PROBLEM - swiftobject122 ferm_active on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:42:38] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.081 second response time [19:42:48] RECOVERY - swiftobject122 conntrack_table_size on swiftobject122 is OK: OK: nf_conntrack is 0 % full [19:42:58] RECOVERY - swiftobject122 SSH on swiftobject122 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:43:02] RECOVERY - swiftobject122 Puppet on swiftobject122 is OK: OK: Puppet is currently enabled, last run 49 minutes ago with 0 failures [19:43:18] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.035 second response time [19:43:28] RECOVERY - swiftobject122 PowerDNS Recursor on swiftobject122 is OK: DNS OK: 0.918 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [19:43:34] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.019 second response time [19:44:16] RECOVERY - swiftobject122 ferm_active on swiftobject122 is OK: OK ferm input default policy is set [19:46:00] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.045 second response time [19:47:59] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.56, 3.74, 3.40 [19:49:57] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.82, 3.05, 3.19 [19:51:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:52:40] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.75, 6.25, 3.96 [19:54:35] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.22, 6.65, 4.39 [19:56:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.37, 4.62, 3.92 [20:00:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 11.00, 7.06, 5.66 [20:02:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.71, 6.97, 5.79 [20:04:21] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.26, 6.24, 5.67 [20:06:32] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.13, 3.32, 2.72 [20:08:28] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.82, 3.46, 2.84 [20:14:17] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.38, 3.07, 2.89 [20:29:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.96, 4.64, 3.66 [20:31:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.50, 3.73, 3.44 [20:33:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.12, 3.10, 3.24 [20:37:45] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.61, 3.42, 2.35 [20:39:45] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.14, 2.58, 2.17 [20:40:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.03, 6.02, 7.88 [20:41:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.83, 4.36, 3.82 [20:43:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.20, 3.93, 3.72 [20:44:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.68, 6.99, 7.82 [20:46:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.86, 6.23, 7.45 [20:49:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.43, 4.77, 4.07 [20:50:30] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [20:55:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.35, 3.37, 3.70 [20:56:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.41, 5.74, 6.74 [20:59:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.47, 2.42, 3.24 [21:03:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.20, 3.31, 3.46 [21:03:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.27, 7.01, 7.00 [21:05:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.66, 4.27, 3.74 [21:13:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.70, 3.85, 3.92 [21:16:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.45, 7.25, 7.07 [21:18:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.70, 7.36, 7.13 [21:21:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.53, 2.77, 3.36 [21:26:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.72, 6.38, 6.80 [21:32:05] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:36:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.76, 6.88, 6.83 [21:44:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.56, 7.88, 7.23 [21:48:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.17, 6.95, 7.03 [21:50:12] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.92, 3.42, 2.84 [21:52:03] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.92, 3.23, 2.69 [21:52:05] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:52:07] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.01, 3.24, 2.84 [21:54:03] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.63, 3.74, 2.94 [21:54:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.88, 6.39, 6.79 [21:57:46] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [21:58:02] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.53, 3.79, 3.17 [21:58:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.26, 7.29, 7.05 [22:00:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.75, 6.71, 6.88 [22:04:03] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 1.59, 2.95, 3.06 [22:04:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.70, 6.53, 6.79 [22:06:22] [02puppet] 07The-Voidwalker closed pull request 03#3211: remove /etc/cron.daily/apt - 13https://github.com/miraheze/puppet/pull/3211 [22:06:23] [02miraheze/puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 [22:06:25] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/50433028c65b...68152448864f [22:06:26] [02miraheze/puppet] 07The-Voidwalker 036815244 - remove /etc/cron.daily/apt (#3211) [22:06:29] [02puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/puppet [22:06:50] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.89, 4.00, 3.28 [22:07:07] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [22:08:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.60, 4.33, 3.28 [22:10:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.83, 3.69, 3.16 [22:12:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.10, 7.70, 7.06 [22:12:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.61, 3.35, 3.25 [22:14:09] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.69, 3.07, 3.02 [22:16:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.30, 3.56, 3.36 [22:18:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.53, 7.77, 7.35 [22:20:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.06, 3.20, 3.24 [22:20:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.30, 8.29, 7.59 [22:22:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.21, 7.29, 7.30 [22:24:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.84, 7.86, 7.46 [22:27:10] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [22:27:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.81, 4.02, 3.59 [22:28:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.41, 7.23, 7.37 [22:29:17] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.86, 3.85, 3.57 [22:33:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.72, 3.11, 3.32 [22:34:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.73, 5.97, 6.79 [22:37:07] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:38:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.08, 7.37, 7.19 [22:39:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.38, 4.98, 4.08 [22:40:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.75, 7.46, 7.24 [22:41:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.03, 3.84, 3.77 [22:45:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.50, 2.51, 3.23 [22:46:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.72, 8.24, 7.56 [22:50:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.98, 7.97, 7.67 [22:52:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.09, 5.88, 3.16 [22:54:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.69, 4.69, 3.07 [22:54:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.82, 7.90, 7.64 [22:56:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.35, 7.32, 7.46 [23:00:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.58, 7.70, 7.49 [23:01:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.47, 3.62, 3.43 [23:02:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.63, 7.24, 7.34 [23:02:54] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1560099721 secs [23:03:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.39, 3.23, 3.32 [23:14:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.26, 6.42, 6.79 [23:19:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.32, 3.52, 3.25 [23:21:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.01, 7.92, 7.24 [23:21:54] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.91, 2.90, 3.05 [23:23:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.12, 7.25, 7.08 [23:26:54] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.08212170005 secs [23:31:00] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.99, 6.20, 6.66 [23:36:03] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.06, 3.33, 2.72 [23:36:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.04, 7.19, 6.88 [23:37:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.62, 2.82, 2.29 [23:38:03] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.65, 3.17, 2.74 [23:38:43] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.53, 6.53, 6.67 [23:39:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.45, 2.71, 2.32 [23:44:24] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.95, 3.98, 3.11 [23:45:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.11, 7.14, 6.87 [23:47:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.71, 7.64, 7.06 [23:47:28] !log [void@puppet141] sudo salt-ssh -E '.*' cmd.run 'touch /etc/cron.daily/apt.disabled' [23:47:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:51:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.79, 7.12, 7.01 [23:52:34] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.87, 4.34, 3.29 [23:56:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.20, 3.84, 3.70 [23:56:24] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.98, 3.79, 3.30 [23:56:55] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.48, 6.92, 6.87 [23:58:07] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.78, 2.75, 3.31 [23:58:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.78, 6.23, 6.62