[00:03:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.09, 3.27, 3.43 [00:04:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.12, 4.36, 3.80 [00:05:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.36, 4.17, 3.70 [00:05:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.69, 4.88, 3.99 [00:07:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.17, 10.73, 10.03 [00:09:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.00, 1.42, 1.98 [00:09:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.00, 10.61, 10.08 [00:13:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.95, 10.04, 9.95 [00:15:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.94, 1.14, 1.67 [00:20:59] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.74, 1.50, 1.35 [00:21:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.99, 11.17, 10.49 [00:22:56] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.16, 1.25, 1.27 [00:23:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.16, 12.02, 10.87 [00:24:13] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.60, 3.17, 2.53 [00:26:13] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.83, 3.01, 2.55 [00:26:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.56, 3.41, 3.97 [00:29:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.36, 11.71, 11.18 [00:35:39] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.23, 11.73, 11.32 [00:36:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.77, 3.80 [00:36:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.13, 3.22, 3.93 [00:37:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.96, 11.09, 11.14 [00:38:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.81, 3.83, 3.82 [00:38:33] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.89, 2.63, 1.82 [00:42:27] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.88, 1.81, 1.67 [00:42:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.59, 3.37, 3.72 [00:44:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.80, 3.19, 3.61 [00:46:21] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.49, 1.56, 1.60 [00:48:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.58, 3.66 [00:49:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.48, 11.10, 11.01 [00:50:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.99, 3.14, 3.38 [00:50:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.76, 3.42, 3.60 [00:51:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.20, 10.46, 10.80 [00:52:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.35, 3.78, 3.71 [00:54:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.35, 3.50, 3.61 [00:55:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.04, 10.89, 10.87 [00:56:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.63, 4.00, 3.78 [00:57:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.24, 10.27, 10.65 [00:58:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.92, 3.80, 3.57 [00:58:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.70, 3.86, 3.76 [01:00:13] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.63, 3.61, 2.96 [01:00:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 3.92, 3.78 [01:02:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.70, 4.09, 3.76 [01:02:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.40, 3.87, 3.78 [01:04:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.72, 3.36, 3.84 [01:04:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.83, 4.06, 3.87 [01:05:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 15.08, 11.63, 10.87 [01:06:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.74, 3.62, 3.73 [01:08:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.81, 3.70, 3.75 [01:08:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.01, 4.20, 3.93 [01:09:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.58, 11.22, 10.93 [01:12:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.49, 3.74, 3.84 [01:16:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.48, 2.01, 1.50 [01:18:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.46, 3.57 [01:18:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.66, 4.18, 3.82 [01:18:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.66, 1.80, 1.48 [01:18:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.66, 4.24, 3.92 [01:20:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.88, 3.61, 3.61 [01:20:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.49, 1.40 [01:24:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 9.85, 5.86, 4.41 [01:26:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.64, 1.92, 1.62 [01:28:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.58, 1.73, 1.59 [01:30:13] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 2.39, 3.21, 3.86 [01:32:19] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:36:13] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.39, 2.11, 3.16 [01:36:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.61, 1.67, 1.63 [01:39:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.38, 9.58, 10.19 [01:49:44] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 2.00, 1.61, 1.30 [01:51:40] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.88, 1.30, 1.22 [02:03:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.13, 2.04, 1.72 [02:07:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.76, 1.88, 1.72 [02:08:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.42, 10.50, 9.88 [02:09:14] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.02, 1.57, 1.62 [02:23:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.00, 11.51, 10.64 [02:25:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.69, 11.07, 10.58 [02:30:19] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [02:33:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.92, 9.71, 8.79 [02:35:46] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.35, 9.20, 8.72 [02:37:53] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.83, 9.10, 9.98 [02:50:37] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.79, 1.67, 1.37 [02:50:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.59, 10.70, 10.28 [02:52:34] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.92, 1.41, 1.31 [03:00:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.64, 2.11, 1.66 [03:02:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.17, 1.82, 1.62 [03:04:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.20, 1.63, 1.57 [03:04:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 11.46, 10.87 [03:06:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.61, 11.45, 10.94 [03:12:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.41, 1.93, 1.68 [03:12:33] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [03:16:37] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.56, 11.02, 10.81 [03:18:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.76, 10.59, 9.60 [03:18:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.80, 2.00, 2.00 [03:18:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.26, 10.89, 10.78 [03:20:23] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.30, 9.63, 9.38 [03:20:29] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.56, 2.27, 2.07 [03:27:33] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [03:30:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.45, 1.52, 1.97 [03:30:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.13, 9.34, 10.16 [03:34:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.46, 3.21, 3.95 [03:36:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 4.00, 4.18 [03:36:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.12, 1.24, 1.69 [03:36:30] [02dns] 07Universal-Omega opened pull request 03#296: miraheze.org: add google-site-verification - 13https://github.com/miraheze/dns/pull/296 [03:36:30] [url] Page not found · GitHub · GitHub | github.com [03:40:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.41, 3.90 [03:46:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.69, 3.22, 3.91 [03:46:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.75, 3.38, 3.88 [03:48:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.16, 3.53, 3.68 [03:48:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.76, 4.56, 4.22 [03:50:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.84, 2.87, 3.42 [03:52:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.49, 3.92, 3.71 [03:54:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.08, 4.10, 3.97 [03:54:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.77, 3.68 [03:56:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.98, 4.47, 3.94 [03:58:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.98, 3.95, 3.82 [03:58:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.68, 3.72, 3.97 [04:00:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.79, 3.70, 3.90 [04:00:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 3.72, 3.72 [04:02:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 3.82, 3.91 [04:02:19] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [04:02:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.98, 3.71, 3.72 [04:03:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.87, 10.82, 10.29 [04:04:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.46, 3.76 [04:05:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.08, 11.43, 10.57 [04:07:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.24, 10.81, 10.44 [04:10:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.51, 3.05, 3.38 [04:12:05] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.74, 2.82, 3.40 [04:16:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.80, 3.88, 3.71 [04:20:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.89, 3.64 [04:22:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.87, 1.40, 1.12 [04:22:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.34, 1.64, 1.30 [04:22:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.07, 3.89, 3.84 [04:23:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.22, 11.03, 10.77 [04:24:08] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.24, 1.26, 1.10 [04:24:14] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.13, 1.45, 1.27 [04:24:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.57, 4.04, 3.89 [04:25:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.20, 11.17, 10.86 [04:26:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 4.00, 3.74 [04:27:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.32, 4.11, 3.66 [04:30:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.86, 3.79 [04:31:02] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.40, 10.25, 9.19 [04:31:08] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [04:32:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.59, 3.95, 3.93 [04:33:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.99, 3.95, 3.76 [04:33:00] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.02, 10.04, 9.26 [04:34:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 4.12, 4.01 [04:35:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.09, 9.60, 10.18 [04:36:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.56, 3.50, 3.79 [04:36:58] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.89, 2.81, 3.33 [04:38:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.38, 3.04, 3.39 [04:42:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.64, 3.38, 3.58 [04:42:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.20, 3.37, 3.38 [04:44:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.35, 3.55 [04:44:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.18, 3.29, 3.35 [04:48:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.55, 3.43 [04:52:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.89, 3.45, 3.43 [04:52:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.80, 3.28, 3.37 [04:53:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.76, 3.24, 3.35 [04:58:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.17, 3.63, 3.55 [04:58:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.23, 4.21, 3.72 [05:00:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.64, 3.56 [05:00:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.86, 3.66, 3.58 [05:02:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.54, 3.53 [05:04:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.22, 3.21, 3.40 [05:05:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.49, 3.38 [05:06:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 4.12, 3.77 [05:10:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.40, 3.62, 3.68 [05:11:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.49, 2.81, 3.17 [05:12:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 3.90, 3.76 [05:17:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.71, 3.43, 3.32 [05:21:41] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.37, 3.86, 3.51 [05:22:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.66, 3.83, 3.92 [05:23:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.11, 3.51, 3.42 [05:25:39] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.45, 3.16, 3.30 [05:27:36] [02dns] 07Reception123 closed pull request 03#296: miraheze.org: add google-site-verification - 13https://github.com/miraheze/dns/pull/296 [05:27:37] [url] Page not found · GitHub · GitHub | github.com [05:27:38] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/b26e3ca42276...f4690861bd0d [05:27:38] [url] Comparing b26e3ca42276...f4690861bd0d · miraheze/dns · GitHub | github.com [05:27:39] [02miraheze/dns] 07Universal-Omega 03f469086 - miraheze.org: add google-site-verification (#296) [05:27:41] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/f4690861bd0d...ed2a9b93a63c [05:27:41] [url] Comparing f4690861bd0d...ed2a9b93a63c · miraheze/dns · GitHub | github.com [05:27:42] [02miraheze/dns] 07Universal-Omega 03ed2a9b9 - Delete lakehub.ga zone (#295) [05:27:44] [02dns] 07Reception123 closed pull request 03#295: Delete lakehub.ga zone - 13https://github.com/miraheze/dns/pull/295 [05:27:44] [url] Page not found · GitHub · GitHub | github.com [05:30:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.36, 2.93, 3.39 [05:32:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.02, 3.90, 3.49 [05:34:35] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.73, 4.04, 3.57 [05:36:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.48, 4.21, 3.74 [05:42:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.96, 3.91 [05:42:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.79, 3.82 [05:44:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 3.93, 3.85 [05:44:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.18, 3.66, 3.84 [05:46:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.86, 3.68, 3.76 [05:48:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.71, 3.77 [05:48:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.81, 3.86, 3.82 [05:48:27] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.53, 1.58, 1.17 [05:50:24] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.16, 1.34, 1.13 [05:50:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.50, 3.95, 3.84 [05:52:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.74, 3.77 [05:52:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.06, 2.70, 3.29 [05:54:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.15, 2.51, 3.23 [06:02:19] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:02:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.30, 3.46 [06:02:54] [02MirahezeMagic] 07Universal-Omega closed pull request 03#313: Recaptcha: use enterprise version - 13https://github.com/miraheze/MirahezeMagic/pull/313 [06:02:55] [url] Page not found · GitHub · GitHub | github.com [06:02:58] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03RhinosF1-patch-1 [06:02:59] [02MirahezeMagic] 07Universal-Omega deleted branch 03RhinosF1-patch-1 - 13https://github.com/miraheze/MirahezeMagic [06:03:00] ... [06:04:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.56, 3.01, 3.33 [06:11:50] PROBLEM - wiki.cyberfurs.org - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406944513320923 seconds [06:16:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.32, 3.53, 3.00 [06:18:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.54, 3.16 [06:18:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.35, 2.99 [06:20:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.39, 3.18, 3.08 [06:21:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.47, 3.30 [06:22:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.91, 2.95, 2.96 [06:23:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.00, 2.94, 3.13 [06:27:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.69, 3.97, 3.44 [06:28:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.29, 4.14, 3.48 [06:29:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.85, 3.47 [06:31:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.70, 4.83, 3.86 [06:31:08] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [06:32:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.29, 4.73, 3.75 [06:40:37] RECOVERY - wiki.cyberfurs.org - reverse DNS on sslhost is OK: SSL OK - wiki.cyberfurs.org reverse DNS resolves to cp21.miraheze.org - CNAME OK [06:56:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.16, 3.47, 3.93 [06:58:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 3.76, 3.98 [07:02:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.59, 3.88 [07:10:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.42, 3.97 [07:12:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.78, 4.04 [07:14:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.49, 3.37, 3.51 [07:16:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.14, 3.15, 3.40 [07:18:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.60, 3.98 [07:22:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.58, 3.86 [07:22:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.74, 9.89, 8.98 [07:24:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.64, 3.86 [07:24:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.32, 9.56, 8.96 [07:27:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.50, 3.44 [07:30:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.85, 3.70, 3.95 [07:32:19] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [07:34:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.75, 3.44, 3.59 [07:34:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.59, 3.83 [07:34:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.38, 3.03, 3.29 [07:36:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.31, 3.53 [07:36:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.57, 3.79 [07:38:05] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.89, 2.98, 3.37 [07:42:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.68, 3.71 [07:44:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.57, 4.20, 3.55 [07:48:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.85, 3.56 [07:58:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.56, 4.14, 3.74 [08:00:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.60, 4.47, 3.81 [08:02:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.35, 3.97, 3.70 [08:02:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.11, 3.47, 3.60 [08:06:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.82, 9.38, 8.58 [08:08:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.86, 3.72 [08:08:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.07, 9.25, 8.63 [08:14:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.41, 3.37, 3.61 [08:16:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.33, 3.10, 3.40 [08:16:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.74, 3.58, 3.98 [08:20:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.22, 3.56, 3.57 [08:24:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.06, 3.51 [08:26:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.67, 3.63 [08:28:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.02, 4.20, 3.83 [08:28:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.54, 3.44 [08:28:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.21, 3.46 [08:30:19] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [08:30:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.55, 3.28, 3.36 [08:30:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.75, 2.70, 3.25 [08:48:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.00, 3.88, 3.65 [08:49:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.76, 3.43 [08:50:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.57, 3.56 [08:56:30] !log reception@mwtask111:~$ sudo -u www-data php /home/reception/rollbackEdits.php --wiki=webkinzguidewiki --user=Miraheze_maintenance_script [08:56:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:56:52] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.67, 3.30, 3.34 [09:00:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.55, 3.54 [09:01:13] MacFan4000: when do you think you could do the final part of https://phabricator.miraheze.org/T9479 so we can close the task? [09:01:14] [url] ⚓ T9479 Offboard RhinosF1 shell | phabricator.miraheze.org [09:02:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.62, 3.25, 3.44 [09:03:08] Reception123: I've done it [09:03:20] https://github.com/FOSSBots/puppet/compare/0200e1cc55c5...3ee1458a508d [09:03:20] [url] Comparing 0200e1cc55c5...3ee1458a508d · FOSSBots/puppet · GitHub | github.com [09:03:41] I'll sync it when puppet runs [09:04:05] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.68, 3.09, 3.36 [09:04:44] thanks RhinosF1 ! [09:08:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.43, 3.46 [09:10:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 3.71, 3.55 [09:11:00] Reception123: cache refreshed [09:11:19] ack [09:12:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.78, 3.30, 3.42 [09:14:05] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.11, 2.94, 3.28 [09:21:50] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.73, 1.66, 1.38 [09:23:49] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.91, 1.30, 1.28 [09:32:19] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:42:45] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.06, 1.79, 1.23 [09:44:42] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.55, 1.34, 1.13 [09:46:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.23, 2.95 [09:48:28] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.83, 2.98, 2.89 [10:01:08] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [10:02:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.71, 3.65, 3.11 [10:03:42] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.90, 3.04, 1.88 [10:04:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.59, 2.87, 2.89 [10:07:38] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.50, 1.65, 1.57 [10:12:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.32, 3.06 [10:14:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.22, 2.87, 2.92 [10:18:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 3.48, 3.13 [10:19:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.42, 3.12 [10:20:41] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.83, 3.52, 3.18 [10:21:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.22, 3.38, 3.14 [10:22:41] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.04, 3.30, 3.14 [10:30:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.31, 3.12 [10:32:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.32, 3.29, 3.14 [10:33:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.30, 2.97 [10:35:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.73, 3.60, 3.13 [10:36:35] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.73, 3.49, 3.29 [10:38:34] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.63, 3.18, 3.20 [10:38:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.07, 3.38, 3.15 [10:46:40] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.44, 1.88, 1.29 [10:48:40] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.91, 1.74, 1.30 [10:50:39] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.74, 1.34, 1.20 [11:03:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.52, 3.51, 3.03 [11:05:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.27, 3.36, 3.02 [11:37:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.15, 2.71 [11:39:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.25, 2.85, 2.66 [12:16:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.42, 2.97 [12:20:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.76, 3.22, 3.00 [12:32:19] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:34:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.16, 2.14, 1.51 [12:36:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.98, 1.75, 1.44 [12:38:11] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.56, 1.70, 1.46 [12:38:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.82, 10.36, 9.80 [12:40:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.65, 4.02, 3.31 [12:42:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.24, 3.25, 3.10 [12:48:44] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 15.95, 13.02, 11.14 [12:49:14] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:30] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:04] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:08] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:51:17] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27067 bytes in 5.772 second response time [12:51:28] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.319 second response time [12:52:03] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.870 second response time [12:52:07] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.026 second response time [12:52:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.73, 3.74, 3.16 [12:52:46] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [12:53:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [12:54:43] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 3.68, 3.22 [12:54:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.29, 11.51, 11.21 [12:55:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [12:56:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.90, 4.08, 3.41 [12:58:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.02, 4.19, 3.47 [13:01:08] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:02:10] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.52, 10.44, 9.05 [13:04:08] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.96, 9.52, 8.88 [13:06:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.06, 4.26, 3.57 [13:10:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.42, 3.60, 3.50 [13:10:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.31, 3.97, 3.85 [13:12:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.38, 4.05, 3.89 [13:13:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.12, 10.00, 8.86 [13:14:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.74, 3.56 [13:14:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.89, 9.69, 10.17 [13:15:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.33, 9.62, 8.86 [13:18:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.94, 3.93, 3.68 [13:18:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.06, 10.07, 10.25 [13:20:35] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.43, 3.65, 3.85 [13:20:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.79, 9.62, 10.05 [13:22:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.53, 3.04, 3.37 [13:22:34] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 4.00, 3.96 [13:24:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.84, 3.91 [13:28:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.51, 3.48 [13:28:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.76, 3.64, 3.98 [13:28:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.61, 4.07, 4.02 [13:28:46] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=metawiki --user MacFan4000 (END - exit=0) [13:28:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:30:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.96, 2.98, 3.29 [13:30:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.91, 3.25, 3.71 [13:34:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.41, 4.14, 3.94 [13:36:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 3.73, 3.83 [13:40:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.85, 3.87, 3.44 [13:42:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.62, 10.80, 9.96 [13:45:09] [02miraheze/TSPortal] 07OwenBaines pushed 033 commits to 03master [+0/-0/±9] 13https://github.com/miraheze/TSPortal/compare/e96106cee3a7...62bc6944d027 [13:45:09] [url] Comparing e96106cee3a7...62bc6944d027 · miraheze/TSPortal · GitHub | github.com [13:45:10] [02miraheze/TSPortal] 07OwenBaines 03b62ec8e - Rename subject to user for Reports [13:45:12] [02miraheze/TSPortal] 07OwenBaines 032777930 - Rename subject to user for DPA [13:45:13] [02miraheze/TSPortal] 07OwenBaines 0362bc694 - [Release] Version 3 [13:45:15] [02miraheze/TSPortal] 07OwenBaines tagged 0362bc694 as 03v3 13https://github.com/miraheze/TSPortal/commit/62bc6944d0271e5260613078319bed036cb6de83 [13:45:16] [02TSPortal] 07OwenBaines created tag 03v3 - 13https://github.com/miraheze/TSPortal [13:45:17] [url] Page not found · GitHub · GitHub | github.com [13:47:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.71, 10.59, 9.91 [13:48:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d73de012b111...92921afb1405 [13:48:07] [url] Comparing d73de012b111...92921afb1405 · miraheze/puppet · GitHub | github.com [13:48:08] [02miraheze/puppet] 07paladox 0392921af - reports: upgrade to v3 [13:50:05] [02TSPortal] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/TSPortal [13:50:05] [url] Page not found · GitHub · GitHub | github.com [13:50:06] [02miraheze/TSPortal] 07paladox deleted branch 03paladox-patch-3 [13:50:08] [02miraheze/TSPortal] 07paladox deleted branch 03paladox-patch-1 [13:50:09] [02TSPortal] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/TSPortal [13:50:10] [url] Page not found · GitHub · GitHub | github.com [13:52:15] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.93, 9.98, 10.03 [13:53:04] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.64, 9.99, 9.84 [14:00:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [14:02:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.39, 11.08, 10.45 [14:06:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.57, 9.86, 10.10 [14:07:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.47, 10.37, 9.76 [14:09:19]