[00:00:12] miraheze/mw-config - Universal-Omega the build passed. [00:00:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.54, 10.54, 9.95 [00:00:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.09, 11.64, 10.48 [00:02:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.60, 2.93, 3.34 [00:02:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 12.00, 11.92, 10.74 [00:03:18] [02mw-config] 07Universal-Omega edited pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [00:03:18] [url] Page not found · GitHub · GitHub | github.com [00:03:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.28, 3.50, 3.76 [00:04:07] RECOVERY - cp30 Disk Space on cp30 is OK: DISK OK - free space: / 7824 MB (20% inode=96%); [00:04:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.98, 11.04, 10.39 [00:04:33] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.86, 2.20, 1.71 [00:06:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.46, 4.04, 3.72 [00:08:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.04, 11.30, 10.74 [00:09:18] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:09:19] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:09:35] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:10:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.24, 10.85, 10.41 [00:10:32] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.64, 1.54, 1.61 [00:10:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.65, 11.55, 10.91 [00:11:17] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.434 second response time [00:11:17] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.017 second response time [00:11:35] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.876 second response time [00:11:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.12, 3.63, 3.84 [00:12:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.54, 9.90, 10.11 [00:13:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.73, 1.70, 1.40 [00:14:02] [02mw-config] 07Universal-Omega edited pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [00:14:03] [url] Page not found · GitHub · GitHub | github.com [00:14:24] [02mw-config] 07Universal-Omega edited pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [00:14:25] ... [00:14:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.34, 11.57, 11.04 [00:15:18] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.07, 1.56, 1.39 [00:16:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.79, 10.65, 10.32 [00:16:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.76, 3.82 [00:16:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.42, 11.55, 11.11 [00:17:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.47, 1.83, 1.24 [00:18:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.58, 9.66, 9.99 [00:19:14] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.65, 2.18, 1.68 [00:19:34] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.94, 1.83, 1.31 [00:21:34] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.12, 1.63, 1.31 [00:22:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.77, 11.27, 11.00 [00:23:08] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.36, 1.87, 1.68 [00:23:32] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.56, 2.07, 1.77 [00:23:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.29, 2.72, 3.31 [00:25:05] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.71, 1.51, 1.57 [00:25:32] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.27, 1.79, 1.71 [00:26:07] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [00:27:32] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.87, 1.47, 1.60 [00:27:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.31, 3.62, 3.53 [00:28:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.25, 3.58, 3.54 [00:28:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.79, 11.82, 11.38 [00:29:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.34, 10.05, 8.90 [00:29:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.88, 3.65 [00:30:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.66, 3.14, 3.38 [00:30:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 12.00, 11.50 [00:31:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.93, 9.87, 8.97 [00:32:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.62, 10.95, 11.16 [00:34:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.67, 10.08, 8.80 [00:34:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.22, 11.33, 11.26 [00:35:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.29, 3.22, 3.86 [00:36:03] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.15, 9.28, 8.69 [00:37:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.81, 10.56, 10.17 [00:37:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.62, 3.85, 3.64 [00:38:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.46, 11.07, 11.24 [00:39:27] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 10.10, 9.91, 9.95 [00:39:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.64, 3.54, 3.53 [00:41:07] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [00:42:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.60, 10.30, 8.77 [00:43:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.47, 4.43, 3.82 [00:43:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.40, 4.36, 4.05 [00:44:28] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.78, 9.44, 8.66 [00:45:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.71, 9.96, 9.90 [00:47:06] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.71, 9.82, 9.88 [00:47:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.55, 3.60 [00:49:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.76, 3.66 [00:51:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.72, 3.32, 3.51 [00:52:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+44/-0/±0] 13https://github.com/miraheze/mw-config/compare/52c2fc49b664...20f562ff8a23 [00:52:09] [url] Comparing 52c2fc49b664...20f562ff8a23 · miraheze/mw-config · GitHub | github.com [00:52:10] [02miraheze/mw-config] 07Universal-Omega 0320f562f - Add composer dependencies [00:52:11] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [00:52:12] [url] Page not found · GitHub · GitHub | github.com [00:52:44] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±12] 13https://github.com/miraheze/mw-config/compare/20f562ff8a23...ef90f3a54208 [00:52:45] [url] Comparing 20f562ff8a23...ef90f3a54208 · miraheze/mw-config · GitHub | github.com [00:52:46] [02miraheze/mw-config] 07github-actions 03ef90f3a - CI: lint code to MediaWiki standards [00:52:47] [02mw-config] 07github-actions[bot] synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [00:52:48] [url] Page not found · GitHub · GitHub | github.com [00:53:21] miraheze/mw-config - Universal-Omega the build passed. [00:53:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.23, 3.05, 3.39 [00:54:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.85, 11.96, 11.16 [00:54:42] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.87, 11.47, 10.45 [00:55:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.26, 3.49, 3.92 [00:56:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.57, 10.55, 10.23 [00:57:01] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 314 bytes in 0.026 second response time [00:58:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.52, 10.00, 10.08 [00:58:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.39, 11.23, 11.11 [00:59:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.26, 4.14, 3.72 [00:59:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.52, 4.60, 4.22 [01:01:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.68, 3.60 [01:01:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.88, 3.57, 3.31 [01:03:01] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 32409 bytes in 0.760 second response time [01:03:57] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.77, 3.20, 3.19 [01:04:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.92, 10.04, 9.06 [01:04:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.98, 11.58, 11.28 [01:05:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/09a3b6c844df...211c854e7a6d [01:05:21] [url] Comparing 09a3b6c844df...211c854e7a6d · miraheze/mw-config · GitHub | github.com [01:05:21] [02miraheze/mw-config] 07Universal-Omega 03211c854 - Update .gitignore [01:05:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.23, 3.62, 3.56 [01:06:03] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.87, 9.32, 8.92 [01:06:33] miraheze/mw-config - Universal-Omega the build passed. [01:07:01] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 314 bytes in 0.026 second response time [01:07:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.97, 3.68, 3.59 [01:08:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.61, 10.78, 11.09 [01:09:02] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 32409 bytes in 1.012 second response time [01:09:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/211c854e7a6d...040e6825045f [01:09:25] [url] Comparing 211c854e7a6d...040e6825045f · miraheze/mw-config · GitHub | github.com [01:09:26] [02miraheze/mw-config] 07Universal-Omega 03040e682 - Update .gitignore [01:09:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ef90f3a54208...a2b3cde2ba26 [01:09:40] [url] Comparing ef90f3a54208...a2b3cde2ba26 · miraheze/mw-config · GitHub | github.com [01:09:41] [02miraheze/mw-config] 07Universal-Omega 03a2b3cde - Update .gitignore [01:09:42] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:09:43] [url] Page not found · GitHub · GitHub | github.com [01:10:14] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-6 [+2999/-0/±1] 13https://github.com/miraheze/mw-config/compare/a2b3cde2ba26...e2c554a7ee9d [01:10:14] [url] Comparing a2b3cde2ba26...e2c554a7ee9d · miraheze/mw-config · GitHub | github.com [01:10:15] [02miraheze/mw-config] 07github-actions 03e2c554a - CI: lint code to MediaWiki standards [01:10:17] [02mw-config] 07github-actions[bot] synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:10:17] [url] Page not found · GitHub · GitHub | github.com [01:10:31] miraheze/mw-config - Universal-Omega the build passed. [01:10:51] miraheze/mw-config - Universal-Omega the build passed. [01:10:54] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.01, 10.95, 9.62 [01:11:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.19, 3.17, 3.39 [01:12:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.91, 11.01, 11.03 [01:12:52] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.55, 9.82, 9.38 [01:13:01] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 314 bytes in 0.024 second response time [01:15:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.96, 11.30, 10.51 [01:15:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.65, 3.33, 3.39 [01:16:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.78, 10.45, 10.90 [01:16:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e2c554a7ee9d...90e839528b0b [01:16:41] [url] Comparing e2c554a7ee9d...90e839528b0b · miraheze/mw-config · GitHub | github.com [01:16:42] [02miraheze/mw-config] 07Universal-Omega 0390e8395 - Update .gitignore [01:16:44] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:16:44] [url] Page not found · GitHub · GitHub | github.com [01:16:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.18, 10.83, 10.43 [01:17:02] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 32410 bytes in 1.043 second response time [01:17:13] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±4] 13https://github.com/miraheze/mw-config/compare/90e839528b0b...4f757f2dc0ae [01:17:14] [url] Comparing 90e839528b0b...4f757f2dc0ae · miraheze/mw-config · GitHub | github.com [01:17:15] [02miraheze/mw-config] 07github-actions 034f757f2 - CI: lint code to MediaWiki standards [01:17:16] [02mw-config] 07github-actions[bot] synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:17:17] [url] Page not found · GitHub · GitHub | github.com [01:17:52] miraheze/mw-config - Universal-Omega the build passed. [01:17:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.89, 3.80, 3.55 [01:19:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.17, 4.36, 3.57 [01:20:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-2999/±1] 13https://github.com/miraheze/mw-config/compare/4f757f2dc0ae...2aa0f848ec3d [01:20:20] [url] Comparing 4f757f2dc0ae...2aa0f848ec3d · miraheze/mw-config · GitHub | github.com [01:20:21] [02miraheze/mw-config] 07Universal-Omega 032aa0f84 - Revert "CI: lint code to MediaWiki standards" [01:20:23] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:20:23] [url] Page not found · GitHub · GitHub | github.com [01:20:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.49, 2.28, 1.56 [01:20:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.76, 11.25, 11.06 [01:20:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.09, 11.57, 10.79 [01:21:10] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/2aa0f848ec3d...55f63519cb1d [01:21:11] [url] Comparing 2aa0f848ec3d...55f63519cb1d · miraheze/mw-config · GitHub | github.com [01:21:11] [02miraheze/mw-config] 07github-actions 0355f6351 - CI: lint code to MediaWiki standards [01:21:13] [02mw-config] 07github-actions[bot] synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:21:13] [url] Page not found · GitHub · GitHub | github.com [01:21:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.81, 3.46 [01:21:47] miraheze/mw-config - Universal-Omega the build passed. [01:21:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.89, 3.81, 3.65 [01:22:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.74, 2.00, 1.54 [01:22:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.35, 10.87, 10.64 [01:23:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.26, 3.33, 3.33 [01:24:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.41, 11.69, 11.31 [01:24:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/55f63519cb1d...eb377df40284 [01:24:48] [url] Comparing 55f63519cb1d...eb377df40284 · miraheze/mw-config · GitHub | github.com [01:24:49] [02miraheze/mw-config] 07Universal-Omega 03eb377df - Update autoload_files.php [01:24:50] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:24:51] [url] Page not found · GitHub · GitHub | github.com [01:25:01] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 314 bytes in 0.019 second response time [01:25:24] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/eb377df40284...7f2c1229a514 [01:25:24] [url] Comparing eb377df40284...7f2c1229a514 · miraheze/mw-config · GitHub | github.com [01:25:25] [02miraheze/mw-config] 07github-actions 037f2c122 - CI: lint code to MediaWiki standards [01:25:26] [02mw-config] 07github-actions[bot] synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:25:27] [url] Page not found · GitHub · GitHub | github.com [01:25:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.28, 2.90, 3.31 [01:26:00] miraheze/mw-config - Universal-Omega the build passed. [01:26:29] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.91, 1.44, 1.42 [01:29:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.83, 3.62, 3.58 [01:32:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-1/±12] 13https://github.com/miraheze/mw-config/compare/7f2c1229a514...7d96ed82e2f6 [01:32:04] [url] Comparing 7f2c1229a514...7d96ed82e2f6 · miraheze/mw-config · GitHub | github.com [01:32:05] [02miraheze/mw-config] 07Universal-Omega 037d96ed8 - . [01:32:06] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:32:07] [url] Page not found · GitHub · GitHub | github.com [01:32:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.26, 11.89, 11.44 [01:32:47] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±12] 13https://github.com/miraheze/mw-config/compare/7d96ed82e2f6...57d6b27bb3ad [01:32:47] [url] Comparing 7d96ed82e2f6...57d6b27bb3ad · miraheze/mw-config · GitHub | github.com [01:32:48] [02miraheze/mw-config] 07github-actions 0357d6b27 - CI: lint code to MediaWiki standards [01:32:50] [02mw-config] 07github-actions[bot] synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:32:50] [url] Page not found · GitHub · GitHub | github.com [01:33:25] miraheze/mw-config - Universal-Omega the build passed. [01:33:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.78, 3.65 [01:34:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.54, 11.60, 11.38 [01:35:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.40, 3.56, 3.58 [01:36:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 2.00, 1.83, 1.57 [01:37:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.91, 3.92, 3.55 [01:37:50] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/57d6b27bb3ad...0b67a0034d1e [01:37:51] [url] Comparing 57d6b27bb3ad...0b67a0034d1e · miraheze/mw-config · GitHub | github.com [01:37:52] [02miraheze/mw-config] 07Universal-Omega 030b67a00 - Update .gitignore [01:37:53] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:37:54] [url] Page not found · GitHub · GitHub | github.com [01:38:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.15, 11.24, 10.76 [01:38:26] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0b67a0034d1e...1bf1f681283f [01:38:27] [url] Comparing 0b67a0034d1e...1bf1f681283f · miraheze/mw-config · GitHub | github.com [01:38:28] [02miraheze/mw-config] 07github-actions 031bf1f68 - CI: lint code to MediaWiki standards [01:38:29] [02mw-config] 07github-actions[bot] synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:38:30] [url] Page not found · GitHub · GitHub | github.com [01:39:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.76, 3.54 [01:39:09] miraheze/mw-config - Universal-Omega the build passed. [01:39:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.60, 9.35, 8.17 [01:39:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-43/±0] 13https://github.com/miraheze/mw-config/compare/1bf1f681283f...f2e8b3aeacec [01:39:53] [url] Comparing 1bf1f681283f...f2e8b3aeacec · miraheze/mw-config · GitHub | github.com [01:39:54] [02miraheze/mw-config] 07Universal-Omega 03f2e8b3a - Delete vendor directory [01:39:55] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:39:56] [url] Page not found · GitHub · GitHub | github.com [01:40:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.19, 10.58, 10.59 [01:40:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f2e8b3aeacec...fd571f25c21d [01:40:16] [url] Comparing f2e8b3aeacec...fd571f25c21d · miraheze/mw-config · GitHub | github.com [01:40:17] [02miraheze/mw-config] 07Universal-Omega 03fd571f2 - Update .gitignore [01:40:19] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:40:19] [url] Page not found · GitHub · GitHub | github.com [01:41:05] miraheze/mw-config - Universal-Omega the build passed. [01:41:06] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [01:41:21] miraheze/mw-config - Universal-Omega the build passed. [01:41:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.98, 9.02, 8.19 [01:42:29] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.25, 1.86, 1.67 [01:44:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.85, 9.55, 10.17 [01:44:14] [02miraheze/mw-config] 07Universal-Omega pushed 034 commits to 03Universal-Omega-patch-6 [+43/-0/±5] 13https://github.com/miraheze/mw-config/compare/fd571f25c21d...ce9ba58a40da [01:44:15] [url] Comparing fd571f25c21d...ce9ba58a40da · miraheze/mw-config · GitHub | github.com [01:44:16] [02miraheze/mw-config] 07Universal-Omega 032a99942 - Merge branch 'Universal-Omega-patch-6' of https://github.com/miraheze/mw-config into Universal-Omega-patch-6 [01:44:16] [url] GitHub - miraheze/mw-config: Production MediaWiki configuration | github.com [01:44:17] [02miraheze/mw-config] 07Universal-Omega 03ce9ba58 - . [01:44:19] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:44:19] [url] Page not found · GitHub · GitHub | github.com [01:44:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.72, 10.12, 9.26 [01:44:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.73, 2.91, 3.26 [01:44:57] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±12] 13https://github.com/miraheze/mw-config/compare/ce9ba58a40da...8b67623a5668 [01:44:58] [url] Comparing ce9ba58a40da...8b67623a5668 · miraheze/mw-config · GitHub | github.com [01:44:59] [02miraheze/mw-config] 07github-actions 038b67623 - CI: lint code to MediaWiki standards [01:45:00] [02mw-config] 07github-actions[bot] synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:45:00] [url] Page not found · GitHub · GitHub | github.com [01:45:35] miraheze/mw-config - Universal-Omega the build passed. [01:45:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.43, 2.85, 3.26 [01:46:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.45, 9.46, 9.13 [01:46:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.73, 1.93, 1.76 [01:46:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.42, 11.29, 11.25 [01:48:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.26, 11.89, 11.01 [01:48:29] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.01, 1.59, 1.66 [01:49:03] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.34, 2.55, 1.77 [01:50:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.41, 11.52, 11.00 [01:50:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8b67623a5668...3e0fb3812ffd [01:50:59] [url] Comparing 8b67623a5668...3e0fb3812ffd · miraheze/mw-config · GitHub | github.com [01:51:00] [02miraheze/mw-config] 07Universal-Omega 033e0fb38 - Update continuousIntegration.yml [01:51:01] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:51:02] [url] Page not found · GitHub · GitHub | github.com [01:51:30] [02miraheze/mw-config] 07Universal-Omega pushed 032 commits to 03Universal-Omega-patch-6 [+0/-0/±13] 13https://github.com/miraheze/mw-config/compare/3e0fb3812ffd...62a1d86869a8 [01:51:30] [url] Comparing 3e0fb3812ffd...62a1d86869a8 · miraheze/mw-config · GitHub | github.com [01:51:31] [02miraheze/mw-config] 07Universal-Omega 0302dbfee - Revert "CI: lint code to MediaWiki standards" [01:51:33] [02miraheze/mw-config] 07Universal-Omega 0362a1d86 - Merge branch 'Universal-Omega-patch-6' of https://github.com/miraheze/mw-config into Universal-Omega-patch-6 [01:51:33] [url] GitHub - miraheze/mw-config: Production MediaWiki configuration | github.com [01:51:34] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:51:35] [url] Page not found · GitHub · GitHub | github.com [01:52:11] miraheze/mw-config - Universal-Omega the build passed. [01:52:29] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.15, 2.08, 1.85 [01:52:37] miraheze/mw-config - Universal-Omega the build passed. [01:53:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/62a1d86869a8...bb07ba2c0240 [01:53:59] [url] Comparing 62a1d86869a8...bb07ba2c0240 · miraheze/mw-config · GitHub | github.com [01:54:00] [02miraheze/mw-config] 07Universal-Omega 03bb07ba2 - Update .gitignore [01:54:01] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [01:54:02] [url] Page not found · GitHub · GitHub | github.com [01:54:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.53, 11.85, 11.20 [01:54:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.59, 1.83, 1.78 [01:54:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.77, 3.38, 3.27 [01:55:05] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 32561 bytes in 4.442 second response time [01:55:19] miraheze/mw-config - Universal-Omega the build passed. [01:55:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.90, 9.93, 9.15 [01:56:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.42, 11.77, 11.55 [01:57:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.90, 9.31, 9.01 [01:58:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.33, 3.28 [01:59:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.89, 3.55 [02:00:08] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.29, 10.08, 8.83 [02:00:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.14, 11.10, 11.19 [02:00:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.79, 9.32, 8.23 [02:00:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.03, 1.38, 1.62 [02:01:03] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.19, 1.88, 1.87 [02:01:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.89, 3.20, 3.34 [02:01:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.23, 3.37, 3.99 [02:02:05] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.59, 9.66, 8.82 [02:02:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.38, 8.98, 8.24 [02:02:51] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.92, 6.64, 5.53 [02:03:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.43, 3.92 [02:04:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.11, 11.29, 11.16 [02:04:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.50, 11.01, 11.11 [02:04:51] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.65, 6.01, 5.45 [02:06:33] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:07:03] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.47, 2.67, 2.12 [02:07:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.38, 3.91, 3.57 [02:08:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.72, 11.86, 11.46 [02:08:37] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 5.586 second response time [02:10:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.38, 11.95, 11.56 [02:11:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.09, 10.34, 9.65 [02:11:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.27, 3.73, 3.58 [02:12:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.03, 10.26, 9.00 [02:12:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.25, 9.69, 8.80 [02:12:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.26, 3.66, 3.39 [02:12:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.71, 11.96, 11.59 [02:13:03] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.89, 1.74, 1.90 [02:13:08] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:13:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.70, 10.80, 9.90 [02:13:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.92, 3.67 [02:14:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.96, 9.93, 9.02 [02:14:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.49, 9.40, 8.82 [02:14:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.57, 3.40 [02:14:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.84, 11.61, 11.52 [02:15:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.10, 10.11, 9.76 [02:16:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.58, 11.16, 11.06 [02:16:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.77, 3.39, 3.36 [02:16:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.23, 11.71, 11.56 [02:17:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.30, 3.46 [02:17:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.67, 3.48, 3.94 [02:18:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.87, 11.83, 11.63 [02:19:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.52, 3.94, 3.69 [02:19:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.77, 4.33, 4.22 [02:20:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.71, 3.49 [02:20:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bb07ba2c0240...63687dd49472 [02:20:32] [url] Comparing bb07ba2c0240...63687dd49472 · miraheze/mw-config · GitHub | github.com [02:20:33] [02miraheze/mw-config] 07Universal-Omega 0363687dd - Update MirahezeFunctions.php [02:20:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.53, 12.01, 11.69 [02:20:35] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [02:20:35] [url] Page not found · GitHub · GitHub | github.com [02:21:03] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.85, 1.21, 1.59 [02:21:49] miraheze/mw-config - Universal-Omega the build passed. [02:21:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.36, 3.58, 3.58 [02:22:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.00, 3.39, 3.39 [02:24:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.76, 11.54, 11.60 [02:25:16] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw112 [02:25:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.47, 2.92, 3.30 [02:25:55] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw112 [02:26:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 2.65, 8.98, 10.75 [02:26:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.85, 9.86, 9.08 [02:27:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [02:27:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.51, 10.46, 9.78 [02:27:55] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [02:28:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.03, 9.00, 8.88 [02:28:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.19, 11.24, 11.41 [02:28:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/040e6825045f...04b5839acda1 [02:28:58] [url] Comparing 040e6825045f...04b5839acda1 · miraheze/mw-config · GitHub | github.com [02:28:59] [02miraheze/mw-config] 07Universal-Omega 0304b5839 - Fix tabbing [02:29:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.76, 9.22, 9.41 [02:29:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.83, 4.00 [02:30:07] miraheze/mw-config - Universal-Omega the build passed. [02:30:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.67, 11.28, 11.40 [02:31:17] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:31:34] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:31:39] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:14] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 11.67, 11.54 [02:33:19] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 4.569 second response time [02:33:37] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 4.478 second response time [02:33:44] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 5.973 second response time [02:34:19] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 6.609 second response time [02:35:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.67, 3.81 [02:37:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.60, 3.78 [02:38:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.40, 9.09, 10.01 [02:38:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.43, 10.74, 11.32 [02:39:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.46, 9.50, 9.13 [02:41:34] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.28, 0.80 [02:41:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.98, 8.96, 8.99 [02:41:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.04, 2.77, 3.40 [02:42:16] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [02:42:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.61, 12.07, 11.71 [02:44:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.22, 11.41, 11.52 [02:45:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.40, 3.55 [02:47:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.29, 9.63, 8.76 [02:47:34] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.15, 1.78, 1.17 [02:47:54] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.00, 2.99, 3.39 [02:49:08] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 10.06, 9.75, 8.92 [02:49:34] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.69, 1.70, 1.21 [02:49:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.25, 10.70, 9.72 [02:50:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.75, 11.00, 11.13 [02:52:02] !log [@mwtask111] starting deploy of {'config': True} to all [02:52:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.20, 11.13, 10.28 [02:52:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:21] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 19s [02:52:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.78, 11.11, 11.18 [02:52:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:53:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.61, 10.13, 9.71 [02:54:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.60, 10.83, 10.30 [02:56:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.21, 11.23, 10.50 [02:56:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.16, 11.35, 9.84 [02:57:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.72, 11.25, 10.27 [02:58:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.29, 10.92, 10.46 [02:58:54] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.68, 10.14, 9.57 [03:00:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.13, 12.15, 10.97 [03:00:26] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 8.72, 2.98, 1.74 [03:00:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.06, 11.64, 11.27 [03:01:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.51, 9.45, 9.77 [03:03:12] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.99, 3.60, 3.29 [03:04:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.75, 11.80, 11.14 [03:05:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.71, 3.30, 3.22 [03:06:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.69, 1.99, 1.68 [03:08:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.95, 1.57, 1.56 [03:08:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.04, 11.79, 11.68 [03:09:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.98, 1.83, 1.59 [03:10:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.67, 11.21, 11.03 [03:12:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.96, 10.66, 10.87 [03:14:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.29, 10.14, 9.43 [03:15:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 3.86, 3.51 [03:16:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.06, 9.18, 9.16 [03:17:05] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.85, 1.67 [03:18:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.13, 10.32, 10.51 [03:18:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.20, 10.69, 10.96 [03:19:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.84, 1.69 [03:20:04] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.86, 3.30, 2.06 [03:20:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.84, 10.35, 10.49 [03:20:18] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.60, 3.88, 3.47 [03:20:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.05, 2.13, 1.66 [03:20:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.50, 10.21, 10.75 [03:21:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 9.60, 5.68, 3.97 [03:22:16] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.75, 10.55, 10.06 [03:22:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 10.72, 10.86 [03:23:03] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:24:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.83, 11.38, 10.84 [03:24:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.27, 1.96, 1.72 [03:25:08] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 6.122 second response time [03:26:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.16, 11.16, 10.39 [03:26:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.27, 10.61, 10.63 [03:26:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.26, 11.04, 11.01 [03:27:52] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.94, 1.88, 1.92 [03:28:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.44, 11.06, 10.45 [03:28:24] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.22, 2.09, 1.82 [03:28:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.22, 11.19, 11.04 [03:30:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.50, 11.40, 10.88 [03:30:24] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.37, 1.82, 1.76 [03:31:21] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:31:34] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:31:47] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:31:53] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:32:03] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:32:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.73, 10.72, 10.72 [03:32:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.02, 11.28, 11.18 [03:32:48] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.18, 1.90, 1.79 [03:33:23] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 4.441 second response time [03:33:42] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.77, 1.24, 1.61 [03:34:09] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 7.921 second response time [03:34:50] PROBLEM - wiki.ssangyongsports.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.ssangyongsports.eu.org All nameservers failed to answer the query. [03:35:42] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.019 second response time [03:35:55] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 1.851 second response time [03:36:02] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.376 second response time [03:36:29] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.50, 1.67 [03:37:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.13, 11.92, 11.21 [03:38:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.14, 10.51, 10.55 [03:39:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.43, 11.02, 10.98 [03:40:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.89, 11.02, 10.74 [03:40:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.13, 11.20, 11.10 [03:42:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.16, 10.85, 10.99 [03:44:35] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.95, 1.91 [03:46:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.85, 10.76, 10.60 [03:46:33] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.37, 2.07, 1.96 [03:46:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.03, 10.78, 10.87 [03:48:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.93, 11.43, 10.89 [03:48:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.91, 1.99, 1.94 [03:48:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.68, 10.84, 10.89 [03:50:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.22, 11.18, 10.99 [03:51:58] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.61, 7.19, 6.15 [03:52:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.92, 9.25, 10.13 [03:52:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.75, 1.66, 1.61 [03:52:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.35, 9.84, 9.41 [03:52:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.23, 10.55, 10.77 [03:53:55] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.25, 6.63, 6.09 [03:54:12] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.18, 1.45, 1.54 [03:54:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.76, 8.70, 9.05 [03:54:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.77, 11.39, 11.05 [03:55:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.24, 8.06, 9.67 [03:56:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.96, 10.44, 10.43 [03:56:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.16, 11.07, 10.99 [03:59:47] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.08, 7.18, 6.45 [04:00:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.81, 9.11, 9.92 [04:01:44] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.77, 6.86, 6.41 [04:02:27] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.57, 10.51, 10.17 [04:02:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.90, 2.93, 3.79 [04:02:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 11.25, 11.07 [04:04:02] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Maps-0f26900 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f65addf6ec98 [04:04:03] [02miraheze/mediawiki] 07dependabot[bot] 03f65addf - Bump extensions/Maps from `dbc0597` to `0f26900` [04:04:05] [02mediawiki] 07dependabot[bot] opened pull request 03#7627: Bump extensions/Maps from `dbc0597` to `0f26900` - 13https://github.com/miraheze/mediawiki/pull/7627 [04:04:05] [url] Page not found · GitHub · GitHub | github.com [04:04:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Maps-0f26900 - 13https://github.com/miraheze/mediawiki [04:04:07] [url] Page not found · GitHub · GitHub | github.com [04:04:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Citizen-eb0147e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7b5111752fce [04:04:10] [02miraheze/mediawiki] 07dependabot[bot] 037b51117 - Bump skins/Citizen from `4f64564` to `eb0147e` [04:04:11] [02mediawiki] 07dependabot[bot] opened pull request 03#7628: Bump skins/Citizen from `4f64564` to `eb0147e` - 13https://github.com/miraheze/mediawiki/pull/7628 [04:04:11] ... [04:04:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Citizen-eb0147e - 13https://github.com/miraheze/mediawiki [04:04:13] [url] Page not found · GitHub · GitHub | github.com [04:04:14] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/MonoBook-b77a569 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e81f7a6b3cad [04:04:15] [02miraheze/mediawiki] 07dependabot[bot] 03e81f7a6 - Bump skins/MonoBook from `a2b96fd` to `b77a569` [04:04:17] [02mediawiki] 07dependabot[bot] opened pull request 03#7629: Bump skins/MonoBook from `a2b96fd` to `b77a569` - 13https://github.com/miraheze/mediawiki/pull/7629 [04:04:17] ... [04:04:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/MonoBook-b77a569 - 13https://github.com/miraheze/mediawiki [04:04:19] [url] Page not found · GitHub · GitHub | github.com [04:04:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Translate-3f1b147 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a9c8ff00b93a [04:04:23] [02miraheze/mediawiki] 07dependabot[bot] 03a9c8ff0 - Bump extensions/Translate from `f182386` to `3f1b147` [04:04:25] RECOVERY - wiki.ssangyongsports.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.ssangyongsports.eu.org reverse DNS resolves to cp21.miraheze.org - CNAME OK [04:04:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Translate-3f1b147 - 13https://github.com/miraheze/mediawiki [04:04:25] ... [04:04:26] [02mediawiki] 07dependabot[bot] opened pull request 03#7630: Bump extensions/Translate from `f182386` to `3f1b147` - 13https://github.com/miraheze/mediawiki/pull/7630 [04:04:27] [url] Page not found · GitHub · GitHub | github.com [04:04:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.62, 3.38, 3.84 [04:04:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Monaco-90b5374 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fde7c49f4a75 [04:04:29] [02miraheze/mediawiki] 07dependabot[bot] 03fde7c49 - Bump skins/Monaco from `98817c4` to `90b5374` [04:04:31] [02mediawiki] 07dependabot[bot] opened pull request 03#7631: Bump skins/Monaco from `98817c4` to `90b5374` - 13https://github.com/miraheze/mediawiki/pull/7631 [04:04:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Monaco-90b5374 - 13https://github.com/miraheze/mediawiki [04:04:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.17, 11.08, 11.02 [04:04:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#7627: Bump extensions/Maps from `dbc0597` to `0f26900` - 13https://github.com/miraheze/mediawiki/pull/7627 [04:04:35] [02mediawiki] 07github-actions[bot] labeled pull request 03#7627: Bump extensions/Maps from `dbc0597` to `0f26900` - 13https://github.com/miraheze/mediawiki/pull/7627 [04:04:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#7627: Bump extensions/Maps from `dbc0597` to `0f26900` - 13https://github.com/miraheze/mediawiki/pull/7627 [04:04:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/TwoColConflict-3628089 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b52db9b586f2 [04:04:40] [02miraheze/mediawiki] 07dependabot[bot] 03b52db9b - Bump extensions/TwoColConflict from `27c8248` to `3628089` [04:04:41] [02mediawiki] 07dependabot[bot] opened pull request 03#7632: Bump extensions/TwoColConflict from `27c8248` to `3628089` - 13https://github.com/miraheze/mediawiki/pull/7632 [04:04:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/TwoColConflict-3628089 - 13https://github.com/miraheze/mediawiki [04:04:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Cargo-876d2fa [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5274736c599e [04:04:46] [02miraheze/mediawiki] 07dependabot[bot] 035274736 - Bump extensions/Cargo from `751b633` to `876d2fa` [04:04:48] [02mediawiki] 07dependabot[bot] opened pull request 03#7633: Bump extensions/Cargo from `751b633` to `876d2fa` - 13https://github.com/miraheze/mediawiki/pull/7633 [04:04:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Cargo-876d2fa - 13https://github.com/miraheze/mediawiki [04:04:50] [02mediawiki] 07github-actions[bot] labeled pull request 03#7628: Bump skins/Citizen from `4f64564` to `eb0147e` - 13https://github.com/miraheze/mediawiki/pull/7628 [04:04:52] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ManageWiki-4771142 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/32fecdaba458 [04:04:53] [02miraheze/mediawiki] 07dependabot[bot] 0332fecda - Bump extensions/ManageWiki from `65ac9c5` to `4771142` [04:04:55] [02mediawiki] 07dependabot[bot] opened pull request 03#7634: Bump extensions/ManageWiki from `65ac9c5` to `4771142` - 13https://github.com/miraheze/mediawiki/pull/7634 [04:04:56] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ManageWiki-4771142 - 13https://github.com/miraheze/mediawiki [04:04:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#7631: Bump skins/Monaco from `98817c4` to `90b5374` - 13https://github.com/miraheze/mediawiki/pull/7631 [04:04:59] [02mediawiki] 07github-actions[bot] labeled pull request 03#7632: Bump extensions/TwoColConflict from `27c8248` to `3628089` - 13https://github.com/miraheze/mediawiki/pull/7632 [04:05:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#7630: Bump extensions/Translate from `f182386` to `3f1b147` - 13https://github.com/miraheze/mediawiki/pull/7630 [04:05:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#7633: Bump extensions/Cargo from `751b633` to `876d2fa` - 13https://github.com/miraheze/mediawiki/pull/7633 [04:05:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#7634: Bump extensions/ManageWiki from `65ac9c5` to `4771142` - 13https://github.com/miraheze/mediawiki/pull/7634 [04:05:37] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.12, 6.78, 6.53 [04:07:54] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/755112725a57...48f568b69c81 [04:07:54] [url] Comparing 755112725a57...48f568b69c81 · miraheze/mediawiki · GitHub | github.com [04:07:55] [02miraheze/mediawiki] 07Universal-Omega 0348f568b - labeler: Add .mustache to codechange [04:08:11] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.32, 11.19, 10.50 [04:08:12] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.33, 1.95, 1.87 [04:08:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.43, 3.78 [04:08:42] [02mediawiki] 07Universal-Omega commented on pull request 03#7629: Bump skins/MonoBook from `a2b96fd` to `b77a569` - 13https://github.com/miraheze/mediawiki/pull/7629#issuecomment-1155960497 [04:08:42] [url] Bump skins/MonoBook from `a2b96fd` to `b77a569` by dependabot[bot] · Pull Request #7629 · miraheze/mediawiki · GitHub | github.com [04:08:44] [02mediawiki] 07dependabot[bot] edited pull request 03#7629: Bump skins/MonoBook from `a2b96fd` to `b77a569` - 13https://github.com/miraheze/mediawiki/pull/7629 [04:08:44] [url] Page not found · GitHub · GitHub | github.com [04:09:28] [02mediawiki] 07dependabot[bot] edited pull request 03#7629: Bump skins/MonoBook from `a2b96fd` to `b77a569` - 13https://github.com/miraheze/mediawiki/pull/7629 [04:09:29] [url] Page not found · GitHub · GitHub | github.com [04:09:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/MonoBook-b77a569 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/e81f7a6b3cad...879688399b09 [04:09:30] [url] Comparing e81f7a6b3cad...879688399b09 · miraheze/mediawiki · GitHub | github.com [04:09:31] [02miraheze/mediawiki] 07dependabot[bot] 038796883 - Bump skins/MonoBook from `a2b96fd` to `b77a569` [04:09:33] [02mediawiki] 07dependabot[bot] synchronize pull request 03#7629: Bump skins/MonoBook from `a2b96fd` to `b77a569` - 13https://github.com/miraheze/mediawiki/pull/7629 [04:09:33] [url] Page not found · GitHub · GitHub | github.com [04:09:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.87, 3.56, 3.90 [04:09:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#7629: Bump skins/MonoBook from `a2b96fd` to `b77a569` - 13https://github.com/miraheze/mediawiki/pull/7629 [04:09:58] [url] Page not found · GitHub · GitHub | github.com [04:10:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.56, 10.82, 10.46 [04:10:10] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.58, 1.80, 1.82 [04:10:28] [02mediawiki] 07Universal-Omega labeled pull request 03#7622: Bump extensions/QuickInstantCommons from `14a7753` to `b07cab1` - 13https://github.com/miraheze/mediawiki/pull/7622 [04:10:28] ... [04:10:29] [02mediawiki] 07Universal-Omega labeled pull request 03#7623: Bump extensions/SocialProfile from `932e4d9` to `280eb97` - 13https://github.com/miraheze/mediawiki/pull/7623 [04:10:29] [url] Page not found · GitHub · GitHub | github.com [04:10:31] [02mediawiki] 07Universal-Omega labeled pull request 03#7624: Bump extensions/PageForms from `0440bbc` to `7847a4d` - 13https://github.com/miraheze/mediawiki/pull/7624 [04:10:31] ... [04:10:32] [02mediawiki] 07Universal-Omega labeled pull request 03#7627: Bump extensions/Maps from `dbc0597` to `0f26900` - 13https://github.com/miraheze/mediawiki/pull/7627 [04:10:32] ... [04:10:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.77, 10.94, 10.85 [04:10:34] [02mediawiki] 07Universal-Omega labeled pull request 03#7628: Bump skins/Citizen from `4f64564` to `eb0147e` - 13https://github.com/miraheze/mediawiki/pull/7628 [04:10:34] [url] Page not found · GitHub · GitHub | github.com [04:10:35] [02mediawiki] 07Universal-Omega labeled pull request 03#7629: Bump skins/MonoBook from `a2b96fd` to `b77a569` - 13https://github.com/miraheze/mediawiki/pull/7629 [04:10:35] [url] Page not found · GitHub · GitHub | github.com [04:10:37] [02mediawiki] 07Universal-Omega labeled pull request 03#7631: Bump skins/Monaco from `98817c4` to `90b5374` - 13https://github.com/miraheze/mediawiki/pull/7631 [04:10:38] [02mediawiki] 07Universal-Omega labeled pull request 03#7630: Bump extensions/Translate from `f182386` to `3f1b147` - 13https://github.com/miraheze/mediawiki/pull/7630 [04:10:40] [02mediawiki] 07Universal-Omega labeled pull request 03#7632: Bump extensions/TwoColConflict from `27c8248` to `3628089` - 13https://github.com/miraheze/mediawiki/pull/7632 [04:10:41] [02mediawiki] 07Universal-Omega labeled pull request 03#7633: Bump extensions/Cargo from `751b633` to `876d2fa` - 13https://github.com/miraheze/mediawiki/pull/7633 [04:10:42] [02mediawiki] 07Universal-Omega labeled pull request 03#7634: Bump extensions/ManageWiki from `65ac9c5` to `4771142` - 13https://github.com/miraheze/mediawiki/pull/7634 [04:10:44] [02mediawiki] 07github-actions[bot] closed pull request 03#7622: Bump extensions/QuickInstantCommons from `14a7753` to `b07cab1` - 13https://github.com/miraheze/mediawiki/pull/7622 [04:10:46] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/QuickInstantCommons-b07cab1 [04:10:47] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/48f568b69c81...c4022e861a5a [04:10:48] [url] Comparing 48f568b69c81...c4022e861a5a · miraheze/mediawiki · GitHub | github.com [04:10:48] [02miraheze/mediawiki] 07dependabot[bot] 03c4022e8 - Bump extensions/QuickInstantCommons from `14a7753` to `b07cab1` (#7622) [04:10:50] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/QuickInstantCommons-b07cab1 - 13https://github.com/miraheze/mediawiki [04:10:50] [url] Page not found · GitHub · GitHub | github.com [04:10:52] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#7622: Bump extensions/QuickInstantCommons from `14a7753` to `b07cab1` - 13https://github.com/miraheze/mediawiki/pull/7622 [04:10:52] [url] Page not found · GitHub · GitHub | github.com [04:10:53] [02mediawiki] 07github-actions[bot] closed pull request 03#7633: Bump extensions/Cargo from `751b633` to `876d2fa` - 13https://github.com/miraheze/mediawiki/pull/7633 [04:10:53] ... [04:10:55] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c4022e861a5a...6d4887f67fce [04:10:55] [url] Comparing c4022e861a5a...6d4887f67fce · miraheze/mediawiki · GitHub | github.com [04:10:56] [02miraheze/mediawiki] 07dependabot[bot] 036d4887f - Bump extensions/Cargo from `751b633` to `876d2fa` (#7633) [04:10:57] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Cargo-876d2fa [04:10:59] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#7633: Bump extensions/Cargo from `751b633` to `876d2fa` - 13https://github.com/miraheze/mediawiki/pull/7633 [04:10:59] [url] Page not found · GitHub · GitHub | github.com [04:11:01] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Cargo-876d2fa - 13https://github.com/miraheze/mediawiki [04:11:01] ... [04:11:02] [02mediawiki] 07github-actions[bot] closed pull request 03#7628: Bump skins/Citizen from `4f64564` to `eb0147e` - 13https://github.com/miraheze/mediawiki/pull/7628 [04:11:03] [url] Page not found · GitHub · GitHub | github.com [04:11:03] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Citizen-eb0147e [04:11:05] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/6d4887f67fce...e0a35d92ec18 [04:11:06] [url] Comparing 6d4887f67fce...e0a35d92ec18 · miraheze/mediawiki · GitHub | github.com [04:11:06] [02miraheze/mediawiki] 07dependabot[bot] 03e0a35d9 - Bump skins/Citizen from `4f64564` to `eb0147e` (#7628) [04:11:08] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Citizen-eb0147e - 13https://github.com/miraheze/mediawiki [04:11:08] [url] Page not found · GitHub · GitHub | github.com [04:11:09] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#7628: Bump skins/Citizen from `4f64564` to `eb0147e` - 13https://github.com/miraheze/mediawiki/pull/7628 [04:11:10] [url] Page not found · GitHub · GitHub | github.com [04:11:11] [02mediawiki] 07github-actions[bot] closed pull request 03#7631: Bump skins/Monaco from `98817c4` to `90b5374` - 13https://github.com/miraheze/mediawiki/pull/7631 [04:11:12] [url] Page not found · GitHub · GitHub | github.com [04:11:13] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Monaco-90b5374 [04:11:14] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/e0a35d92ec18...f4fd632ae8ed [04:11:15] [url] Comparing e0a35d92ec18...f4fd632ae8ed · miraheze/mediawiki · GitHub | github.com [04:11:16] [02miraheze/mediawiki] 07dependabot[bot] 03f4fd632 - Bump skins/Monaco from `98817c4` to `90b5374` (#7631) [04:11:17] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Monaco-90b5374 - 13https://github.com/miraheze/mediawiki [04:11:17] ... [04:11:18] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#7631: Bump skins/Monaco from `98817c4` to `90b5374` - 13https://github.com/miraheze/mediawiki/pull/7631 [04:11:19] [url] Page not found · GitHub · GitHub | github.com [04:11:20] [02mediawiki] 07github-actions[bot] closed pull request 03#7632: Bump extensions/TwoColConflict from `27c8248` to `3628089` - 13https://github.com/miraheze/mediawiki/pull/7632 [04:11:21] ... [04:11:21] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/f4fd632ae8ed...86c286190b6c [04:11:22] [url] Comparing f4fd632ae8ed...86c286190b6c · miraheze/mediawiki · GitHub | github.com [04:11:23] [02miraheze/mediawiki] 07dependabot[bot] 0386c2861 - Bump extensions/TwoColConflict from `27c8248` to `3628089` (#7632) [04:11:25] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/TwoColConflict-3628089 [04:11:26] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/TwoColConflict-3628089 - 13https://github.com/miraheze/mediawiki [04:11:26] [url] Page not found · GitHub · GitHub | github.com [04:11:27] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#7632: Bump extensions/TwoColConflict from `27c8248` to `3628089` - 13https://github.com/miraheze/mediawiki/pull/7632 [04:11:28] [url] Page not found · GitHub · GitHub | github.com [04:11:29] [02mediawiki] 07github-actions[bot] closed pull request 03#7627: Bump extensions/Maps from `dbc0597` to `0f26900` - 13https://github.com/miraheze/mediawiki/pull/7627 [04:11:30] [url] Page not found · GitHub · GitHub | github.com [04:11:31] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/86c286190b6c...3858d838d352 [04:11:31] [url] Comparing 86c286190b6c...3858d838d352 · miraheze/mediawiki · GitHub | github.com [04:11:32] [02miraheze/mediawiki] 07dependabot[bot] 033858d83 - Bump extensions/Maps from `dbc0597` to `0f26900` (#7627) [04:11:34] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Maps-0f26900 [04:11:35] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Maps-0f26900 - 13https://github.com/miraheze/mediawiki [04:11:35] [url] Page not found · GitHub · GitHub | github.com [04:11:37] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#7627: Bump extensions/Maps from `dbc0597` to `0f26900` - 13https://github.com/miraheze/mediawiki/pull/7627 [04:11:37] ... [04:11:38] [02mediawiki] 07github-actions[bot] closed pull request 03#7629: Bump skins/MonoBook from `a2b96fd` to `b77a569` - 13https://github.com/miraheze/mediawiki/pull/7629 [04:11:39] [url] Page not found · GitHub · GitHub | github.com [04:11:40] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/3858d838d352...93eef5e58391 [04:11:40] [url] Comparing 3858d838d352...93eef5e58391 · miraheze/mediawiki · GitHub | github.com [04:11:41] [02miraheze/mediawiki] 07dependabot[bot] 0393eef5e - Bump skins/MonoBook from `a2b96fd` to `b77a569` (#7629) [04:11:43] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/skins/MonoBook-b77a569 - 13https://github.com/miraheze/mediawiki [04:11:43] ... [04:11:44] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/skins/MonoBook-b77a569 [04:11:46] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#7629: Bump skins/MonoBook from `a2b96fd` to `b77a569` - 13https://github.com/miraheze/mediawiki/pull/7629 [04:11:46] [url] Page not found · GitHub · GitHub | github.com [04:11:47] [02mediawiki] 07github-actions[bot] closed pull request 03#7634: Bump extensions/ManageWiki from `65ac9c5` to `4771142` - 13https://github.com/miraheze/mediawiki/pull/7634 [04:11:48] [url] Page not found · GitHub · GitHub | github.com [04:11:49] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/93eef5e58391...d3643aecf4bf [04:11:49] [url] Comparing 93eef5e58391...d3643aecf4bf · miraheze/mediawiki · GitHub | github.com [04:11:50] [02miraheze/mediawiki] 07dependabot[bot] 03d3643ae - Bump extensions/ManageWiki from `65ac9c5` to `4771142` (#7634) [04:11:52] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ManageWiki-4771142 [04:11:53] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ManageWiki-4771142 - 13https://github.com/miraheze/mediawiki [04:11:53] [url] Page not found · GitHub · GitHub | github.com [04:11:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.43, 3.77, 3.93 [04:11:55] [02mediawiki] 07github-actions[bot] closed pull request 03#7623: Bump extensions/SocialProfile from `932e4d9` to `280eb97` - 13https://github.com/miraheze/mediawiki/pull/7623 [04:11:55] [url] Page not found · GitHub · GitHub | github.com [04:11:56] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#7634: Bump extensions/ManageWiki from `65ac9c5` to `4771142` - 13https://github.com/miraheze/mediawiki/pull/7634 [04:11:56] ... [04:11:58] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/d3643aecf4bf...33dfe29090f5 [04:11:58] [url] Comparing d3643aecf4bf...33dfe29090f5 · miraheze/mediawiki · GitHub | github.com [04:11:59] [02miraheze/mediawiki] 07dependabot[bot] 0333dfe29 - Bump extensions/SocialProfile from `932e4d9` to `280eb97` (#7623) [04:12:01] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/SocialProfile-280eb97 [04:12:02] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/SocialProfile-280eb97 - 13https://github.com/miraheze/mediawiki [04:12:02] [url] Page not found · GitHub · GitHub | github.com [04:12:04] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#7623: Bump extensions/SocialProfile from `932e4d9` to `280eb97` - 13https://github.com/miraheze/mediawiki/pull/7623 [04:12:04] ... [04:12:05] [02mediawiki] 07github-actions[bot] closed pull request 03#7624: Bump extensions/PageForms from `0440bbc` to `7847a4d` - 13https://github.com/miraheze/mediawiki/pull/7624 [04:12:06] [url] Page not found · GitHub · GitHub | github.com [04:12:07] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/33dfe29090f5...4022c744fa37 [04:12:07] [url] Comparing 33dfe29090f5...4022c744fa37 · miraheze/mediawiki · GitHub | github.com [04:12:08] [02miraheze/mediawiki] 07dependabot[bot] 034022c74 - Bump extensions/PageForms from `0440bbc` to `7847a4d` (#7624) [04:12:10] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PageForms-7847a4d [04:12:11] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PageForms-7847a4d - 13https://github.com/miraheze/mediawiki [04:12:11] [url] Page not found · GitHub · GitHub | github.com [04:12:13] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#7624: Bump extensions/PageForms from `0440bbc` to `7847a4d` - 13https://github.com/miraheze/mediawiki/pull/7624 [04:12:13] [url] Page not found · GitHub · GitHub | github.com [04:12:14] [02mediawiki] 07github-actions[bot] closed pull request 03#7630: Bump extensions/Translate from `f182386` to `3f1b147` - 13https://github.com/miraheze/mediawiki/pull/7630 [04:12:14] [url] Page not found · GitHub · GitHub | github.com [04:12:16] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Translate-3f1b147 [04:12:17] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Translate-3f1b147 - 13https://github.com/miraheze/mediawiki [04:12:17] ... [04:12:19] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/4022c744fa37...100d2b74279b [04:12:19] [url] Comparing 4022c744fa37...100d2b74279b · miraheze/mediawiki · GitHub | github.com [04:12:20] [02miraheze/mediawiki] 07dependabot[bot] 03100d2b7 - Bump extensions/Translate from `f182386` to `3f1b147` (#7630) [04:12:22] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#7630: Bump extensions/Translate from `f182386` to `3f1b147` - 13https://github.com/miraheze/mediawiki/pull/7630 [04:12:22] [url] Page not found · GitHub · GitHub | github.com [04:12:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.00, 10.71, 10.76 [04:13:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.44, 3.79 [04:13:55] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.20, 9.46, 10.00 [04:14:01] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/100d2b74279b...547ccd6a7dde [04:14:01] [url] Comparing 100d2b74279b...547ccd6a7dde · miraheze/mediawiki · GitHub | github.com [04:14:02] [02miraheze/mediawiki] 07Universal-Omega 03547ccd6 - Update automerge.yml [04:14:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.00, 9.81, 8.71 [04:16:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.95, 9.67, 8.50 [04:17:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 3.81, 3.86 [04:18:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.65, 9.52, 8.61 [04:18:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.12, 10.64, 10.62 [04:19:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.42, 11.06, 10.47 [04:20:01] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.85, 1.81 [04:20:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.77, 8.85, 8.83 [04:20:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.65, 10.70, 10.65 [04:21:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.78, 10.63, 10.39 [04:21:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.84, 3.88 [04:21:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.61, 1.76, 1.78 [04:23:57] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.90, 1.84 [04:24:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.64, 10.42, 9.53 [04:24:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.13, 11.11, 10.79 [04:25:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.95, 10.02, 10.16 [04:25:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.19, 3.93, 3.85 [04:25:54] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.53, 1.74, 1.78 [04:26:08] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.72, 9.40, 9.26 [04:26:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.92, 4.51, 3.96 [04:26:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.69, 10.19, 10.52 [04:32:27] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.58, 11.48, 10.76 [04:32:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.67, 3.99, 3.91 [04:32:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.13, 9.34, 10.14 [04:33:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.77, 3.58, 3.88 [04:36:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.76, 3.91, 3.87 [04:36:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.90, 10.82, 10.58 [04:37:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.34, 4.07, 4.01 [04:38:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.93, 3.76, 3.81 [04:39:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.00, 3.76, 3.91 [04:40:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.11, 9.06, 9.90 [04:40:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.18, 9.44, 10.05 [04:41:06] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [04:42:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.59, 4.22, 3.94 [04:45:34] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.50, 1.64, 1.70 [04:46:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.91, 3.87 [04:47:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.51, 9.75, 8.99 [04:47:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.62, 4.59, 4.16 [04:48:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.77, 4.08, 3.95 [04:49:34] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.85, 1.76, 1.73 [04:49:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.91, 9.80, 9.83 [04:49:47] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.01, 8.73, 8.71 [04:49:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.69, 3.78, 3.91 [04:50:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.27, 3.44, 3.74 [04:51:34] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.29, 1.89, 1.78 [04:51:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.80, 9.29, 9.63 [04:53:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.51, 4.35, 4.03 [04:54:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 4.14, 3.97 [04:54:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.72, 10.38, 9.67 [04:54:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.39, 10.32, 9.52 [04:55:34] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.58, 1.80, 1.78 [04:56:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.14, 9.86, 9.57 [04:56:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.06, 9.50, 9.32 [04:57:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.25, 10.15, 9.89 [05:00:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.33, 10.39, 9.73 [05:01:34] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.86, 2.08, 1.86 [05:02:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.27, 9.66, 9.55 [05:03:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.45, 9.89, 9.93 [05:07:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-43/±0] 13https://github.com/miraheze/mw-config/compare/63687dd49472...fa48f8b22a9d [05:07:15] [url] Comparing 63687dd49472...fa48f8b22a9d · miraheze/mw-config · GitHub | github.com [05:07:16] [02miraheze/mw-config] 07Universal-Omega 03fa48f8b - Delete vendor directory [05:07:17] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [05:07:18] [url] Page not found · GitHub · GitHub | github.com [05:07:36] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [05:07:36] [url] Page not found · GitHub · GitHub | github.com [05:07:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/fa48f8b22a9d...27a830e4b742 [05:07:38] [url] Comparing fa48f8b22a9d...27a830e4b742 · miraheze/mw-config · GitHub | github.com [05:07:39] [02miraheze/mw-config] 07Universal-Omega 0327a830e - Update .gitignore [05:08:04] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [05:08:05] [url] Page not found · GitHub · GitHub | github.com [05:08:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/27a830e4b742...964926d2feda [05:08:06] [url] Comparing 27a830e4b742...964926d2feda · miraheze/mw-config · GitHub | github.com [05:08:07] [02miraheze/mw-config] 07Universal-Omega 03964926d - Update continuousIntegration.yml [05:08:27] miraheze/mw-config - Universal-Omega the build passed. [05:08:47] miraheze/mw-config - Universal-Omega the build passed. [05:09:13] miraheze/mw-config - Universal-Omega the build passed. [05:09:20] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.99, 6.43, 5.75 [05:09:37] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [05:11:16] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.69, 6.28, 5.76 [05:11:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.01, 10.49, 9.93 [05:13:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.36, 9.71, 9.71 [05:17:26] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.240 second response time [05:17:28] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [05:17:36] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.008 second response time [05:17:41] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.160 second response time [05:17:47] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.247 second response time [05:17:54] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:17:56] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 15.03, 12.11, 10.37 [05:18:17] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.08, 2.05, 1.42 [05:18:17] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:24] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:30] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.013 second response time [05:19:11] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 15.67, 13.04, 11.08 [05:19:32] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 1.310 second response time [05:19:43] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.403 second response time [05:19:50] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.026 second response time [05:19:51] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.348 second response time [05:20:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.80, 11.45, 10.39 [05:20:13] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.46, 1.53, 1.30 [05:20:20] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 4.542 second response time [05:20:24] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 2.896 second response time [05:20:28] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.022 second response time [05:21:19] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [05:21:34] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.589 second response time [05:21:36] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.018 second response time [05:21:50] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.685 second response time [05:22:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.98, 10.04, 9.99 [05:23:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.00, 11.27, 10.79 [05:23:19] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [05:28:18] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:30:19] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 2.202 second response time [05:31:45] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.32, 6.65, 5.99 [05:33:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.82, 9.80, 8.98 [05:34:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.04, 11.11, 10.81 [05:35:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.12, 8.88, 8.75 [05:36:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.66, 10.70, 10.71 [05:37:34] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.92, 7.82, 6.68 [05:39:31] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.31, 7.82, 6.84 [05:44:13] !log re-start deleteWikis.php [05:44:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:47:18] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.43, 8.22, 7.34 [05:48:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.44, 9.47, 10.10 [05:49:14] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.31, 7.90, 7.33 [05:49:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.05, 10.44, 9.59 [05:50:57] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.14, 10.45, 9.91 [05:51:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.73, 9.73, 9.43 [05:52:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.24, 10.08, 10.21 [05:52:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.75, 10.77, 10.09 [05:53:07] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.19, 7.76, 7.39 [05:54:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.72, 9.23, 9.85 [05:54:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.56, 9.33, 9.66 [05:55:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.96, 10.87, 9.93 [05:56:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.62, 3.99 [05:57:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.04, 9.30, 9.47 [05:58:17] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+0/-2/±2] 13https://github.com/miraheze/ssl/compare/fedee6e86195...39e86c5cd7a3 [05:58:18] [url] Comparing fedee6e86195...39e86c5cd7a3 · miraheze/ssl · GitHub | github.com [05:58:18] [02miraheze/ssl] 07Reception123 030b24bb6 - rm wiki.chippyapps.com (no longer pointing) [05:58:20] [02miraheze/ssl] 07Reception123 0339e86c5 - rm privacywiki.xyz (no longer pointing) [05:58:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.73, 10.24, 9.96 [05:58:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.40, 10.13, 10.04 [06:00:28] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+0/-2/±3] 13https://github.com/miraheze/ssl/compare/39e86c5cd7a3...10e20cac594f [06:00:29] [url] Comparing 39e86c5cd7a3...10e20cac594f · miraheze/ssl · GitHub | github.com [06:00:30] [02miraheze/ssl] 07Reception123 03519d663 - rm aarthal.com (no longer pointing) [06:00:31] [02miraheze/ssl] 07Reception123 0310e20ca - rm erikapedia.com (no longer pointing) [06:00:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.07, 10.12, 10.05 [06:00:54] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.10, 7.68, 7.59 [06:02:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.67, 9.37, 9.77 [06:03:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.97, 10.33, 9.89 [06:03:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.13, 3.30, 3.88 [06:05:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.03, 10.00, 9.83 [06:05:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.72, 3.91, 4.04 [06:07:23] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.16, 3.31, 3.80 [06:08:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.43, 10.32, 10.02 [06:09:24] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 2.281 second response time [06:10:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.31, 3.75, 3.69 [06:10:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.36, 9.72, 9.85 [06:10:51] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.81, 7.18, 7.19 [06:11:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.91, 9.25, 8.70 [06:12:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.63, 3.27, 3.52 [06:12:51] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.97, 6.83, 7.07 [06:13:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.82, 10.24, 9.86 [06:15:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.68, 10.15, 9.88 [06:15:41] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.77, 8.75, 8.68 [06:16:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.30, 3.91, 3.69 [06:17:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.17, 4.51, 4.13 [06:17:54] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:18:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.41, 10.96, 10.30 [06:19:54] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.495 second response time [06:20:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.39, 9.30, 9.78 [06:20:51] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.51, 7.45, 7.17 [06:22:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.89, 3.66, 3.69 [06:22:51] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.94, 7.37, 7.19 [06:24:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.93, 11.17, 10.40 [06:24:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.62, 10.58, 10.03 [06:25:11] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:25:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.56, 3.89 [06:26:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.25, 4.20, 3.87 [06:26:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.89, 11.58, 10.66 [06:27:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.43, 5.07, 4.40 [06:30:42] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.14, 5.84, 6.55 [06:30:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.47, 11.68, 10.87 [06:31:12] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 2.495 second response time [06:32:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.89, 11.13, 10.76 [06:32:45] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.59, 10.42, 9.74 [06:34:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.97, 9.39, 9.47 [06:34:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.73, 11.81, 11.03 [06:35:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.72, 11.47, 10.56 [06:37:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.31, 10.52, 10.32 [06:38:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.70, 11.29, 11.05 [06:39:00] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.53, 9.84, 10.09 [06:40:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.95, 9.79, 9.49 [06:42:16] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.25, 9.72, 9.51 [06:44:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.44, 3.55, 3.87 [06:44:37] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.02, 11.55, 11.06 [06:44:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.56, 10.56, 10.28 [06:45:18] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:45:20] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:45:32] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:46:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.97, 11.80, 11.23 [06:46:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.72, 3.89 [06:46:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.33, 10.08, 10.16 [06:47:12] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.084 second response time [06:47:15] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.312 second response time [06:47:31] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.570 second response time [06:48:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.81, 3.80, 3.90 [06:52:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.26, 3.60, 3.92 [06:54:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.24, 11.06, 10.45 [06:54:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.21, 11.18, 10.95 [06:56:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.23, 10.52, 10.34 [06:56:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.59, 11.28, 11.03 [06:56:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 3.65, 3.74 [06:56:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.21, 3.98, 3.97 [06:58:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.25, 3.52, 3.69 [07:00:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.40, 11.01, 10.55 [07:00:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.56, 11.05, 10.94 [07:02:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.94, 10.19, 10.31 [07:02:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.37, 9.94, 10.54 [07:02:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.23, 3.55, 3.65 [07:04:14] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 4.49, 8.29, 9.61 [07:04:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.65, 8.58, 9.96 [07:04:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.59, 3.17, 3.49 [07:05:21] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.012 second response time [07:06:06] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:06:26] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:07:20] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:07:25] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:08:12] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 8.066 second response time [07:08:31] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 6.010 second response time [07:09:15] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 1.196 second response time [07:09:17] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.12, 6.44, 6.17 [07:09:21] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 1.407 second response time [07:09:22] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 2.291 second response time [07:10:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.93, 3.14, 3.39 [07:13:14] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.22, 7.31, 6.54 [07:14:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.29, 4.10, 3.77 [07:15:13] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.68, 6.70, 6.41 [07:16:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.09, 3.78, 3.69 [07:18:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.53, 3.92 [07:21:09] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.09, 7.72, 6.83 [07:22:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.70, 3.95 [07:22:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.70, 2.88, 3.30 [07:23:07] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.76, 7.64, 6.91 [07:25:52] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.33, 11.74, 10.32 [07:26:10] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.30, 2.78, 3.39 [07:26:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.89, 11.37, 10.37 [07:27:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.66, 10.64, 10.08 [07:29:04] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.33, 6.68, 6.73 [07:29:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.81, 10.20, 9.97 [07:30:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.93, 9.56, 9.90 [07:32:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.08, 3.31, 3.43 [07:32:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.57, 3.88, 3.79 [07:32:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.84, 10.02, 8.98 [07:33:07] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.51, 2.08, 1.34 [07:33:56] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:33:58] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.13, 4.03, 3.68 [07:34:16] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:34:24] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.35, 9.23, 8.83 [07:34:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.24, 4.41, 3.67 [07:35:05] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.92, 1.64, 1.27 [07:36:01] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 6.371 second response time [07:36:14] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 3.914 second response time [07:40:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.19, 3.66, 3.68 [07:42:41] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [07:43:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.89, 3.88 [07:44:36] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [07:44:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.36, 3.70, 3.66 [07:45:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.94, 3.90 [07:46:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.53, 3.61 [07:47:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.70, 9.55, 8.94 [07:48:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.70, 3.66 [07:49:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.81, 3.86 [07:49:42] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.85, 10.99, 9.53 [07:50:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.60, 3.62 [07:51:42] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.15, 9.77, 9.29 [07:52:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.84, 10.00, 8.56 [07:54:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.55, 10.87, 10.22 [07:54:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.79, 10.33, 9.59 [07:54:31] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.08, 9.57, 8.57 [07:55:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.66, 3.79, 3.71 [07:56:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.68, 10.50, 10.18 [07:56:36] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 3.66, 3.58 [07:56:42] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.17, 1.91, 1.50 [07:58:14] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.64, 9.71, 9.54 [07:58:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.38, 3.48 [07:59:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.30, 3.16, 3.47 [08:00:40] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.94, 1.73, 1.55 [08:02:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.30, 10.34, 9.86 [08:02:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.56, 3.61, 3.53 [08:02:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.64, 1.39, 1.45 [08:03:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.63, 3.74, 3.63 [08:04:14] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.06, 10.07, 9.86 [08:04:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.28, 3.23, 3.87 [08:04:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.77, 2.94, 3.29 [08:05:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.12, 3.54, 3.58 [08:06:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.80, 9.39, 10.02 [08:07:28] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.02, 2.93, 3.34 [08:12:10] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:15:00] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.45, 10.82, 10.31 [08:16:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.52, 9.50, 9.89 [08:18:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.95, 3.59, 3.52 [08:21:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 15.27, 11.18, 9.94 [08:22:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.79, 3.18, 3.43 [08:22:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.49, 11.57, 10.58 [08:23:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.44, 10.40, 9.83 [08:24:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.90, 2.73, 3.23 [08:24:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.92, 10.27, 10.23 [08:25:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.80, 8.77, 9.30 [08:26:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.83, 9.27, 9.85 [08:36:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.04, 10.21, 10.13 [08:38:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.21, 11.20, 10.50 [08:40:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.55, 11.05, 10.51 [08:41:39] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [08:47:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.98, 3.85, 3.23 [08:48:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.21, 10.58, 9.58 [08:49:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.84, 3.74, 3.29 [08:50:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.27, 11.02, 9.88 [08:51:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.58, 3.39, 3.21 [08:52:47] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.38, 10.10, 9.68 [09:03:36] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [09:04:27] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.16, 9.61, 10.12 [09:08:36] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [09:10:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.95, 10.05, 10.21 [09:12:22] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.63, 8.79, 9.73 [09:18:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.75, 10.87, 10.22 [09:20:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.36, 9.97, 9.98 [09:28:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.43, 10.44, 10.06 [09:30:09] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.97, 9.62, 9.81 [09:30:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.84, 2.90, 2.51 [09:32:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.88, 2.79, 2.51 [09:44:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 3.21, 2.76 [09:44:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.37, 2.88 [09:46:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.40, 2.75, 2.64 [09:46:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.54, 2.65, 2.68 [09:50:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.66, 10.69, 9.88 [09:52:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.48, 11.10, 10.14 [09:54:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.46, 2.99, 2.75 [09:54:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.85, 10.12, 9.89 [09:55:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.81, 3.62, 3.17 [09:55:54] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.25, 2.21, 1.62 [09:57:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.86, 3.26, 2.89 [09:57:52] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.41, 1.86, 1.56 [09:57:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.53, 3.19 [09:59:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.68, 4.11, 3.21 [09:59:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.92, 5.20, 3.86 [10:00:10] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.52, 4.51, 3.48 [10:01:48] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.83, 1.56, 1.53 [10:05:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 3.95, 3.41 [10:06:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.95, 4.75, 3.74 [10:09:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.99, 3.80 [10:10:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.88, 3.59 [10:13:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.07, 4.13, 3.87 [10:16:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.82, 4.43, 3.87 [10:18:40] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.82, 1.88, 1.60 [10:20:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.54, 3.91, 3.82 [10:20:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.53, 1.64, 1.54 [10:21:43] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.49, 3.66, 3.83 [10:26:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.10, 2.98, 3.40 [10:27:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.56, 3.78, 3.83 [10:27:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.46, 3.91 [10:29:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.73, 3.40, 3.68 [10:37:52] paladox: https://phabricator.miraheze.org/T9383 - I guess we can do my idea when you can since John said yes [10:37:53] [url] ⚓ T9383 db112 disk space very low | phabricator.miraheze.org [10:39:28] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.27, 2.87, 3.31 [10:40:40] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:41:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.28, 3.46 [10:42:39] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.368 second response time [10:43:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.26, 3.44 [10:45:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.61, 4.20, 3.74 [10:49:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.84, 3.83, 3.68 [10:53:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.38, 4.10, 3.79 [10:55:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.75, 3.65, 3.64 [10:58:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.91, 3.49, 3.31 [10:58:36] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.71, 3.29 [11:00:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 3.69, 3.40 [11:00:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.86, 3.54, 3.29 [11:02:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.06, 3.47, 3.35 [11:02:36] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.03, 3.16, 3.20 [11:04:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.29, 3.31, 3.30 [11:04:05] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.57, 6.52, 5.91 [11:06:05] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.38, 6.20, 5.87 [11:11:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.48, 2.97, 3.33 [11:17:00] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.85, 10.63, 9.67 [11:17:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.99, 3.33, 3.33 [11:18:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.25, 9.94, 9.56 [11:19:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.42, 3.76, 3.49 [11:23:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.85, 3.60 [11:23:34] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 3.55, 3.28 [11:23:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.43, 1.61, 1.99 [11:25:26] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.31, 1.83, 1.47 [11:25:31] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.94, 3.00, 3.12 [11:25:56] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.93, 1.86, 1.48 [11:26:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.58, 10.26, 9.72 [11:27:23] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.54, 1.69, 1.46 [11:27:54] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.19, 1.62, 1.44 [11:28:14] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.013 second response time [11:28:34] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.010 second response time [11:28:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.62, 10.04, 9.70 [11:29:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.42, 3.06, 3.35 [11:29:23] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.252 second response time [11:29:28] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.010 second response time [11:29:37] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.235 second response time [11:30:11] PROBLEM - mail121 webmail.miraheze.org HTTPS on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.22, 1.46, 1.67 [11:49:43] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.97, 1.72, 1.66 [11:50:59] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 3.29, 2.92 [11:51:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.88, 4.05, 3.41 [11:52:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.95, 3.36, 2.89 [11:53:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.75, 3.75, 3.39 [11:53:43] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.43, 1.68, 1.67 [11:54:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.02, 2.78, 2.72 [11:54:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.56, 3.13 [11:55:04] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.40, 4.08, 3.53 [11:56:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.49, 4.16, 3.39 [11:58:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.50, 3.83, 3.17 [12:00:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.91, 3.75, 3.21 [12:01:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.91, 3.68 [12:01:27] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.47, 1.71, 1.40 [12:01:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.75, 1.70, 1.67 [12:03:24] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.36, 1.52, 1.36 [12:03:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.53, 1.64, 1.65 [12:05:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.45, 4.55, 3.98 [12:06:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.49, 3.07, 3.11 [12:06:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.11, 3.46, 3.55 [12:06:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.73, 3.87, 3.80 [12:10:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 3.57, 3.66 [12:13:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.88, 1.74 [12:14:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.64, 2.80, 3.28 [12:14:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.12, 3.68, 3.70 [12:16:22] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.48, 1.73, 1.44 [12:18:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.66, 10.62, 10.08 [12:18:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.29, 3.84, 3.71 [12:18:59] !log db112: MariaDB [(none)]> PURGE BINARY LOGS BEFORE '2022-06-16 00:00:00'; [12:19:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:20:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.50, 9.35, 9.67 [12:20:18] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.74, 1.92, 1.59 [12:20:52] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.50, 6.11, 5.44 [12:20:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.33, 3.53 [12:21:39] !log stop mysql on db112 to clean bin logs manually as due to full disk the purge command is stalled. [12:21:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:22:51] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.39, 6.75, 5.80 [12:22:52] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 2.99, 3.37 [12:23:42] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [12:23:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.75, 1.93, 1.89 [12:23:44] RECOVERY - mail121 webmail.miraheze.org HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 200 OK - 6654 bytes in 0.024 second response time [12:23:53] !log removed 3 binlogs from db112 (around the same time as paladox started, didn't see) [12:23:57] paladox: we can probably start mysql again now I think [12:24:12] PROBLEM - db112 MariaDB on db112 is CRITICAL: Can't connect to MySQL server on 'db112.miraheze.org' (115) [12:24:15] ohhh [12:24:28] the command is still running [12:24:28] I just delete them from /srv/mariadb so it's easier than having to stop mysql [12:24:39] yeh you have to stop mysql i thought [12:24:44] if you are manually deleting bin logs [12:24:48] not for that method, only if you want to delete with the DB method [12:24:55] mine worked last time without stopping it [12:25:03] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.42, 11.32, 9.78 [12:25:05] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4167 bytes in 0.157 second response time [12:25:08] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.014 second response time [12:25:14] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:21] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:26] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:53] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:02] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.00, 11.64, 10.46 [12:26:13] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.19, 1.89, 1.66 [12:26:20] ok it's stopped [12:26:38] we can delete a few more bin logs [12:26:52] yeah sure [12:27:00] Reception123: you also have to delete from mysql-bin.index! [12:27:10] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 7.963 second response time [12:27:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.15, 3.11 [12:27:13] paladox: oh, I didn't know. Will remember that next time [12:27:15] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 5.822 second response time [12:27:19] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 3.960 second response time [12:27:19] though in terms of space it definitely worked [12:27:26] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.850 second response time [12:27:52] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.282 second response time [12:28:01] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.630 second response time [12:28:43] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 11403 MB (8% inode=99%); [12:28:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.02, 10.48, 9.88 [12:29:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.51, 2.94, 3.04 [12:29:20] paladox: think it's ok to start mysql now? [12:29:26] Reception123: started it [12:29:30] thanks! [12:29:33] !log starting mysql on db112 [12:29:33] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 5.712 second response time [12:29:38] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 99046 bytes in 0.799 second response time [12:29:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:29:42] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.097 second response time [12:29:46] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 99046 bytes in 0.764 second response time [12:30:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.21, 10.40, 10.29 [12:30:09] !log DELETED and DROPPED https://phabricator.miraheze.org/P465 [12:30:09] [url] ✎ P465 Deleted wikis 15/06/2022 (PART 2) | phabricator.miraheze.org [12:30:10] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.98, 1.99, 1.75 [12:30:10] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 99077 bytes in 0.312 second response time [12:30:12] RECOVERY - db112 MariaDB on db112 is OK: Uptime: 49 Threads: 30 Questions: 38517 Slow queries: 589 Opens: 583 Open tables: 577 Queries per second avg: 786.061 [12:30:13] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.247 second response time [12:30:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:30:50] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.13, 9.75, 9.68 [12:31:05] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 18991 bytes in 0.143 second response time [12:31:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.11, 1.95, 1.89 [12:32:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.64, 9.93, 10.11 [12:32:08] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.68, 2.02, 1.80 [12:33:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.32, 3.05 [12:33:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.79, 1.90, 1.89 [12:34:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.41, 1.86, 1.77 [12:34:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.89, 11.34, 10.28 [12:34:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.65, 3.37 [12:34:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.07, 3.43, 3.24 [12:35:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.47, 2.97, 2.95 [12:36:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.13, 11.95, 10.89 [12:36:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.63, 9.75, 9.82 [12:36:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.71, 3.39, 3.32 [12:37:41] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.08, 7.35, 6.37 [12:38:02] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.89, 1.44, 1.63 [12:38:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.95, 10.12, 10.36 [12:38:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.05, 2.97, 3.11 [12:39:37] !log matomo101: ./console core:delete-logs-data --dates=2015-01-01,2020-06-14 [12:39:39] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.93, 6.65, 6.22 [12:40:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.56, 8.58, 9.77 [12:40:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:41:18] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:22] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:23] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:30] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:34] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:39] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:53] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:43:12] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.021 second response time [12:43:22] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 6.060 second response time [12:43:24] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 6.125 second response time [12:43:30] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 5.459 second response time [12:43:31] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.050 second response time [12:43:42] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23899 bytes in 5.030 second response time [12:43:48] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.318 second response time [12:47:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.41, 1.92, 1.82 [12:49:33] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.45, 6.99, 6.43 [12:49:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.86, 1.81 [12:49:56] db101: 274G available -> 294G available [12:49:57] db111: 305G available -> 322G available [12:49:57] db121: 255G available -> 271G available [12:51:19] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.73, 9.54, 9.10 [12:52:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.70, 9.25, 8.94 [12:53:18] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.325 second response time [12:53:30] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.48, 6.66, 6.47 [12:53:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.94, 3.29, 3.13 [12:54:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.89, 9.63, 9.17 [12:54:14] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.30, 8.97, 8.86 [12:55:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.03, 2.74, 2.95 [12:57:43] PROBLEM - db112 Disk Space on db112 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:58:04] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:06] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:06] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 16.17, 12.21, 10.20 [12:58:06] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:06] PROBLEM - cp31 Stunnel HTTP for mon111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:09] PROBLEM - mail121 webmail.miraheze.org HTTPS on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:09] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.009 second response time [12:58:13] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.235 second response time [12:58:14] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.008 second response time [12:58:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 15.28, 11.55, 9.84 [12:58:14] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.085 second response time [12:58:22] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.012 second response time [12:58:26] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:27] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:44] PROBLEM - mon111 HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:46] PROBLEM - mon111 grafana.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:55] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.240 second response time [12:59:19] PROBLEM - db112 SSH on db112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:33] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:34] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:38] PROBLEM - cp21 Stunnel HTTP for mon111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:39] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 12318 MB (9% inode=99%); [12:59:39] PROBLEM - cp31 Stunnel HTTP for phab121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:55] PROBLEM - cp30 Stunnel HTTP for phab121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:02] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 2.407 second response time [13:00:02] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 2.036 second response time [13:00:03] RECOVERY - cp31 Stunnel HTTP for mon111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 34513 bytes in 0.397 second response time [13:00:03] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 5.335 second response time [13:00:10] RECOVERY - mail121 webmail.miraheze.org HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 200 OK - 6153 bytes in 1.623 second response time [13:00:10] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 99069 bytes in 1.542 second response time [13:00:12] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 99077 bytes in 2.560 second response time [13:00:20] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.451 second response time [13:00:20] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 6.273 second response time [13:00:35] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 34.47, 36.22, 17.44 [13:00:41] RECOVERY - mon111 HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 200 OK - 34505 bytes in 0.031 second response time [13:00:44] RECOVERY - mon111 grafana.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 200 OK - 34513 bytes in 0.028 second response time [13:00:56] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 99069 bytes in 1.041 second response time [13:01:00] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw111 mw112 [13:01:09] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw111 [13:01:15] RECOVERY - db112 SSH on db112 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [13:01:29] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.087 second response time [13:01:32] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 18991 bytes in 0.139 second response time [13:01:33] RECOVERY - cp21 Stunnel HTTP for mon111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 34467 bytes in 0.038 second response time [13:01:37] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [13:01:38] RECOVERY - cp31 Stunnel HTTP for phab121 on cp31 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.357 second response time [13:01:43] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.31, 1.49, 1.65 [13:01:55] RECOVERY - cp30 Stunnel HTTP for phab121 on cp30 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.414 second response time [13:02:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.92, 11.51, 10.58 [13:02:09] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [13:02:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.71, 11.72, 10.38 [13:02:21] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.214 second response time [13:02:38] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23899 bytes in 3.133 second response time [13:02:41] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 3.859 second response time [13:03:00] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [13:03:09] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [13:03:37] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [13:04:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.26, 11.97, 10.84 [13:04:09] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [13:04:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 15.71, 13.14, 11.05 [13:06:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.06, 11.58, 10.86 [13:06:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.90, 11.43, 10.71 [13:10:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.42, 7.88, 9.53 [13:10:14] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.89, 8.81, 9.79 [13:11:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/fc632b7823a6 [13:11:35] [02miraheze/puppet] 07paladox 03fc632b7 - matomo: Add missing grants to piwik grants (matomo) [13:11:37] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [13:11:37] [url] Page not found · GitHub · GitHub | github.com [13:11:38] [02puppet] 07paladox opened pull request 03#2672: matomo: Add missing grants to piwik grants (matomo) - 13https://github.com/miraheze/puppet/pull/2672 [13:11:39] [url] Page not found · GitHub · GitHub | github.com [13:11:49] [02puppet] 07paladox commented on pull request 03#2672: matomo: Add missing grants to piwik grants (matomo) - 13https://github.com/miraheze/puppet/pull/2672#issuecomment-1156454416 [13:11:49] [url] matomo: Add missing grants to piwik grants (matomo) by paladox · Pull Request #2672 · miraheze/puppet · GitHub | github.com [13:12:15] [02puppet] 07paladox closed pull request 03#2672: matomo: Add missing grants to piwik grants (matomo) - 13https://github.com/miraheze/puppet/pull/2672 [13:12:15] [url] Page not found · GitHub · GitHub | github.com [13:12:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1285febd5649...acea0806300b [13:12:17] [url] Comparing 1285febd5649...acea0806300b · miraheze/puppet · GitHub | github.com [13:12:18] [02miraheze/puppet] 07paladox 03acea080 - matomo: Add missing grants to piwik grants (matomo) (#2672) [13:12:19] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [13:12:21] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [13:12:21] [url] Page not found · GitHub · GitHub | github.com [13:15:47] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.239 second response time [13:16:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.11, 10.28, 9.88 [13:16:12] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.011 second response time [13:16:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.04, 10.42, 10.00 [13:16:14] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.014 second response time [13:16:27] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.013 second response time [13:17:05] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.236 second response time [13:18:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.25, 10.40, 9.97 [13:18:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.85, 10.28, 9.97 [13:20:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.22, 11.22, 10.33 [13:20:20] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:25] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:25] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:26] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:27] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:29] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:30] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:32] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:59] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:21:25] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 1.73, 2.39, 5.85 [13:22:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.02, 3.31, 2.64 [13:23:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.14, 4.74, 3.56 [13:23:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 8.80, 5.40, 3.74 [13:24:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.79, 11.46, 10.58 [13:24:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.77, 3.27, 2.73 [13:24:42] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:27:18] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.35, 2.80, 4.96 [13:28:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.51, 11.02, 10.74 [13:28:20] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.025 second response time [13:28:28] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 2.847 second response time [13:28:35] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.313 second response time [13:28:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.88, 3.09 [13:28:41] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 5.025 second response time [13:28:43] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 2.734 second response time [13:28:49] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.253 second response time [13:28:50] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.788 second response time [13:28:52] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 7.287 second response time [13:28:54] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.949 second response time [13:30:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.87, 3.19 [13:32:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 15.00, 12.37, 11.26 [13:33:41] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.704 second response time [13:34:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.35, 11.93, 11.46 [13:34:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.29, 11.24, 10.99 [13:34:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.99, 3.99, 3.35 [13:35:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.72, 3.75, 3.86 [13:35:45] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.70, 1.57 [13:36:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.28, 12.78, 11.82 [13:36:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.15, 12.45, 11.47 [13:36:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.84, 3.38, 3.20 [13:37:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.24, 3.73, 3.89 [13:37:27] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:37:43] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.31, 1.56, 1.54 [13:37:47] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [13:39:27] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.328 second response time [13:39:53] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23899 bytes in 7.410 second response time [13:41:28] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.65, 2.62, 3.32 [13:42:40] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:42:49] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:42:52] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:42:55] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:44:08] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [13:47:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.49, 3.58 [13:47:22] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:24] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:33] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:36] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:38] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:43] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [13:47:58] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [13:48:01] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:08] ^ paladox hmm I really don't get what's going on [13:48:49] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 7.567 second response time [13:48:54] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.019 second response time [13:49:02] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.314 second response time [13:49:03] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.462 second response time [13:49:17] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.334 second response time [13:49:18] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.022 second response time [13:49:31] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.361 second response time [13:49:32] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.516 second response time [13:49:35] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.024 second response time [13:49:41] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw112 [13:50:00] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.458 second response time [13:50:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 5.16, 10.81, 11.89 [13:50:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 3.18, 3.28 [13:51:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.46, 3.55 [13:51:37] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [13:51:38] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [13:52:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.14, 11.69, 12.06 [13:52:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.72, 3.06, 3.23 [13:53:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.23, 3.08, 3.40 [13:53:27] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:35] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 3.855 second response time [13:53:38] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:55] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:58] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:21] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.102 second response time [13:55:30] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.306 second response time [13:55:34] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.329 second response time [13:55:52] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.018 second response time [13:56:05] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23899 bytes in 9.048 second response time [13:58:40] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:46] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:52] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:54] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.15, 1.86, 1.68 [14:00:30] PROBLEM - mon111 Puppet on mon111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:00:34] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.024 second response time [14:00:36] PROBLEM - reports121 Puppet on reports121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:00:42] PROBLEM - graylog121 Puppet on graylog121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:00:46] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.717 second response time [14:00:53] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 1.107 second response time [14:00:54] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.334 second response time [14:02:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.54, 3.24 [14:02:16] PROBLEM - gluster121 Puppet on gluster121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:03:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.61, 3.45 [14:03:51] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.245 second response time [14:03:57] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.012 second response time [14:04:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.89, 3.23, 3.15 [14:04:42] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:48] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 2.00, 1.96, 1.79 [14:05:03] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:07] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [14:05:07] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [14:05:08] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:21] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:24] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:26] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:16] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.57, 3.08, 3.27 [14:07:10] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [14:07:37] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [14:08:45] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 5.029 second response time [14:09:01] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.327 second response time [14:09:07] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [14:09:15] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.447 second response time [14:09:19] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 1.129 second response time [14:09:22] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.018 second response time [14:09:26] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 2.168 second response time [14:09:33] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23899 bytes in 3.806 second response time [14:09:46] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.345 second response time [14:10:07] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [14:11:37] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [14:14:11] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.107 second response time [14:14:20] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.410 second response time [14:16:20] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.15, 1.92, 1.82 [14:18:07] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:09] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:10] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.62, 1.84, 1.80 [14:18:23] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:45] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:02] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.468 second response time [14:20:07] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 2.431 second response time [14:20:14] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 4.364 second response time [14:22:47] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:58] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:05] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:47] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:11] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:12] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.460 second response time [14:24:13] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:42] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:56] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.33, 1.58, 1.69 [14:26:50] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 8.857 second response time [14:26:59] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 3.682 second response time [14:27:15] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 6.901 second response time [14:27:46] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 1.493 second response time [14:28:10] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 1.554 second response time [14:28:15] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 3.698 second response time [14:28:16] RECOVERY - gluster121 Puppet on gluster121 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [14:28:23] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 8.135 second response time [14:28:30] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [14:28:36] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:28:41] RECOVERY - graylog121 Puppet on graylog121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:30:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.36, 3.73, 3.33 [14:31:19] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:20] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:32] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:47] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:28] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:35] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:53] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:09] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw112 [14:34:17] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.66, 3.00, 3.14 [14:35:37] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [14:35:42] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 9.919 second response time [14:35:54] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 9.352 second response time [14:36:26] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.020 second response time [14:36:35] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 0.556 second response time [14:37:03] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.306 second response time [14:37:04] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.030 second response time [14:37:09] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [14:37:12] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 1.708 second response time [14:37:19] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.683 second response time [14:37:25] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.312 second response time [14:37:37] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [14:39:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.70, 3.83, 3.41 [14:40:09] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:12] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:43] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:55] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:07] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [14:41:14] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:23] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:37] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:45] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.08, 3.11, 3.20 [14:43:37] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [14:43:53] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 9.760 second response time [14:44:01] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 5.770 second response time [14:44:13] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.019 second response time [14:44:17] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.325 second response time [14:44:43] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 1.513 second response time [14:44:57] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 1.548 second response time [14:45:18] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.620 second response time [14:45:36] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 2.856 second response time [14:45:37] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [14:45:46] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 8.991 second response time [14:46:07] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [14:46:09] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [14:48:09] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [14:48:19] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:26] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.911 second response time [14:48:29] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:30] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:49] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:50] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:49:20] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:49:39] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:09] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw111 [14:52:08] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:12] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:34] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:57:09] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [15:01:09] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw111 [15:01:43] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 4.488 second response time [15:02:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.89, 3.14, 2.64 [15:03:09] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [15:04:07] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.59, 2.34, 1.65 [15:04:09] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [15:04:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.06, 3.42, 3.07 [15:04:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.64, 3.25 [15:04:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.75, 2.97, 2.65 [15:06:03] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:33] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.48, 3.09, 2.99 [15:06:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.57, 3.28, 3.17 [15:06:57] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 9.548 second response time [15:07:08] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 6.081 second response time [15:08:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.93, 1.71, 1.55 [15:10:02] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.80, 1.38, 1.45 [15:10:09] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [15:11:05] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:11:44] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.960 second response time [15:11:46] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.311 second response time [15:11:47] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 5.982 second response time [15:12:08] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.306 second response time [15:12:36] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 2.771 second response time [15:12:43] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 4.146 second response time [15:12:54] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.031 second response time [15:12:59] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.021 second response time [15:13:09] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw111 [15:15:41] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:16:10] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:16:12] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:16:15] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:16:18] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:16:55] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:16:56] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:17:09] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.618 second response time [15:17:09] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [15:17:10] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 2.187 second response time [15:18:52] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.75, 1.64, 1.55 [15:19:37] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw111 mw112 [15:19:42] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 3.366 second response time [15:20:01] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.311 second response time [15:20:09] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.322 second response time [15:20:15] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.355 second response time [15:20:18] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23899 bytes in 1.247 second response time [15:20:21] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.306 second response time [15:20:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.73, 3.79, 3.34 [15:20:45] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.329 second response time [15:20:50] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.101 second response time [15:21:09] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw112 [15:21:11] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 3.366 second response time [15:22:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.73, 3.40, 3.26 [15:22:42] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.30, 1.53, 1.53 [15:23:09] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [15:23:34] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [15:23:37] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [15:25:22] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 3.138 second response time [15:27:09] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw111 [15:28:13] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:38] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:39] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:45] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:55] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:00] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:05] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:05] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:09] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [15:29:14] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:41] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:55] PROBLEM - bast121 NTP time on bast121 is CRITICAL: NTP CRITICAL: Offset 0.7065435052 secs [15:32:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.38, 4.14, 3.47 [15:32:09] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [15:32:38] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 2.771 second response time [15:32:56] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.306 second response time [15:32:56] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.314 second response time [15:33:25] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 8.218 second response time [15:33:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.70, 3.56, 3.34 [15:34:17] PROBLEM - gluster111 NTP time on gluster111 is CRITICAL: NTP CRITICAL: Offset 0.7063143551 secs [15:35:37] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [15:36:09] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [15:36:21] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 2.625 second response time [15:36:45] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 4.065 second response time [15:36:52] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:07] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:25] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:46] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.08, 2.97, 3.16 [15:38:06] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.30, 1.76, 1.61 [15:39:07] PROBLEM - jobchron121 NTP time on jobchron121 is CRITICAL: NTP CRITICAL: Offset 0.7052312493 secs [15:40:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.82, 1.74, 1.63 [15:40:39] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:01] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 6.938 second response time [15:41:07] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:11] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 3.994 second response time [15:41:35] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.610 second response time [15:41:36] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.326 second response time [15:41:37] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [15:41:40] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.326 second response time [15:41:50] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.992 second response time [15:41:52] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.697 second response time [15:41:56] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.018 second response time [15:42:39] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [15:43:03] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 1.552 second response time [15:43:52] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.10, 1.89, 1.71 [15:45:09] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.017 second response time [15:45:22] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.250 second response time [15:45:51] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:05] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:05] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:06] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:14] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:20] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:03] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.249 second response time [15:50:37] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 8.115 second response time [15:50:49] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 9.208 second response time [15:51:13] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 2.208 second response time [15:51:18] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.312 second response time [15:51:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.35, 3.05 [15:51:37] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [15:51:56] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw112 [15:52:32] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 7.686 second response time [15:52:34] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 9.297 second response time [15:53:29] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset -0.003935575485 secs [15:55:06] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.07, 3.36, 3.15 [15:55:37] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [15:55:56] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [15:56:25] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 9.092 second response time [15:56:30] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 6.709 second response time [15:57:39] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [15:57:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.97, 2.00, 1.92 [15:59:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.50, 3.41, 3.27 [15:59:21] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:28] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:50] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:56] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:15] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:44] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:47] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:02:55] RECOVERY - bast121 NTP time on bast121 is OK: NTP OK: Offset -0.006029307842 secs [16:03:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.72, 3.33, 3.30 [16:05:04] RECOVERY - gluster111 NTP time on gluster111 is OK: NTP OK: Offset -0.006889283657 secs [16:05:09] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw112 [16:05:58] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [16:07:09] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [16:07:43] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.10, 2.37, 2.07 [16:09:57] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [16:15:55] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [16:19:55] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [16:23:53] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [16:24:33] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [16:24:33] [url] Page not found · GitHub · GitHub | github.com [16:24:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/964926d2feda...d897d7ef948b [16:24:35] [url] Comparing 964926d2feda...d897d7ef948b · miraheze/mw-config · GitHub | github.com [16:24:36] [02miraheze/mw-config] 07Universal-Omega 03d897d7e - Update MirahezeFunctions.php [16:24:36] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.85, 4.81, 3.31 [16:24:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.12, 3.95, 3.17 [16:25:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.44, 6.01, 4.23 [16:25:49] miraheze/mw-config - Universal-Omega the build passed. [16:25:53] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [16:27:01] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [16:27:41] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw111 [16:31:35] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [16:32:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d897d7ef948b...c01cf7baebee [16:32:09] [url] Comparing d897d7ef948b...c01cf7baebee · miraheze/mw-config · GitHub | github.com [16:32:10] [02miraheze/mw-config] 07Universal-Omega 03c01cf7b - Update MirahezeFunctions.php [16:32:11] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [16:32:12] [url] Page not found · GitHub · GitHub | github.com [16:32:53] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [16:33:25] miraheze/mw-config - Universal-Omega the build passed. [16:36:44] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [16:36:44] [url] Page not found · GitHub · GitHub | github.com [16:36:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c01cf7baebee...758d8a5074bd [16:36:46] [url] Comparing c01cf7baebee...758d8a5074bd · miraheze/mw-config · GitHub | github.com [16:36:47] [02miraheze/mw-config] 07Universal-Omega 03758d8a5 - Update MirahezeFunctions.php [16:37:57] miraheze/mw-config - Universal-Omega the build passed. [16:39:37] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [16:40:01] [02mw-config] 07AgentIsai opened pull request 03#4758: Sitenotice for upgrade - 13https://github.com/miraheze/mw-config/pull/4758 [16:40:02] [url] Page not found · GitHub · GitHub | github.com [16:40:05] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 341 bytes in 0.260 second response time [16:41:10] miraheze/mw-config - AgentIsai the build passed. [16:41:18] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw112 [16:41:51] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 3.531 second response time [16:42:03] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 7.996 second response time [16:42:04] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 7.711 second response time [16:42:08] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 6.371 second response time [16:42:12] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 1.585 second response time [16:43:12] CosmicAlpha: somethhing very wrong with beta config [16:43:13] Fatal error: Uncaught ConfigException: DiscussionTools requires $wgLocaltimezone to be set in /srv/mediawiki/w/extensions/DiscussionTools/includes/Hooks/RegistrationHooks.php:27 [16:43:15] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [16:43:30] RhinosF1: Aware. [16:43:49] CosmicAlpha: ok [16:44:04] let's be careful in case beta needed for post upgrade mess [16:45:37] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [16:45:43] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.24, 1.38, 1.97 [16:46:04] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:46:16] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:46:19] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.013 second response time [16:46:19] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:46:23] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:47:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.54, 1.95 [16:47:46] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 5.558 second response time [16:47:46] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 8.160 second response time [16:47:58] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.023 second response time [16:48:11] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.287 second response time [16:48:22] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 3.503 second response time [16:48:45] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.496 second response time [16:49:13] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 8.979 second response time [16:49:14] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 8.634 second response time [16:49:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.27, 1.40, 1.84 [16:50:23] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 6.330 second response time [16:50:37] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23899 bytes in 8.749 second response time [16:53:14] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:37] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:43] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:07] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.53, 3.93 [16:54:41] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:05] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.24, 1.33, 1.67 [16:56:10] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [16:56:10] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw111 [16:56:10] [url] Page not found · GitHub · GitHub | github.com [16:56:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/758d8a5074bd...e7eeffadf9eb [16:56:12] [url] Comparing 758d8a5074bd...e7eeffadf9eb · miraheze/mw-config · GitHub | github.com [16:56:13] [02miraheze/mw-config] 07Universal-Omega 03e7eeffa - Update MirahezeFunctions.php [16:56:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.93, 4.07, 4.08 [16:57:23] miraheze/mw-config - Universal-Omega the build passed. [16:57:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e7eeffadf9eb...8e2b36cfe316 [16:57:42] [url] Comparing e7eeffadf9eb...8e2b36cfe316 · miraheze/mw-config · GitHub | github.com [16:57:43] [02miraheze/mw-config] 07Universal-Omega 038e2b36c - Update MirahezeFunctions.php [16:57:45] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [16:57:45] [url] Page not found · GitHub · GitHub | github.com [16:57:52] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 31731 bytes in 2.624 second response time [16:58:14] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 3.451 second response time [16:58:25] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:58:39] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.020 second response time [16:58:50] miraheze/mw-config - Universal-Omega the build passed. [16:59:04] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.436 second response time [16:59:34] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 1.525 second response time [16:59:35] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 5.320 second response time [16:59:46] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw112 [17:00:04] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.473 second response time [17:00:20] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.419 second response time [17:01:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.79, 1.57, 1.66 [17:01:46] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [17:01:54] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [17:01:55] [url] Page not found · GitHub · GitHub | github.com [17:01:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8e2b36cfe316...2bfd638270e5 [17:01:56] [url] Comparing 8e2b36cfe316...2bfd638270e5 · miraheze/mw-config · GitHub | github.com [17:01:57] [02miraheze/mw-config] 07Universal-Omega 032bfd638 - Update MirahezeFunctions.php [17:02:09] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [17:02:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.79, 3.56, 3.88 [17:03:04] miraheze/mw-config - Universal-Omega the build passed. [17:03:32] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.64, 1.56, 1.64 [17:03:45] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:55] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:00] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.01, 3.53, 3.99 [17:04:24] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:31] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.84, 3.94 [17:04:37] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:41] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:41] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:49] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.248 second response time [17:05:59] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 3.75, 4.02 [17:07:46] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 4.677 second response time [17:08:09] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [17:08:39] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 0.950 second response time RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 1.429 second response time [17:08:40] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.750 second response time [17:08:58] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 4.484 second response time [17:09:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2bfd638270e5...cd73332626cf [17:09:01] [url] Comparing 2bfd638270e5...cd73332626cf · miraheze/mw-config · GitHub | github.com [17:09:02] [02miraheze/mw-config] 07Universal-Omega 03cd73332 - Update MirahezeFunctions.php [17:09:03] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [17:09:03] [url] Page not found · GitHub · GitHub | github.com [17:09:45] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [17:10:00] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.316 second response time [17:10:02] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.329 second response time [17:10:09] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [17:10:15] miraheze/mw-config - Universal-Omega the build passed. [17:10:16] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.345 second response time [17:10:34] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.024 second response time [17:10:39] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.027 second response time [17:11:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 4.51, 2.86, 2.10 [17:13:44] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [17:14:14] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:21] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:25] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [17:14:26] [url] Page not found · GitHub · GitHub | github.com [17:14:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cd73332626cf...a269273e7d6b [17:14:27] [url] Comparing cd73332626cf...a269273e7d6b · miraheze/mw-config · GitHub | github.com [17:14:28] [02miraheze/mw-config] 07Universal-Omega 03a269273 - revert [17:14:30] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.88, 3.97 [17:14:42] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:48] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:52] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:53] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:56] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:15:09] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [17:15:09] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:15:10] [url] Page not found · GitHub · GitHub | github.com [17:15:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a269273e7d6b...808fa3172f8e [17:15:11] [url] Comparing a269273e7d6b...808fa3172f8e · miraheze/mw-config · GitHub | github.com [17:15:12] [02miraheze/mw-config] 07Universal-Omega 03808fa31 - Update MirahezeFunctions.php [17:15:14] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:15:35] miraheze/mw-config - Universal-Omega the build has errored. [17:16:21] miraheze/mw-config - Universal-Omega the build passed. [17:17:11] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [17:17:12] [url] Page not found · GitHub · GitHub | github.com [17:17:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/808fa3172f8e...b9cfae4febe9 [17:17:13] [url] Comparing 808fa3172f8e...b9cfae4febe9 · miraheze/mw-config · GitHub | github.com [17:17:14] [02miraheze/mw-config] 07Universal-Omega 03b9cfae4 - Update MirahezeFunctions.php [17:17:37] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [17:18:23] miraheze/mw-config - Universal-Omega the build passed. [17:18:35] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Use the JsonReader composer library for reading config cache - 13https://github.com/miraheze/mw-config/pull/4757 [17:18:36] [url] Page not found · GitHub · GitHub | github.com [17:18:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b9cfae4febe9...b2ddc645cf60 [17:18:38] [url] Comparing b9cfae4febe9...b2ddc645cf60 · miraheze/mw-config · GitHub | github.com [17:18:38] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.331 second response time [17:18:38] [02miraheze/mw-config] 07Universal-Omega 03b2ddc64 - Update composer.json [17:18:46] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.021 second response time [17:18:50] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.634 second response time [17:18:51] [02mw-config] 07Universal-Omega edited pull request 03#4757: Improve cache performance - 13https://github.com/miraheze/mw-config/pull/4757 [17:18:51] [url] Page not found · GitHub · GitHub | github.com [17:18:51] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.035 second response time [17:19:09] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw111 mw112 [17:19:20] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 5.870 second response time [17:19:37] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [17:19:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.56, 3.47, 3.93 [17:19:48] miraheze/mw-config - Universal-Omega the build passed. [17:19:53] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 99069 bytes in 1.228 second response time [17:20:38] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 9.222 second response time [17:20:41] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 9.352 second response time [17:20:45] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.622 second response time [17:20:50] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 99069 bytes in 0.844 second response time [17:20:55] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 99077 bytes in 1.407 second response time [17:21:00] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.300 second response time [17:21:06] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 8.477 second response time [17:21:09] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [17:21:16] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 9.763 second response time [17:21:37] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 7.880 second response time [17:21:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.82, 3.92, 4.04 [17:24:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.78, 4.01, 3.81 [17:28:48] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:42] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.282 second response time [17:31:42] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.70, 10.14, 7.96 [17:32:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.28, 10.51, 11.89 [17:32:05] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.39, 5.83, 4.34 [17:32:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.27, 10.24, 11.73 [17:32:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.58, 3.91, 4.00 [17:32:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b2ddc645cf60...d5e7ea2171f1 [17:32:41] [url] Comparing b2ddc645cf60...d5e7ea2171f1 · miraheze/mw-config · GitHub | github.com [17:32:41] [02miraheze/mw-config] 07Universal-Omega 03d5e7ea2 - Update MirahezeFunctions.php [17:32:43] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Improve cache performance - 13https://github.com/miraheze/mw-config/pull/4757 [17:32:43] [url] Page not found · GitHub · GitHub | github.com [17:33:42] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.28, 9.98, 8.19 [17:33:52] miraheze/mw-config - Universal-Omega the build passed. [17:34:05] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.59, 5.87, 4.55 [17:34:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.30, 11.33, 11.94 [17:34:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.12, 4.06, 4.04 [17:35:34] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:35:39] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:35:53] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:36:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.04, 11.31, 11.88 [17:37:33] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.259 second response time [17:37:34] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 0.457 second response time [17:37:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.91, 1.63, 1.29 [17:37:48] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.026 second response time [17:38:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.57, 3.54, 3.87 [17:39:35] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.95, 1.34, 1.22 [17:39:36] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [17:40:05] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.40, 7.21, 5.51 [17:41:07] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.73, 9.80, 7.94 [17:45:07] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.09, 9.76, 8.42 [17:46:44] !log DELETED and DROPPED https://phabricator.miraheze.org/P466 (dbs that had no entries in cw_wikis) [17:46:45] [url] ✎ P466 Deleted wikis 15/06/2022 (Part 3 - databases with no cw_wikis entry) | phabricator.miraheze.org [17:47:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.05, 3.18, 3.79 [17:47:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:48:05] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.52, 7.51, 6.53 [17:48:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.19, 8.06, 9.83 [17:48:14] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.39, 8.18, 9.81 [17:48:53] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:18] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:19] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:21] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:23] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:30] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:31] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:52] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:50:04] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:50:04] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:50:29] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:50:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.09, 2.78, 3.40 [17:50:43] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:50:45] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:08] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:20] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 7.451 second response time [17:51:32] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 5.201 second response time [17:52:05] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.75, 6.13, 6.20 [17:52:29] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.036 second response time [17:52:38] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.402 second response time [17:52:41] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.692 second response time [17:53:02] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.030 second response time [17:53:04] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 9.792 second response time [17:53:14] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.023 second response time [17:53:16] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 0.237 second response time [17:53:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.61, 3.27, 3.57 [17:53:33] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.430 second response time [17:53:38] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 8.082 second response time [17:53:56] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 5.516 second response time [17:54:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.49, 9.25, 9.63 [17:54:14] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 2.478 second response time [17:54:14] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 2.721 second response time [17:54:54] [02mw-config] 07Universal-Omega edited pull request 03#4757: Improve cache performance - 13https://github.com/miraheze/mw-config/pull/4757 [17:54:54] [url] Page not found · GitHub · GitHub | github.com [17:55:21] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.96, 2.14, 1.55 [17:55:31] [02mw-config] 07Universal-Omega edited pull request 03#4757: Improve cache performance - 13https://github.com/miraheze/mw-config/pull/4757 [17:55:32] [url] Page not found · GitHub · GitHub | github.com [17:56:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.81, 8.91, 9.47 [17:58:05] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.71, 6.83, 6.48 [17:58:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.95, 3.33, 3.43 [17:59:17] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.19, 1.69, 1.51 [17:59:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.28, 3.64, 3.76 [18:00:05] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.35, 5.90, 6.18 [18:00:36] [02mw-config] 07Universal-Omega edited pull request 03#4757: Improve cache performance - 13https://github.com/miraheze/mw-config/pull/4757 [18:00:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.70, 3.07, 3.32 [18:00:37] [url] Page not found · GitHub · GitHub | github.com [18:01:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.95, 3.86 [18:02:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.68, 10.82, 10.18 [18:02:10] [02mw-config] 07Universal-Omega edited pull request 03#4757: Improve cache performance - 13https://github.com/miraheze/mw-config/pull/4757 [18:02:11] ... [18:02:47] [02mw-config] 07Universal-Omega edited pull request 03#4757: Improve cache performance - 13https://github.com/miraheze/mw-config/pull/4757 [18:02:47] ... [18:02:51] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.50, 2.86, 2.20 [18:03:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.97, 1.90, 2.00 [18:04:05] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.26, 7.89, 6.94 [18:04:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.02, 10.11, 9.20 [18:04:48] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.98, 2.94, 2.31 [18:05:02] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:05:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 4.77, 2.76, 2.29 [18:06:05] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.90, 7.18, 6.80 [18:06:24] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.94, 9.71, 9.16 [18:06:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.31, 3.35 [18:07:16] PROBLEM - test101 APT on test101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [18:07:40] !log [@test101] starting deploy of {'config': True} to all [18:07:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:07:52] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 12s [18:08:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.20, 11.03, 10.41 [18:08:05] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.38, 6.41, 6.56 [18:08:12] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:08:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:08:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.86, 3.29, 3.34 [18:09:21] RECOVERY - test101 APT on test101 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [18:10:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.19, 10.68, 10.39 [18:10:12] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 32410 bytes in 1.700 second response time [18:11:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.62, 3.66, 3.87 [18:12:02] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.42, 1.80, 1.57 [18:12:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.77, 9.97, 10.15 [18:12:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.26, 3.74, 3.51 [18:13:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d5e7ea2171f1...e30dda4dd6be [18:13:03] [url] Comparing d5e7ea2171f1...e30dda4dd6be · miraheze/mw-config · GitHub | github.com [18:13:03] [02miraheze/mw-config] 07Universal-Omega 03e30dda4 - Update MirahezeFunctions.php [18:13:05] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Improve cache performance - 13https://github.com/miraheze/mw-config/pull/4757 [18:13:05] [url] Page not found · GitHub · GitHub | github.com [18:13:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.24, 4.72, 4.21 [18:14:00] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.22, 1.56, 1.51 [18:14:16] miraheze/mw-config - Universal-Omega the build passed. [18:15:29] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:16:02] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Improve cache performance - 13https://github.com/miraheze/mw-config/pull/4757 [18:16:02] [url] Page not found · GitHub · GitHub | github.com [18:16:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e30dda4dd6be...d0b54a30c82c [18:16:04] [url] Comparing e30dda4dd6be...d0b54a30c82c · miraheze/mw-config · GitHub | github.com [18:16:05] [02miraheze/mw-config] 07Universal-Omega 03d0b54a3 - Update MirahezeFunctions.php [18:16:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.69, 10.60, 10.34 [18:16:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.97, 3.46, 3.52 [18:17:14] miraheze/mw-config - Universal-Omega the build passed. [18:17:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.48, 10.95, 10.14 [18:17:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.42, 3.55, 3.85 [18:19:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.69, 11.08, 10.28 [18:20:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.92, 10.46, 8.80 [18:20:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.87, 3.16, 3.40 [18:21:15] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.87, 10.18, 10.04 [18:22:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.04, 9.38, 8.60 [18:22:57] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.69, 6.79, 6.53 [18:23:10] PROBLEM - test101 Check Gluster Clients on test101 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [18:23:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 0.88, 0.34 [18:24:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.17, 11.09, 10.69 [18:25:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.05, 3.63, 3.68 [18:25:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.56, 1.15, 0.51 [18:26:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.55, 10.65, 10.57 [18:26:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.37, 3.82, 3.58 [18:26:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.33, 2.04, 1.71 [18:26:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.56, 7.31, 6.85 [18:27:13] [02mw-config] 07Universal-Omega closed pull request 03#4757: Improve cache performance - 13https://github.com/miraheze/mw-config/pull/4757 [18:27:13] [url] Page not found · GitHub · GitHub | github.com [18:27:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/04b5839acda1...5fdb8e05c9de [18:27:15] [url] Comparing 04b5839acda1...5fdb8e05c9de · miraheze/mw-config · GitHub | github.com [18:27:16] [02miraheze/mw-config] 07Universal-Omega 035fdb8e0 - Improve cache performance (#4757) [18:27:17] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [18:27:19] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [18:27:19] [url] Page not found · GitHub · GitHub | github.com [18:27:49] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [18:28:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:14] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s [18:28:20] miraheze/mw-config - Universal-Omega the build passed. [18:28:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.86, 9.92, 8.94 [18:28:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.47, 1.90, 1.70 [18:28:54] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.45, 6.61, 6.64 [18:30:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.99, 9.65, 10.20 [18:30:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.10, 8.39, 8.49 [18:30:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.16, 3.80, 3.66 [18:32:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 4.07, 3.78 [18:32:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.75, 2.33, 1.89 [18:34:40] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.70, 1.98, 1.81 [18:36:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.31, 2.01, 1.84 [18:38:40] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.51, 1.81, 1.79 [18:40:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.84, 2.04, 1.87 [18:41:10] RECOVERY - test101 Check Gluster Clients on test101 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [18:42:40] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.41, 1.75, 1.78 [18:43:08] [02puppet] 07Universal-Omega edited pull request 03#2669: mediawiki: use REL1_38 - 13https://github.com/miraheze/puppet/pull/2669 [18:43:09] [url] Page not found · GitHub · GitHub | github.com [18:43:38] [02mw-config] 07Universal-Omega edited pull request 03#4754: Set all wikis read-only for MediaWiki 1.38 upgrade - 13https://github.com/miraheze/mw-config/pull/4754 [18:43:38] ... [18:44:08] [02puppet] 07paladox closed pull request 03#2671: upgradetool: switch to new mass plan - 13https://github.com/miraheze/puppet/pull/2671 [18:44:08] ... [18:44:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/acea0806300b...c583832aae65 [18:44:10] [url] Comparing acea0806300b...c583832aae65 · miraheze/puppet · GitHub | github.com [18:44:11] [02miraheze/puppet] 07RhinosF1 03c583832 - upgradetool: switch to new mass plan (#2671) [18:44:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.12, 1.52, 1.69 [18:44:57] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:46:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.83, 3.72, 3.85 [18:47:04] !log upgrade kernel on db101/111/112/121 [18:47:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:48:51] RECOVERY - db101 APT on db101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:49:36] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [18:49:59] RECOVERY - db112 APT on db112 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:50:35] RECOVERY - db111 APT on db111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:50:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.02, 3.54, 3.72 [18:50:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.36, 2.03, 1.83 [18:50:41] !log upgrade kernel on gluster101/111/121 [18:50:45] [02mw-config] 07Universal-Omega assigned pull request 03#4758: Sitenotice for upgrade - 13https://github.com/miraheze/mw-config/pull/4758 [18:50:45] [url] Page not found · GitHub · GitHub | github.com [18:50:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:50:55] [02mw-config] 07Universal-Omega assigned pull request 03#4754: Set all wikis read-only for MediaWiki 1.38 upgrade - 13https://github.com/miraheze/mw-config/pull/4754 [18:50:55] [url] Page not found · GitHub · GitHub | github.com [18:51:02] [02mw-config] 07Universal-Omega assigned pull request 03#4745: Add ImportDump extension - 13https://github.com/miraheze/mw-config/pull/4745 [18:51:03] [url] Page not found · GitHub · GitHub | github.com [18:51:12] [02mw-config] 07Universal-Omega assigned pull request 03#4741: Update skin settings for MediaWiki 1.38 - 13https://github.com/miraheze/mw-config/pull/4741 [18:51:12] ... [18:51:53] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.89, 2.11, 1.59 [18:52:40] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.81, 2.00, 1.85 [18:52:41] RECOVERY - db121 APT on db121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:53:16] RECOVERY - gluster101 APT on gluster101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:53:18] !log upgrade kernel on es101/111/121 [18:53:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:49] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.85, 1.68, 1.50 [18:55:05] RECOVERY - es101 APT on es101 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [18:55:08] RECOVERY - gluster121 APT on gluster121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:55:37] !log upgrade kernel on ns2 [18:55:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5fdb8e05c9de...709bf597e96b [18:55:40] [url] Comparing 5fdb8e05c9de...709bf597e96b · miraheze/mw-config · GitHub | github.com [18:55:41] [02miraheze/mw-config] 07Universal-Omega 03709bf59 - Fix `$wgNamespaceRobotPolicies` for the special namespace [18:55:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:42] miraheze/mw-config - Universal-Omega the build passed. [18:56:48] RECOVERY - gluster111 APT on gluster111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:57:18] RECOVERY - es121 APT on es121 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [18:57:48] RECOVERY - es111 APT on es111 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [18:58:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.48, 1.57, 1.70 [18:58:52] RECOVERY - ns2 APT on ns2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:02:41] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.69, 1.67, 1.71 [19:03:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.42, 10.40, 9.20 [19:04:08] Is piwik still on db112? [19:04:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.43, 1.65, 1.70 [19:05:18] paladox: ^^ (you were workin on that) [19:05:29] JohnLewis: yes [19:05:42] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 10.08, 9.92, 9.15 [19:05:50] Good, we’ll work a solution out as moving it off 112 would be more harmful than productive [19:05:52] it took too long to copy to db111 before the maintenance window [19:06:03] oh? [19:06:41] Well the main reason we moved it was it was generating a significant amount of traffic that was bringing 111 down regularly [19:07:22] JohnLewis: so how do we fix the issue of it eating all the disk space on 112 and bringing that down [19:07:40] Receptions proposal is essentially it [19:07:57] There’s a ridiculous amount of data being stored, most of which we don’t need anymore [19:08:14] Plus we store everything twice [19:08:21] i had a look and the recent data is like march 2022 [19:08:28] i saw nothing before that date [19:08:37] in log and action table [19:08:50] (the two that are eating up most of the disk space) [19:09:30] We’ll have to reduce retention then of that data potentially [19:10:03] ah yeh [19:10:08] i was looking at the settings [19:10:19] didn't really want to change them incase - leaving that up to you [19:10:25] yeah the issue with my proposal was that we didn't really find a way to do it [19:10:27] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.12, 6.49, 6.02 [19:10:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.03, 1.92, 1.78 [19:11:14] I doubt we really use that kind of data too much - we should be able to reduce it as a stop gap at least anyway. If we really need it, it can wait until we have disk space [19:12:27] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.63, 6.39, 6.03 [19:12:31] The only thing I'd like us to retain (or at least save somehow) are mostly unique visits/mo [19:12:37] as I think it's interesting for us to compare those [19:12:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.93, 1.64, 1.70 [19:12:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.47, 9.84, 8.95 [19:12:55] but worst case those could just be screenshotted and saved somewhere [19:14:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.63, 9.69, 9.03 [19:14:43] Reception123: they’re stored separately and won’t be something we’d look to remove at all [19:14:58] oh [19:15:04] RECOVERY - mem101 APT on mem101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:15:05] but then what's taking up most of the 77GB? [19:15:24] !log upgrade kernel on mem101/121 [19:15:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:15:52] Log event information like browser data etc I imagine? [19:16:20] !log upgrade kernel on reports121 [19:16:23] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.13, 7.34, 6.50 [19:16:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:16:37] !log upgrade kernel on ns1 and cp* [19:16:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.44, 1.92, 1.79 [19:16:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:17:26] RECOVERY - reports121 APT on reports121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:18:20] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:18:22] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.19, 6.85, 6.42 [19:18:40] RECOVERY - mem121 APT on mem121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:18:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.44, 1.62, 1.69 [19:18:55] RECOVERY - cp30 APT on cp30 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:18:57] !log [@test101] starting deploy of {'config': True} to all [19:18:58] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [19:19:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:35] RECOVERY - cp31 APT on cp31 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:20:03] RECOVERY - cp20 APT on cp20 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:20:20] RECOVERY - cp21 APT on cp21 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:20:21] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.80, 6.78, 6.44 [19:20:37] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.09, 2.18, 1.47 [19:21:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.41, 10.12, 9.63 [19:22:14] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 4183 MB (10% inode=96%); [19:22:17] !log [@mwtask111] starting deploy of {'config': True} to all [19:22:30] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 12s [19:22:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.28, 10.00, 9.34 [19:22:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:24:18] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.25, 7.78, 6.91 [19:26:33] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.25, 1.81, 1.56 [19:29:06] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [19:30:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.74, 10.01, 9.73 [19:30:33] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.50, 1.30, 1.42 [19:30:46] [02puppet] 07Universal-Omega opened pull request 03#2673: mwupgradetool: Use deploy-mediawiki to sync config - 13https://github.com/miraheze/puppet/pull/2673 [19:30:46] [url] Page not found · GitHub · GitHub | github.com [19:30:59] RhinosF1: ^ [19:34:12] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.00, 7.49, 7.31 [19:34:24] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.36, 9.95, 10.05 [19:36:11] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.79, 7.99, 7.51 [19:36:50] [02mw-config] 07Universal-Omega closed pull request 03#4744: remove separate allowEntityImport from gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/4744 [19:36:50] [url] Page not found · GitHub · GitHub | github.com [19:36:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/709bf597e96b...a36468a48747 [19:36:52] [url] Comparing 709bf597e96b...a36468a48747 · miraheze/mw-config · GitHub | github.com [19:36:53] [02miraheze/mw-config] 07ugochimobi 03a36468a - remove separate allowEntityImport from gratisdatawiki (#4744) [19:38:04] miraheze/mw-config - Universal-Omega the build passed. [19:38:13] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.26, 6.96, 7.19 [19:39:06] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [19:39:32] [02mw-config] 07Universal-Omega reviewed pull request 03#4730 commit - 13https://github.com/miraheze/mw-config/pull/4730#discussion_r898344820 [19:39:33] [url] T9334: set up wgDefaultLanguageVariant and wgDisabledVariants for hkrailwiki by ugochimobi · Pull Request #4730 · miraheze/mw-config · GitHub | github.com [19:41:19] [02mw-config] 07Universal-Omega reviewed pull request 03#4728 commit - 13https://github.com/miraheze/mw-config/pull/4728#discussion_r898345992 [19:41:20] [url] T9328: set base of wgMinervaShowCategories to true for criticalrolewiki by ugochimobi · Pull Request #4728 · miraheze/mw-config · GitHub | github.com [19:41:35] [02mw-config] 07Universal-Omega reviewed pull request 03#4728 commit - 13https://github.com/miraheze/mw-config/pull/4728#discussion_r898346378 [19:41:35] [url] T9328: set base of wgMinervaShowCategories to true for criticalrolewiki by ugochimobi · Pull Request #4728 · miraheze/mw-config · GitHub | github.com [19:44:10] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.18, 5.67, 6.60 [19:46:03] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.22, 3.21, 2.68 [19:48:03] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.64, 2.90, 2.62 [19:49:13] !log [@test101] starting deploy of {'config': True} to all [19:49:14] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [19:49:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:59] !log [@mwtask111] starting deploy of {'config': True} to all [19:52:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:07] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 7s [19:52:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.20, 2.41, 1.70 [19:57:07] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.71, 1.58, 1.39 [19:57:22] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.01, 1.89, 1.59 [19:58:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/b6c7f2204984 [19:58:18] [02miraheze/mw-config] 07Universal-Omega 03b6c7f22 - Fix [19:58:19] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [19:58:20] [url] Page not found · GitHub · GitHub | github.com [19:58:21] [02mw-config] 07Universal-Omega opened pull request 03#4759: Fix - 13https://github.com/miraheze/mw-config/pull/4759 [19:58:22] [url] Page not found · GitHub · GitHub | github.com [19:58:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b6c7f2204984...fb8a659eae54 [19:58:44] [url] Comparing b6c7f2204984...fb8a659eae54 · miraheze/mw-config · GitHub | github.com [19:58:44] [02miraheze/mw-config] 07Universal-Omega 03fb8a659 - Update LocalSettings.php [19:58:46] [02mw-config] 07Universal-Omega synchronize pull request 03#4759: Fix - 13https://github.com/miraheze/mw-config/pull/4759 [19:58:46] [url] Page not found · GitHub · GitHub | github.com [19:59:05] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.22, 1.46, 1.37 [19:59:23] [02mw-config] 07Universal-Omega closed pull request 03#4759: Fix - 13https://github.com/miraheze/mw-config/pull/4759 [19:59:24] [url] Page not found · GitHub · GitHub | github.com [19:59:25] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [19:59:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/a36468a48747...7d6c2ffe2664 [19:59:27] [url] Comparing a36468a48747...7d6c2ffe2664 · miraheze/mw-config · GitHub | github.com [19:59:28] [02miraheze/mw-config] 07Universal-Omega 037d6c2ff - Fix (#4759) [19:59:29] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [19:59:30] [url] Page not found · GitHub · GitHub | github.com [19:59:34] miraheze/mw-config - Universal-Omega the build passed. [19:59:37] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [19:59:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:47] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [19:59:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:54] miraheze/mw-config - Universal-Omega the build passed. [20:00:27] miraheze/mw-config - Universal-Omega the build passed. [20:00:34] !log gluster: gluster volume set static read-only on [20:00:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:16] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.18, 1.53, 1.52 [20:01:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7d6c2ffe2664...a97e3dec19a6 [20:01:38] [url] Comparing 7d6c2ffe2664...a97e3dec19a6 · miraheze/mw-config · GitHub | github.com [20:01:39] [02miraheze/mw-config] 07Universal-Omega 03a97e3de - Set all wikis read-only for MediaWiki 1.38 upgrade (#4754) [20:01:40] [02mw-config] 07Universal-Omega closed pull request 03#4754: Set all wikis read-only for MediaWiki 1.38 upgrade - 13https://github.com/miraheze/mw-config/pull/4754 [20:01:41] ... [20:01:42] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://github.com/miraheze/mw-config [20:01:42] [url] Page not found · GitHub · GitHub | github.com [20:01:43] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 [20:01:45] [02mw-config] 07Universal-Omega closed pull request 03#4758: Sitenotice for upgrade - 13https://github.com/miraheze/mw-config/pull/4758 [20:01:45] [url] Page not found · GitHub · GitHub | github.com [20:01:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a97e3dec19a6...97abc790c953 [20:01:47] [url] Comparing a97e3dec19a6...97abc790c953 · miraheze/mw-config · GitHub | github.com [20:01:48] [02miraheze/mw-config] 07AgentIsai 0397abc79 - Sitenotice for upgrade (#4758) [20:02:53] miraheze/mw-config - Universal-Omega the build passed. [20:02:58] .op [20:02:58] Attempting to OP... [20:03:25] puppet in progress [20:03:42] !log [@mwtask111] starting deploy of {'config': True} to all [20:03:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:50] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 8s [20:04:26] !log read-only for mw* confirmed [20:04:34] paladox: ^ good to go [20:04:43] ack, gonna stop vms now [20:04:51] Reception123: want to handle cp* and ns1 reboot? [20:05:18] paladox: sure [20:05:23] tell me when to reboot [20:06:11] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.91, 1.81, 1.40 [20:06:32] uhh, i think we need to open port 22 on cloud* so we don't require bast [20:06:44] yeah I can't ssh to cp [20:06:45] JohnLewis: ^ [20:07:06] well [20:07:10] maybe i can just use bast121 [20:07:14] PROBLEM - bast101 Disk Space on bast101 is CRITICAL: connect to address 2a10:6740::6:101 port 5666: No route to hostconnect to host 2a10:6740::6:101 port 5666: No route to host [20:07:18] then bast10 [20:07:18] or that ye [20:07:20] *101 [20:07:29] just don't kill them both at once [20:07:31] PROBLEM - bast101 conntrack_table_size on bast101 is CRITICAL: connect to address 2a10:6740::6:101 port 5666: No route to hostconnect to host 2a10:6740::6:101 port 5666: No route to host [20:07:31] PROBLEM - bast101 Puppet on bast101 is CRITICAL: connect to address 2a10:6740::6:101 port 5666: No route to hostconnect to host 2a10:6740::6:101 port 5666: No route to host [20:07:42] !log stopping all vms on cloud10 [20:07:49] paladox: can we downtime icinga alerts [20:08:02] do renember that !log will do nothing right now with RO [20:08:05] opening port 22 on cloud is the one server you don't want public access to afterall :P [20:08:11] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.83, 1.46, 1.32 [20:08:15] PROBLEM - bast101 PowerDNS Recursor on bast101 is CRITICAL: connect to address 2a10:6740::6:101 port 5666: No route to hostconnect to host 2a10:6740::6:101 port 5666: No route to host [20:08:15] PROBLEM - ping6 on bast101 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:101) [20:08:19] we can shut ircecho [20:08:22] PROBLEM - bast101 ferm_active on bast101 is CRITICAL: connect to address 2a10:6740::6:101 port 5666: No route to hostconnect to host 2a10:6740::6:101 port 5666: No route to host [20:08:22] PROBLEM - bast101 SSH on bast101 is CRITICAL: connect to address 2a10:6740::6:101 and port 22: No route to host [20:08:22] PROBLEM - gluster101 Disk Space on gluster101 is CRITICAL: connect to address 2a10:6740::6:104 port 5666: No route to hostconnect to host 2a10:6740::6:104 port 5666: No route to host [20:08:31] PROBLEM - bast101 APT on bast101 is CRITICAL: connect to address 2a10:6740::6:101 port 5666: No route to hostconnect to host 2a10:6740::6:101 port 5666: No route to host [20:08:32] yeah though still good to !log just to have an easy record in case something goes wrong [20:09:29] Thanks paladox [20:09:44] i'm just waiting for db101 to shut now [20:09:46] Please still !log as Reception123 says [20:09:47] then i can do the update [20:09:52] Cool [20:09:54] i have [20:10:03] ok its stopped [20:10:05] updating [20:10:22] !log upgrade proxmox and kernel on cloud10 [20:12:31] !log reboot cloud10 [20:12:53] I am having problems [20:16:16] [discord] We are upgrading MediaWiki, as the sitenotice on all wikis for the last week has informed users. During this time, you may be unable to access your wiki. [20:16:32] Agent: they already quit [20:17:21] [discord] I should hop on IRC heh [20:17:29] hey [20:17:36] mediawiki!? [20:17:40] "We are upgrading MediaWiki, as the sitenotice on all wikis for the last week has informed users. During this time, you may be unable to access your wiki." [20:17:51] Guest84: ^^ [20:18:01] thank you [20:20:14] cloud10 is back [20:20:19] kept gluster down as agreed [20:20:26] now doing cloud11 [20:21:10] !log started all vms on cloud10 [20:21:15] !log stopped all vms on cloud11 [20:22:39] !log upgrade proxmox and kernel on cloud11 [20:30:10] !log reboot cloud11 [20:37:48] !log start all vms on cloud1 [20:37:52] !log start all vms on cloud11 [20:38:40] PROBLEM - gluster111 Disk Space on gluster111 is CRITICAL: connect to address 2a10:6740::6:203 port 5666: No route to hostconnect to host 2a10:6740::6:203 port 5666: No route to host [20:38:40] RECOVERY - Host matomo101 is UP: PING OK - Packet loss = 0%, RTA = 1.29 ms [20:38:40] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23272 bytes in 0.375 second response time [20:38:40] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset 0.001173377037 secs [20:38:42] RECOVERY - db101 NTP time on db101 is OK: NTP OK: Offset -0.005226016045 secs [20:38:44] !log stop all vms on cloud12 [20:38:46] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.251 second response time [20:38:46] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.249 second response time [20:38:46] RECOVERY - ping6 on db101 is OK: PING OK - Packet loss = 0%, RTA = 1.10 ms [20:38:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [20:38:48] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 1.33, 1.24, 1.07 [20:38:51] RECOVERY - cp30 Stunnel HTTP for test101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15895 bytes in 0.319 second response time [20:38:51] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:38:53] RECOVERY - Host mw102 is UP: PING OK - Packet loss = 0%, RTA = 4.08 ms [20:38:53] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 24% [20:38:53] RECOVERY - db101 MariaDB on db101 is OK: Uptime: 1106 Threads: 62 Questions: 472429 Slow queries: 27986 Opens: 12095 Open tables: 12089 Queries per second avg: 427.150 [20:38:54] RECOVERY - mem101 Current Load on mem101 is OK: OK - load average: 0.15, 0.06, 0.06 [20:38:55] RECOVERY - Host es101 is UP: PING OK - Packet loss = 0%, RTA = 8.14 ms [20:38:56] RECOVERY - es101 PowerDNS Recursor on es101 is OK: DNS OK: 0.164 seconds response time. miraheze.org returns 198.244.148.90,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a,51.195.220.68 [20:38:56] PROBLEM - mw102 Check Gluster Clients on mw102 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [20:38:56] RECOVERY - es101 conntrack_table_size on es101 is OK: OK: nf_conntrack is 0 % full [20:38:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.83, 6.43, 4.31 [20:38:56] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: connect to address 2a10:6740::6:106 port 5666: No route to hostconnect to host 2a10:6740::6:106 port 5666: No route to host [20:38:56] RECOVERY - es101 APT on es101 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [20:38:57] paladox: might wanna stop ircecho again [20:38:57] RECOVERY - es101 NTP time on es101 is OK: NTP OK: Offset 0.0006962716579 secs [20:38:57] RECOVERY - ping6 on es101 is OK: PING OK - Packet loss = 0%, RTA = 1.77 ms [20:38:58] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 9 processes with command name 'php-fpm7.4' [20:39:04] done [20:41:55] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:42:28] PROBLEM - mw121 nutcracker process on mw121 is CRITICAL: connect to address 2a10:6740::6:309 port 5666: No route to hostconnect to host 2a10:6740::6:309 port 5666: No route to host [20:42:29] PROBLEM - mw122 php-fpm on mw122 is CRITICAL: connect to address 2a10:6740::6:310 port 5666: No route to hostconnect to host 2a10:6740::6:310 port 5666: No route to host [20:42:29] PROBLEM - mw122 JobRunner Service on mw122 is CRITICAL: connect to address 2a10:6740::6:310 port 5666: No route to hostconnect to host 2a10:6740::6:310 port 5666: No route to host [20:42:32] PROBLEM - mw121 Disk Space on mw121 is CRITICAL: connect to address 2a10:6740::6:309 port 5666: No route to hostconnect to host 2a10:6740::6:309 port 5666: No route to host [20:42:38] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) PROBLEM - db121 Disk Space on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: No route to hostconnect to host 2a10:6740::6:302 port 5666: No route to host [20:42:38] PROBLEM - db121 SSH on db121 is CRITICAL: connect to address 2a10:6740::6:302 and port 22: No route to host [20:42:39] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 1.90, 4.23, 2.16 [20:42:39] PROBLEM - mw122 PowerDNS Recursor on mw122 is CRITICAL: connect to address 2a10:6740::6:310 port 5666: No route to hostconnect to host 2a10:6740::6:310 port 5666: No route to host [20:42:41] PROBLEM - mw121 NTP time on mw121 is CRITICAL: connect to address 2a10:6740::6:309 port 5666: No route to hostconnect to host 2a10:6740::6:309 port 5666: No route to host [20:42:41] PROBLEM - mw121 Puppet on mw121 is CRITICAL: connect to address 2a10:6740::6:309 port 5666: No route to hostconnect to host 2a10:6740::6:309 port 5666: No route to host [20:42:41] PROBLEM - graylog121 HTTPS on graylog121 is CRITICAL: connect to address 2a10:6740::6:305 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [20:42:42] !log upgrade proxmox and kernel on cloud12 [20:42:43] PROBLEM - mem121 APT on mem121 is CRITICAL: connect to address 2a10:6740::6:308 port 5666: No route to hostconnect to host 2a10:6740::6:308 port 5666: No route to host [20:42:43] PROBLEM - mem121 conntrack_table_size on mem121 is CRITICAL: connect to address 2a10:6740::6:308 port 5666: No route to hostconnect to host 2a10:6740::6:308 port 5666: No route to host [20:42:44] RECOVERY - mw111 Check Gluster Clients on mw111 is OK: PROCS OK: 2 processes with args '/usr/sbin/glusterfs' [20:42:44] PROBLEM - bast121 conntrack_table_size on bast121 is CRITICAL: connect to address 2a10:6740::6:301 port 5666: No route to hostconnect to host 2a10:6740::6:301 port 5666: No route to host [20:42:44] PROBLEM - ping6 on bast121 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:301) [20:42:44] PROBLEM - bast121 SSH on bast121 is CRITICAL: connect to address 2a10:6740::6:301 and port 22: No route to host [20:42:44] PROBLEM - bast121 ferm_active on bast121 is CRITICAL: connect to address 2a10:6740::6:301 port 5666: No route to hostconnect to host 2a10:6740::6:301 port 5666: No route to host [20:42:44] PROBLEM - bast121 NTP time on bast121 is CRITICAL: connect to address 2a10:6740::6:301 port 5666: No route to hostconnect to host 2a10:6740::6:301 port 5666: No route to host [20:42:45] PROBLEM - db121 PowerDNS Recursor on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: No route to hostconnect to host 2a10:6740::6:302 port 5666: No route to host [20:42:45] PROBLEM - db121 Current Load on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: No route to hostconnect to host 2a10:6740::6:302 port 5666: No route to host [20:42:57] PROBLEM - Host gluster121 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:304) [20:42:57] PROBLEM - gluster121 APT on gluster121 is CRITICAL: connect to address 2a10:6740::6:304 port 5666: No route to hostconnect to host 2a10:6740::6:304 port 5666: No route to host [20:42:58] RECOVERY - es111 Puppet on es111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:42:58] PROBLEM - es121 conntrack_table_size on es121 is CRITICAL: connect to address 2a10:6740::6:303 port 5666: No route to hostconnect to host 2a10:6740::6:303 port 5666: No route to host [20:42:59] PROBLEM - es121 ferm_active on es121 is CRITICAL: connect to address 2a10:6740::6:303 port 5666: No route to hostconnect to host 2a10:6740::6:303 port 5666: No route to host [20:42:59] PROBLEM - db121 conntrack_table_size on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: No route to hostconnect to host 2a10:6740::6:302 port 5666: No route to host [20:43:00] PROBLEM - reports121 Disk Space on reports121 is CRITICAL: connect to address 2a10:6740::6:312 port 5666: No route to hostconnect to host 2a10:6740::6:312 port 5666: No route to host [20:43:00] PROBLEM - Host reports121 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:312) [20:43:01] PROBLEM - ping6 on reports121 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:312) [20:43:01] PROBLEM - cp31 Stunnel HTTP for reports121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:02] PROBLEM - jobchron121 Redis Process on jobchron121 is CRITICAL: connect to address 2a10:6740::6:306 port 5666: No route to hostconnect to host 2a10:6740::6:306 port 5666: No route to host [20:43:02] PROBLEM - jobchron121 SSH on jobchron121 is CRITICAL: connect to address 2a10:6740::6:306 and port 22: No route to host [20:43:02] hmm [20:43:03] PROBLEM - jobchron121 ferm_active on jobchron121 is CRITICAL: connect to address 2a10:6740::6:306 port 5666: No route to hostconnect to host 2a10:6740::6:306 port 5666: No route to host [20:43:03] PROBLEM - mem121 ferm_active on mem121 is CRITICAL: connect to address 2a10:6740::6:308 port 5666: No route to hostconnect to host 2a10:6740::6:308 port 5666: No route to host [20:43:04] PROBLEM - bast121 Current Load on bast121 is CRITICAL: connect to address 2a10:6740::6:301 port 5666: No route to hostconnect to host 2a10:6740::6:301 port 5666: No route to host [20:43:04] PROBLEM - es121 Disk Space on es121 is CRITICAL: connect to address 2a10:6740::6:303 port 5666: No route to hostconnect to host 2a10:6740::6:303 port 5666: No route to host [20:43:05] PROBLEM - Host es121 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:303) [20:43:05] PROBLEM - mw122 SSH on mw122 is CRITICAL: connect to address 2a10:6740::6:310 and port 22: No route to host [20:43:17] PROBLEM - mem121 memcached on mem121 is CRITICAL: connect to address 2a10:6740::6:308 and port 11211: No route to host [20:43:17] PROBLEM - Host mem121 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:308) [20:44:40] paladox: does puppet try and restart ircecho? [20:44:47] yeh [20:44:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f05afb442636 [20:45:00] [02miraheze/puppet] 07paladox 03f05afb4 - gluster: upgrade to version 10 [20:45:01] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [20:45:02] [url] Page not found · GitHub · GitHub | github.com [20:45:03] [02puppet] 07paladox opened pull request 03#2674: gluster: upgrade to version 10 - 13https://github.com/miraheze/puppet/pull/2674 [20:45:03] [url] Page not found · GitHub · GitHub | github.com [20:45:06] !log reboot cloud12 [20:45:35] perhaps +q icinga-miraheze? [20:46:20] paladox: maybe stop puppet on Mon? [20:46:42] done [20:47:03] !log paladox: disabled puppet on Mon [20:48:41] the other solution would've been to do a +q but meh [20:50:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c583832aae65...6fa1c716afa0 [20:50:03] [url] Comparing c583832aae65...6fa1c716afa0 · miraheze/puppet · GitHub | github.com [20:50:04] [02miraheze/puppet] 07paladox 036fa1c71 - gluster: upgrade to version 10 (#2674) [20:50:05] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [20:50:07] [02puppet] 07paladox closed pull request 03#2674: gluster: upgrade to version 10 - 13https://github.com/miraheze/puppet/pull/2674 [20:50:07] [url] Page not found · GitHub · GitHub | github.com [20:50:08] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [20:50:09] ... [20:52:12] !log start all vms on cloud12 [21:01:04] mw102, and mw112 still can't access wikis (503). all others can. [21:07:12] hmm ssl errors with gluster [21:10:45] Full error? [21:12:46] CosmicAlpha: you can work on mw upgrade [21:12:56] i'm still handling the gluster issue [21:13:06] they changed the path to load ssl to /var/lib/ssl [21:14:49] paladox: OK, but mw112, and mw102 still give 503s. My deploy might fail. The rest of me servers work though. [21:15:06] hmm, i'm not sure why either of those fail [21:21:43] CosmicAlpha: upgrade tool uses --force [21:22:02] It might scream at you though [21:22:08] RhinosF1: using --force still fails sometimes on test101. [21:22:12] Or check_up might go funny [21:22:48] CosmicAlpha: might be worth waiting to be honest [21:27:38] it works! [21:30:13] I'm off to sleep in next 10 minutes or so [21:34:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ff9605286c7e [21:34:17] [02miraheze/puppet] 07paladox 03ff96052 - gluster: fix ssl settings for mount [21:34:19] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [21:34:19] [url] Page not found · GitHub · GitHub | github.com [21:34:20] [02puppet] 07paladox opened pull request 03#2675: gluster: fix ssl settings for mount - 13https://github.com/miraheze/puppet/pull/2675 [21:34:21] [url] Page not found · GitHub · GitHub | github.com [21:34:45] CosmicAlpha: is it worth considering starting the prep on task via upgrade tool if it works [21:34:56] So you don't have to wait as long [21:35:11] sure [21:35:43] That could at least make sure REL1_38 is cloned [21:35:55] puppet will fail probably though. [21:35:59] I'm off to sleep though [21:36:02] Fun [21:37:31] RhinosF1: Yeah, like I thought, it fails read only check because it can't check to those servers if they are down. [21:37:44] CosmicAlpha: ah :( [21:37:54] Can you not just press enter to go past [21:37:58] Or does it except? [21:38:05] it gives exception [21:38:12] Urgh [21:38:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ff9605286c7e...c1cbfb58963b [21:38:22] [url] Comparing ff9605286c7e...c1cbfb58963b · miraheze/puppet · GitHub | github.com [21:38:23] [02miraheze/puppet] 07paladox 03c1cbfb5 - Update init.pp [21:38:25] [02puppet] 07paladox synchronize pull request 03#2675: gluster: fix ssl settings for mount - 13https://github.com/miraheze/puppet/pull/2675 [21:38:25] ... [21:38:47] CosmicAlpha: I'm off to sleep [21:38:55] I guess you wait then until they're fixed [21:38:59] Ok, night. [21:39:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c1cbfb58963b...5d997959f827 [21:39:09] [url] Comparing c1cbfb58963b...5d997959f827 · miraheze/puppet · GitHub | github.com [21:39:10] [02miraheze/puppet] 07paladox 035d99795 - Update mount.pp [21:39:11] [02puppet] 07paladox synchronize pull request 03#2675: gluster: fix ssl settings for mount - 13https://github.com/miraheze/puppet/pull/2675 [21:39:12] [url] Page not found · GitHub · GitHub | github.com [21:39:19] [02puppet] 07paladox edited pull request 03#2675: gluster: fix ssl path - 13https://github.com/miraheze/puppet/pull/2675 [21:39:19] [url] Page not found · GitHub · GitHub | github.com [21:39:29] [02puppet] 07paladox synchronize pull request 03#2675: gluster: fix ssl path - 13https://github.com/miraheze/puppet/pull/2675 [21:39:29] [url] Page not found · GitHub · GitHub | github.com [21:39:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5d997959f827...ad235bfedce0 [21:39:31] [url] Comparing 5d997959f827...ad235bfedce0 · miraheze/puppet · GitHub | github.com [21:39:32] [02miraheze/puppet] 07paladox 03ad235bf - Update secure-access [21:39:38] [02puppet] 07paladox closed pull request 03#2675: gluster: fix ssl path - 13https://github.com/miraheze/puppet/pull/2675 [21:39:38] [url] Page not found · GitHub · GitHub | github.com [21:39:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/6fa1c716afa0...61614c6c5845 [21:39:40] [url] Comparing 6fa1c716afa0...61614c6c5845 · miraheze/puppet · GitHub | github.com [21:39:41] [02miraheze/puppet] 07paladox 0361614c6 - gluster: fix ssl path (#2675) [21:39:42] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [21:39:43] [url] Page not found · GitHub · GitHub | github.com [21:39:44] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [21:47:44] wikis aren't readable currently btw [21:48:25] on any of the mw servers (I've tried proxying through all) [21:52:00] gratisdata is [21:52:08] unstable though [21:52:16] working now [22:10:17] paladox: can you merge https://github.com/miraheze/puppet/pull/2673 and https://github.com/miraheze/puppet/pull/2669? And then I will start mwtask111 clone now (manually) so that I can deploy when everything is up. Otherwise I think we may end up going over, and I want to try and prevent that. [22:10:18] [url] mwupgradetool: Use deploy-mediawiki to sync config by Universal-Omega · Pull Request #2673 · miraheze/puppet · GitHub | github.com [22:10:19] [url] mediawiki: use REL1_38 by Universal-Omega · Pull Request #2669 · miraheze/puppet · GitHub | github.com [22:10:34] [02puppet] 07paladox closed pull request 03#2673: mwupgradetool: Use deploy-mediawiki to sync config - 13https://github.com/miraheze/puppet/pull/2673 [22:10:34] [url] Page not found · GitHub · GitHub | github.com [22:10:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/61614c6c5845...ca7c8a896caf [22:10:36] [url] Comparing 61614c6c5845...ca7c8a896caf · miraheze/puppet · GitHub | github.com [22:10:37] [02miraheze/puppet] 07Universal-Omega 03ca7c8a8 - mwupgradetool: Use deploy-mediawiki to sync config (#2673) [22:10:47] [02puppet] 07paladox closed pull request 03#2669: mediawiki: use REL1_38 - 13https://github.com/miraheze/puppet/pull/2669 [22:10:47] [url] Page not found · GitHub · GitHub | github.com [22:10:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ca7c8a896caf...e44f4375a63d [22:10:49] [url] Comparing ca7c8a896caf...e44f4375a63d · miraheze/puppet · GitHub | github.com [22:10:50] [02miraheze/puppet] 07Universal-Omega 03e44f437 - mediawiki: use REL1_38 (#2669) [22:11:00] thank you [22:12:24] !log sudo -u www-data rm -rf /srv/mediawiki-staging/w on mwtask111 [22:15:01] !log universalomega@mwtask111:/srv/mediawiki-staging$ sudo -u www-data git clone https://github.com/miraheze/mediawiki w --recurse-submodules --branch=REL1_38 [22:15:02] [url] GitHub - miraheze/mediawiki: The collaborative editing software that runs Wikipedia. | github.com [22:16:17] !log and with --depth=5 on above [22:26:30] !log disable puppet on mwtask111 [22:39:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ee90a3ea73c8 [22:39:19] [02miraheze/puppet] 07paladox 03ee90a3e - gluster: fix firewall rules [22:39:21] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [22:39:21] [url] Page not found · GitHub · GitHub | github.com [22:39:23] [02puppet] 07paladox opened pull request 03#2676: gluster: fix firewall rules - 13https://github.com/miraheze/puppet/pull/2676 [22:39:23] [url] Page not found · GitHub · GitHub | github.com [22:40:28] [02puppet] 07paladox synchronize pull request 03#2676: gluster: fix firewall rules - 13https://github.com/miraheze/puppet/pull/2676 [22:40:28] [url] Page not found · GitHub · GitHub | github.com [22:40:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ee90a3ea73c8...3b0db7745833 [22:40:30] [url] Comparing ee90a3ea73c8...3b0db7745833 · miraheze/puppet · GitHub | github.com [22:40:31] [02miraheze/puppet] 07paladox 033b0db77 - Update gluster.pp [22:41:17] [02puppet] 07paladox synchronize pull request 03#2676: gluster: fix firewall rules - 13https://github.com/miraheze/puppet/pull/2676 [22:41:17] [url] Page not found · GitHub · GitHub | github.com [22:41:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3b0db7745833...7bb31726a320 [22:41:19] [url] Comparing 3b0db7745833...7bb31726a320 · miraheze/puppet · GitHub | github.com [22:41:20] [02miraheze/puppet] 07paladox 037bb3172 - Update gluster.pp [22:41:30] i had a problem [22:42:00] [02puppet] 07paladox synchronize pull request 03#2676: gluster: fix firewall rules - 13https://github.com/miraheze/puppet/pull/2676 [22:42:00] [url] Page not found · GitHub · GitHub | github.com [22:42:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7bb31726a320...62864a545de3 [22:42:02] [url] Comparing 7bb31726a320...62864a545de3 · miraheze/puppet · GitHub | github.com [22:42:03] [02miraheze/puppet] 07paladox 0362864a5 - Update gluster.pp [22:42:40] !log universalomega@mwtask111:/srv/mediawiki-staging/w$ sudo -u www-data http_proxy=http://bast.miraheze.org:8080 composer install --no-dev [22:43:06] !log enable and run puppet on mwtask111 [22:53:21] !log [universalomega@mwtask111] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True, 'force': True, 'ignoretime': True} to mwtask111 [23:10:41] [02puppet] 07paladox synchronize pull request 03#2676: gluster: fix firewall rules - 13https://github.com/miraheze/puppet/pull/2676 [23:10:42] [url] Page not found · GitHub · GitHub | github.com [23:10:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/62864a545de3...04baaf971900 [23:10:43] [url] Comparing 62864a545de3...04baaf971900 · miraheze/puppet · GitHub | github.com [23:10:44] [02miraheze/puppet] 07paladox 0304baaf9 - Update gluster.pp [23:12:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4754-Universal-Omega-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/a678259d64da [23:12:24] [02miraheze/mw-config] 07Universal-Omega 03a678259 - Revert "Set all wikis read-only for MediaWiki 1.38 upgrade (#4754)" [23:12:26] [02mw-config] 07Universal-Omega created branch 03revert-4754-Universal-Omega-patch-5 - 13https://github.com/miraheze/mw-config [23:12:26] [url] Page not found · GitHub · GitHub | github.com [23:12:27] [02mw-config] 07Universal-Omega opened pull request 03#4760: Revert "Set all wikis read-only for MediaWiki 1.38 upgrade" - 13https://github.com/miraheze/mw-config/pull/4760 [23:12:28] [url] Page not found · GitHub · GitHub | github.com [23:13:39] miraheze/mw-config - Universal-Omega the build passed. [23:15:09] !log [universalomega@mwtask111] finished deploy of {'world': True, 'l10n': True, 'extensionlist': True, 'force': True, 'ignoretime': True} to mwtask111 - SUCCESS in 1307s [23:15:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/97abc790c953...47595e6a5c8a [23:15:32] [url] Comparing 97abc790c953...47595e6a5c8a · miraheze/mw-config · GitHub | github.com [23:15:33] [02miraheze/mw-config] 07Universal-Omega 0347595e6 - Fix [23:16:32] miraheze/mw-config - Universal-Omega the build passed. [23:17:38] !log universalomega@mwtask111:/srv/mediawiki-staging/w$ deploy-mediawiki --world --extension-list --l10n --force --ignore-time --servers=mw101 [23:18:32] !log [@test101] starting deploy of {'config': True} to all [23:18:33] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [23:21:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/47595e6a5c8a...fc7be61ef6d0 [23:21:41] [url] Comparing 47595e6a5c8a...fc7be61ef6d0 · miraheze/mw-config · GitHub | github.com [23:21:42] [02miraheze/mw-config] 07Universal-Omega 03fc7be61 - Update LocalSettings.php [23:21:59] !log [universalomega@mwtask111] finished deploy of {'world': True, 'l10n': True, 'extensionlist': True, 'force': True, 'servers': 'mw101', 'ignoretime': True} to mw101 - SUCCESS in 307s [23:22:46] miraheze/mw-config - Universal-Omega the build passed. [23:23:16] [02puppet] 07paladox synchronize pull request 03#2676: gluster: fix firewall rules - 13https://github.com/miraheze/puppet/pull/2676 [23:23:17] [url] Page not found · GitHub · GitHub | github.com [23:23:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/04baaf971900...bf1e2875abf6 [23:23:18] [url] Comparing 04baaf971900...bf1e2875abf6 · miraheze/puppet · GitHub | github.com [23:23:19] [02miraheze/puppet] 07paladox 03bf1e287 - Update gluster.pp [23:23:27] !log universalomega@mwtask111:/srv/mediawiki-staging/w$ deploy-mediawiki --world --extension-list --l10n --force --ignore-time --servers=mw102,mw111,mw112,mw121,mw122 [23:23:40] [02puppet] 07paladox closed pull request 03#2676: gluster: fix firewall rules - 13https://github.com/miraheze/puppet/pull/2676 [23:23:41] [url] Page not found · GitHub · GitHub | github.com [23:23:42] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [23:23:43] [url] Page not found · GitHub · GitHub | github.com [23:23:44] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [23:24:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e44f4375a63d...9b544fcdb8e3 [23:24:03] [url] Comparing e44f4375a63d...9b544fcdb8e3 · miraheze/puppet · GitHub | github.com [23:24:04] [02miraheze/puppet] 07paladox 039b544fc - gluster: fix firewall rules (#2676) [23:24:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/mw-config/compare/fc7be61ef6d0...02987de51c25 [23:24:26] [url] Comparing fc7be61ef6d0...02987de51c25 · miraheze/mw-config · GitHub | github.com [23:24:27] [02miraheze/mw-config] 07Universal-Omega 0302987de - Delete 138pre-patch.sql [23:25:29] miraheze/mw-config - Universal-Omega the build passed. [23:28:58] !log sudo -u www-data rsync --inplace -r --delete -e "ssh -i /srv/mediawiki-staging/deploykey" /srv/mediawiki/w/ www-data@mw102.miraheze.org:/srv/mediawiki/w on mwtask111 [23:29:45] !log sudo -u www-data rsync --inplace -r --delete -e "ssh -i /srv/mediawiki-staging/deploykey" /srv/mediawiki/w/ www-data@mw111.miraheze.org:/srv/mediawiki/w on mwtask111 [23:30:20] paladox: am doing fully manually now since the deploy tool does not work with --force [23:30:25] yehok [23:30:29] whoops sorry [23:30:32] i meant ok [23:30:45] had yeh typed in for something and i don't remember what lol [23:31:02] lol yeah I've done that before... [23:31:28] mw111 now erroring, is that expected [23:32:04] yeah [23:32:11] until it is finished deploying [23:32:23] ack [23:32:50] !log sudo -u www-data rsync --inplace -r --delete -e "ssh -i /srv/mediawiki-staging/deploykey" /srv/mediawiki/w/ www-data@mw112.miraheze.org:/srv/mediawiki/w on mwtask111 [23:34:43] !log sudo -u www-data rsync --inplace -r --delete -e "ssh -i /srv/mediawiki-staging/deploykey" /srv/mediawiki/w/ www-data@mw121.miraheze.org:/srv/mediawiki/w on mwtask111 [23:34:58] RECOVERY - mon111 IRCEcho on mon111 is OK: PROCS OK: 1 process with args '/usr/local/bin/ircecho' [23:35:12] RECOVERY - mw122 Check Gluster Clients on mw122 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [23:35:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.66, 3.20 [23:35:47] paladox: is gluster out of read-only? [23:35:54] CosmicAlpha: yeh [23:35:59] Great! [23:36:15] About 5-10 more minutes is all and I should be able to take MediaWiki out. [23:36:29] nice [23:36:30] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:36:34] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:36:35] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.24, 4.30, 3.75 [23:37:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.64, 4.03, 3.42 [23:37:45] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 341 bytes in 0.097 second response time [23:38:40] !log sudo -u www-data rsync --inplace -r --delete -e "ssh -i /srv/mediawiki-staging/deploykey" /srv/mediawiki/w/ www-data@mw122.miraheze.org:/srv/mediawiki/w on mwtask111 [23:39:05] hmm reception didn't reboot cp* [23:39:08] gonna do that now [23:39:14] !log rebooting cp* [23:39:20] !log rebooting ns1 [23:39:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.99, 3.49 [23:39:38] paladox: he asked you when and you didn’t answer [23:39:41] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 25010 bytes in 0.264 second response time [23:39:45] oh [23:40:09] thanks MacFan4000 [23:40:26] [02mw-config] 07Universal-Omega closed pull request 03#4760: Revert "Set all wikis read-only for MediaWiki 1.38 upgrade" - 13https://github.com/miraheze/mw-config/pull/4760 [23:40:27] [url] Page not found · GitHub · GitHub | github.com [23:40:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/02987de51c25...5b0286b61dc0 [23:40:29] [url] Comparing 02987de51c25...5b0286b61dc0 · miraheze/mw-config · GitHub | github.com [23:40:29] [02miraheze/mw-config] 07Universal-Omega 035b0286b - Revert "Set all wikis read-only for MediaWiki 1.38 upgrade" (#4760) [23:40:31] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4754-Universal-Omega-patch-5 [23:40:32] [02mw-config] 07Universal-Omega deleted branch 03revert-4754-Universal-Omega-patch-5 - 13https://github.com/miraheze/mw-config [23:40:33] [url] Page not found · GitHub · GitHub | github.com [23:40:51] [02mw-config] 07Universal-Omega closed pull request 03#4745: Add ImportDump extension - 13https://github.com/miraheze/mw-config/pull/4745 [23:40:51] [url] Page not found · GitHub · GitHub | github.com [23:40:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/5b0286b61dc0...4e43d5d565e8 [23:40:53] [url] Comparing 5b0286b61dc0...4e43d5d565e8 · miraheze/mw-config · GitHub | github.com [23:40:54] [02miraheze/mw-config] 07Universal-Omega 034e43d5d - Add ImportDump extension (#4745) [23:40:55] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [23:40:56] [url] Page not found · GitHub · GitHub | github.com [23:40:57] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [23:41:00] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: connect to address localhost and port 8103: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:41:04] [02mw-config] 07Universal-Omega closed pull request 03#4741: Update skin settings for MediaWiki 1.38 - 13https://github.com/miraheze/mw-config/pull/4741 [23:41:04] [url] Page not found · GitHub · GitHub | github.com [23:41:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/4e43d5d565e8...4e4392a8c149 [23:41:06] [url] Comparing 4e43d5d565e8...4e4392a8c149 · miraheze/mw-config · GitHub | github.com [23:41:07] [02miraheze/mw-config] 07Universal-Omega 034e4392a - Update skin settings for MediaWiki 1.38 (#4741) [23:41:08] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: connect to address localhost and port 8103: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:41:08] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [23:41:10] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [23:41:10] ... [23:41:19] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: connect to address localhost and port 8103: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:41:35] miraheze/mw-config - Universal-Omega the build passed. [23:41:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.89, 3.63, 3.67 [23:41:45] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 7079 MB (18% inode=96%); [23:42:03] !log [universalomega@mwtask111] starting deploy of {'l10n': True, 'extensionlist': True, 'force': True, 'ignoretime': True} to all [23:42:12] miraheze/mw-config - Universal-Omega the build passed. [23:42:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.10, 0.34, 0.12 [23:42:24] Agent: might be good to have a post-upgrade site notice ready to go [23:42:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.61, 3.25, 3.52 [23:42:38] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 6752 MB (17% inode=96%); [23:42:51] MacFan4000 still in read-only, I just merged the PRs but disabled puppet till ready to go. [23:42:56] RECOVERY - mw102 Puppet on mw102 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:05] RECOVERY - cp30 Disk Space on cp30 is OK: DISK OK - free space: / 6585 MB (17% inode=96%); [23:43:13] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:43:14] Yup still good to have it ready to go though [23:43:27] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [23:43:35] also i noticed your message earlier MacFan4000 about the notices you sent out, wanted to say thanks. [23:43:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.70, 4.04, 3.78 [23:43:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:44:05] PROBLEM - cp31 HTTPS on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 6116 bytes in 0.322 second response time [23:44:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 8.51, 4.88, 4.06 [23:44:50] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:45:07] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.018 second response time [23:45:08] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 94% [23:45:15] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.018 second response time [23:45:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.11, 4.90, 3.93 [23:46:20] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 5.33, 3.54, 1.51 [23:46:34] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw112 [23:47:06] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 33% [23:47:13] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:47:15] [discord] Yeah, doing a PR for sitenotice right now... [23:47:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:47:44] [discord] yes [23:47:46] [discord] lets [23:47:47] [discord] GO [23:47:59] !log [@test101] starting deploy of {'config': True} to all [23:48:00] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [23:48:05] RECOVERY - cp31 HTTPS on cp31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3387 bytes in 0.646 second response time [23:48:18] I think we’re fairly close [23:48:32] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [23:48:36] [discord] lets fricking go [23:49:26] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 4.065 second response time [23:49:41] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:45] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 2.215 second response time [23:50:02] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:31] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgVectorDefaultSkinVersion --sourcelist=false --remove (START) [23:50:35] RECOVERY - mw112 Puppet on mw112 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [23:50:50] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 2.142 second response time [23:51:10] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgVectorDefaultSkinVersion --sourcelist=false --remove (END - exit=2) [23:51:11] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgVectorDefaultSkinVersionForExistingAccounts --sourcelist=false --remove (START) [23:51:12] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgVectorDefaultSkinVersionForExistingAccounts --sourcelist=false --remove (END - exit=2) [23:51:13] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgVectorDefaultSkinVersionForNewAccounts --sourcelist=false --remove (START) [23:51:14] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgVectorDefaultSkinVersionForNewAccounts --sourcelist=false --remove (END - exit=2) [23:51:15] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgVectorUseWvuiSearch --sourcelist=false --remove (START) [23:51:16] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgVectorUseWvuiSearch --sourcelist=false --remove (END - exit=2) [23:51:26] Whoops, did not mean to start that... [23:51:43] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 2.979 second response time [23:51:44] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.312 second response time [23:52:18] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [23:52:26] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw112 [23:52:36] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [23:52:49] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 25036 bytes in 0.684 second response time [23:53:17] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [23:53:36] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 2.702 second response time [23:53:39] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 6.148 second response time [23:53:49] mw112 is still down... [23:53:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:55] it keeps failing! [23:54:03] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [23:54:16] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [23:54:20] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.79, 1.95, 1.62 [23:54:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:54:23] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [23:54:24] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [23:54:39] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [23:54:55] yeah --force does nothing [23:55:00] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [23:55:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:10] !log reboot mw112 [23:55:15] CosmicAlpha: ^ [23:55:23] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [23:55:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:35] thanks paladox [23:55:55] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [23:56:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:56:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.98, 1.67, 1.56 [23:56:56] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: connect to address 2a10:6740::6:207 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [23:57:12] paladox: ^ [23:57:23] yeh it's rebooting [23:57:26] it's up now [23:57:39] PROBLEM - mw112 Check Gluster Clients on mw112 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [23:57:50] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [23:58:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:58:06] still fails [23:58:47] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'servers': 'mw121,mw122'} to mw121,mw122 - SUCCESS in 16s [23:58:55] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 25036 bytes in 0.339 second response time [23:59:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:59:21] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw112