[00:00:56] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 34117 bytes in 0.617 second response time [00:03:18] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.17, 3.36, 2.51 [00:05:18] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 1.66, 2.69, 2.37 [00:15:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.56, 3.00, 3.83 [00:17:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.36, 3.44, 3.88 [00:19:10] !log [@test101] starting deploy of {'config': True} to all [00:19:11] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [00:19:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:19:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.21, 1.81, 1.52 [00:31:36] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.34, 1.67, 1.50 [00:35:36] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.16, 1.91, 1.63 [00:37:32] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.57, 1.81, 1.63 [00:41:26] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.22, 1.60, 1.59 [00:43:56] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.96, 3.07, 1.93 [00:48:43] !log [@test101] starting deploy of {'config': True} to all [00:48:44] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [00:48:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:48:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:49:19] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [00:51:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.24, 3.03, 3.96 [00:51:54] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.03, 1.77, 1.76 [00:53:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 3.48, 4.00 [00:53:53] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.95, 1.49, 1.66 [01:02:59] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:03:05] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:03:13] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:04:10] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:04:25] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:00] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.494 second response time [01:05:00] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.214 second response time [01:05:11] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.019 second response time [01:06:07] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.332 second response time [01:06:24] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23887 bytes in 0.370 second response time [01:09:43] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.44, 2.03, 1.75 [01:11:42] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.39, 1.76, 1.69 [01:13:40] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.43, 1.64, 1.64 [01:18:34] !log [@test101] starting deploy of {'config': True} to all [01:18:35] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [01:18:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:18:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:20:49] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:27:13] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [01:27:13] [url] Page not found · GitHub · GitHub | github.com [01:27:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/28a69f7a314f...a9f475bcca5b [01:27:15] [url] Comparing 28a69f7a314f...a9f475bcca5b · miraheze/mw-config · GitHub | github.com [01:27:16] [02miraheze/mw-config] 07Universal-Omega 03a9f475b - Update LocalSettings.php [01:28:24] miraheze/mw-config - Universal-Omega the build passed. [01:35:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a9f475bcca5b...9c644a95e93c [01:35:13] [url] Comparing a9f475bcca5b...9c644a95e93c · miraheze/mw-config · GitHub | github.com [01:35:14] [02miraheze/mw-config] 07Universal-Omega 039c644a9 - Update MirahezeFunctions.php [01:35:15] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [01:35:16] [url] Page not found · GitHub · GitHub | github.com [01:36:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9c644a95e93c...2bf4d95e2005 [01:36:09] [url] Comparing 9c644a95e93c...2bf4d95e2005 · miraheze/mw-config · GitHub | github.com [01:36:10] [02miraheze/mw-config] 07Universal-Omega 032bf4d95 - Update LocalSettings.php [01:36:12] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [01:36:12] [url] Page not found · GitHub · GitHub | github.com [01:36:31] miraheze/mw-config - Universal-Omega the build passed. [01:37:22] miraheze/mw-config - Universal-Omega the build passed. [01:38:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2bf4d95e2005...d1ce913c7488 [01:38:24] [url] Comparing 2bf4d95e2005...d1ce913c7488 · miraheze/mw-config · GitHub | github.com [01:38:25] [02miraheze/mw-config] 07Universal-Omega 03d1ce913 - Update LocalSettings.php [01:38:27] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [01:38:27] [url] Page not found · GitHub · GitHub | github.com [01:39:45] miraheze/mw-config - Universal-Omega the build passed. [01:42:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d1ce913c7488...d1fa05488d5b [01:42:15] [url] Comparing d1ce913c7488...d1fa05488d5b · miraheze/mw-config · GitHub | github.com [01:42:16] [02miraheze/mw-config] 07Universal-Omega 03d1fa054 - Update MirahezeFunctions.php [01:42:17] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [01:42:18] [url] Page not found · GitHub · GitHub | github.com [01:43:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d1fa05488d5b...329758e15d2f [01:43:03] [url] Comparing d1fa05488d5b...329758e15d2f · miraheze/mw-config · GitHub | github.com [01:43:04] [02miraheze/mw-config] 07Universal-Omega 03329758e - Update MirahezeFunctions.php [01:43:06] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [01:43:06] [url] Page not found · GitHub · GitHub | github.com [01:43:17] miraheze/mw-config - Universal-Omega the build has errored. [01:44:16] miraheze/mw-config - Universal-Omega the build passed. [01:48:58] !log [@test101] starting deploy of {'config': True} to all [01:48:59] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [01:49:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:49:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:50:09] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.06, 2.13, 1.59 [01:51:54] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [01:51:55] [url] Page not found · GitHub · GitHub | github.com [01:51:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/329758e15d2f...f71e77e541b5 [01:51:56] [url] Comparing 329758e15d2f...f71e77e541b5 · miraheze/mw-config · GitHub | github.com [01:51:57] [02miraheze/mw-config] 07Universal-Omega 03f71e77e - Update MirahezeFunctions.php [01:53:05] miraheze/mw-config - Universal-Omega the build passed. [01:54:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.19, 1.81, 1.59 [01:55:59] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.73, 1.45, 1.48 [01:56:14] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.22, 3.11, 2.00 [02:04:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.28, 1.78, 1.80 [02:04:42] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [02:04:42] [url] Page not found · GitHub · GitHub | github.com [02:04:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f71e77e541b5...9450ef8eda3f [02:04:44] [url] Comparing f71e77e541b5...9450ef8eda3f · miraheze/mw-config · GitHub | github.com [02:04:45] [02miraheze/mw-config] 07Universal-Omega 039450ef8 - Update MirahezeFunctions.php [02:05:45] miraheze/mw-config - Universal-Omega the build passed. [02:06:08] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.88, 1.46, 1.68 [02:14:02] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.13, 1.75, 1.75 [02:15:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.37, 3.94 [02:16:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.19, 1.61, 1.69 [02:17:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.69, 3.98 [02:19:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9450ef8eda3f...229f9de9e124 [02:19:07] [url] Comparing 9450ef8eda3f...229f9de9e124 · miraheze/mw-config · GitHub | github.com [02:19:08] [02miraheze/mw-config] 07Universal-Omega 03229f9de - Update LocalSettings.php [02:19:10] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [02:19:10] [url] Page not found · GitHub · GitHub | github.com [02:19:17] !log [@test101] starting deploy of {'config': True} to all [02:19:18] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [02:19:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.54, 3.89 [02:20:26] miraheze/mw-config - Universal-Omega the build has errored. [02:21:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/229f9de9e124...08284a51cf77 [02:21:20] [url] Comparing 229f9de9e124...08284a51cf77 · miraheze/mw-config · GitHub | github.com [02:21:21] [02miraheze/mw-config] 07Universal-Omega 0308284a5 - Update LocalSettings.php [02:21:23] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [02:21:23] [url] Page not found · GitHub · GitHub | github.com [02:22:27] miraheze/mw-config - Universal-Omega the build passed. [02:23:16] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.17, 1.71, 1.39 [02:25:12] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.13, 1.41, 1.31 [02:25:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.92, 4.16, 4.06 [02:29:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.84, 3.97 [02:34:48] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.08, 1.87, 1.61 [02:35:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.92, 3.75, 3.81 [02:36:47] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.39, 1.61, 1.54 [02:37:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.71, 3.78 [02:39:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 3.79, 3.79 [02:41:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.07, 3.55, 3.71 [02:44:41] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.62, 1.94, 1.65 [02:45:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.62, 3.66 [02:46:40] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.67, 1.86, 1.66 [02:48:39] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.02, 2.15, 1.78 [02:48:49] !log [@test101] starting deploy of {'config': True} to all [02:48:50] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [02:49:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:49:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.73, 1.81, 1.56 [02:54:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.98, 1.53, 1.49 [02:58:34] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.48, 1.89, 1.88 [03:02:31] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.67, 1.93, 1.87 [03:04:30] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.54, 1.91, 1.89 [03:06:29] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.09, 2.00, 1.93 [03:08:28] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.26, 1.75, 1.84 [03:12:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.98, 1.42, 1.69 [03:18:57] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [03:19:13] !log [@test101] starting deploy of {'config': True} to all [03:19:14] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [03:19:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:19:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:47:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.34, 3.39, 3.98 [03:48:44] !log [@test101] starting deploy of {'config': True} to all [03:48:45] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [03:48:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:48:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:49:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.21, 1.95, 1.60 [03:49:29] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.91, 3.59, 2.12 [03:49:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.65, 4.00 [03:51:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.04, 3.51, 3.90 [03:53:11] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.09, 1.61, 1.55 [03:55:25] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.59, 1.62, 1.69 [03:55:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.73, 3.90 [04:03:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.49, 3.85 [04:12:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.35, 3.55, 3.94 [04:15:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.58, 3.65 [04:16:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.84, 3.97 [04:18:55] !log [@test101] starting deploy of {'config': True} to all [04:18:56] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [04:19:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:19:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:20:58] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:21:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.60, 3.66 [04:29:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.11, 3.95, 3.69 [04:31:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.13, 3.69, 3.63 [04:37:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.72, 1.95, 1.53 [04:39:11] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.90, 1.55, 1.43 [04:41:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 3.46, 3.51 [04:42:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.47, 4.00 [04:43:35] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.86, 2.84, 3.28 [04:47:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.01, 3.45, 3.99 [04:47:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.44, 3.44 [04:48:56] !log [@test101] starting deploy of {'config': True} to all [04:48:57] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [04:49:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:49:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:49:35] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.40, 3.13, 3.33 [04:50:30] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [04:52:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.81, 3.60, 3.69 [04:53:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.67, 3.87 [04:55:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.91, 3.80, 3.90 [04:56:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.49, 3.61 [04:58:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.08, 3.91, 3.74 [05:00:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.25, 3.47, 3.59 [05:08:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.45, 3.04, 3.36 [05:18:24] !log [@test101] starting deploy of {'config': True} to all [05:18:25] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [05:18:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:18:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:21:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.99, 3.01, 3.33 [05:24:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.80, 1.48, 1.15 [05:24:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.29, 4.44, 3.55 [05:25:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.55, 5.25, 4.14 [05:25:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.41, 4.88, 3.78 [05:26:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.04, 1.35, 1.15 [05:49:11] !log [@test101] starting deploy of {'config': True} to all [05:49:11] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [05:49:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:49:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:53:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.37, 3.49, 3.97 [06:01:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 3.66, 3.78 [06:08:14] PROBLEM - mabi.myomi.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mabi.myomi.net All nameservers failed to answer the query. [06:16:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.19, 3.21, 3.85 [06:17:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.80, 3.21, 3.87 [06:18:12] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.86, 2.08, 1.32 [06:20:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 3.43, 3.77 [06:20:40] !log [@test101] starting deploy of {'config': True} to all [06:20:41] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [06:20:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:20:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:23:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.50, 3.78, 3.87 [06:27:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.69, 3.57, 3.79 [06:28:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.85, 2.00, 1.70 [06:32:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.07, 3.95, 3.76 [06:33:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.95, 1.77 [06:34:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.86, 3.72, 3.95 [06:34:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.72, 3.71 [06:35:56] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.89, 1.76 [06:36:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.06, 3.70, 3.90 [06:36:56] RECOVERY - mabi.myomi.net - reverse DNS on sslhost is OK: SSL OK - mabi.myomi.net reverse DNS resolves to cp21.miraheze.org - CNAME OK [06:37:54] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.93, 1.80 [06:38:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.39, 3.35, 3.76 [06:40:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 2.67, 3.24 [06:41:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.99, 2.00, 1.87 [06:42:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.03, 2.58, 3.34 [06:43:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.44, 3.02, 3.79 [06:43:49] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.15, 2.02, 1.89 [06:45:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.04, 3.72, 3.95 [06:49:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.60, 3.53, 3.86 [06:50:37] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:51:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.10, 4.27, 4.10 [06:51:25] !log [@test101] starting deploy of {'config': True} to all [06:51:26] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [06:51:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:51:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:52:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.24, 3.84, 3.57 [06:52:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.23, 4.19, 3.63 [06:54:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.59, 3.41, 3.45 [06:54:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.39, 3.47, 3.43 [06:56:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.97, 2.91, 3.23 [06:57:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.78, 3.98 [06:59:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.57, 4.15, 4.10 [07:02:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.65, 3.05, 3.26 [07:03:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.82, 3.32, 3.81 [07:03:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.55, 1.85, 2.00 [07:05:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.29, 3.82, 3.92 [07:06:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.95, 3.50, 3.38 [07:07:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 3.56, 3.81 [07:08:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.23, 3.04, 3.22 [07:09:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 3.97, 3.93 [07:11:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.53, 3.77 [07:13:23] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.08, 1.36, 1.70 [07:19:58] !log [@test101] starting deploy of {'config': True} to all [07:19:59] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [07:20:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:20:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:21:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.68, 1.71 [07:23:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.21, 2.93, 3.31 [07:23:12] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.47, 1.60, 1.68 [07:27:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.21, 3.36, 3.42 [07:27:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.67, 3.74, 3.20 [07:29:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.77, 3.11, 3.31 [07:32:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.42, 2.93 [07:33:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 4.58, 3.93 [07:33:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.07, 3.95, 3.50 [07:34:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.64, 3.14, 2.89 [07:35:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.97, 3.77 [07:37:04] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.05, 3.16, 3.28 [07:41:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.17, 3.09, 3.39 [07:42:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 3.89, 3.53 [07:44:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.83, 3.53, 3.45 [07:46:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.68, 3.16, 3.31 [07:48:25] !log [@test101] starting deploy of {'config': True} to all [07:48:26] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [07:48:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:48:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:49:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 3.80, 3.56 [07:51:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.65, 3.54 [07:53:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 3.64, 3.53 [07:55:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.81, 3.23, 3.39 [08:11:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.06, 3.68, 3.42 [08:13:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.59, 3.42 [08:19:00] !log [@test101] starting deploy of {'config': True} to all [08:19:01] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [08:19:13] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [08:19:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.53, 3.31, 3.38 [08:19:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:20:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:37:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 3.89, 3.51 [08:39:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.99, 3.09, 3.26 [08:48:54] !log [@test101] starting deploy of {'config': True} to all [08:48:55] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [08:48:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:49:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:50:43] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [09:04:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.18, 3.46, 3.22 [09:06:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.04, 3.34, 3.20 [09:18:56] !log [@test101] starting deploy of {'config': True} to all [09:18:57] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [09:19:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:19:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:19:25] is test101 stuck [09:19:46] CosmicAlpha: you broke test [09:19:56] i'll stop puppet [09:23:10] PROBLEM - test101 Puppet on test101 is WARNING: WARNING: Puppet is currently disabled, message: UO has pulled another config branch - stopping to avoid puppet having a fit and deploying every half an hour (RhinosF1), last run 4 minutes ago with 0 failures [09:23:12] !log stopped puppet on test101 to avoid spam broken deploys due to test being ahead of origin [09:23:16] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.14, 1.88, 1.65 [09:23:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:26:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.52, 3.47, 3.28 [09:27:15] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.20, 1.83, 1.70 [09:28:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.91, 3.13, 3.17 [09:31:12] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.22, 1.56, 1.63 [09:33:30] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.61, 3.08, 1.93 [09:33:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.55, 3.35 [09:35:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.64, 3.28, 3.28 [09:37:08] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.32, 2.14, 1.92 [09:37:30] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.80, 1.93, 1.73 [09:39:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.64, 1.50, 1.59 [09:45:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.31, 1.91, 1.97 [09:49:47] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [09:53:00] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.24, 1.38, 1.68 [09:58:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.67, 3.30 [10:01:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.06, 3.12, 3.17 [10:05:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.45, 3.31 [10:07:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.22, 3.22, 3.23 [10:20:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.59, 3.56, 3.46 [10:21:41] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.17, 2.05, 1.71 [10:23:08] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.38, 1.82, 1.47 [10:24:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.78, 3.22, 3.37 [10:25:01] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.04, 1.86, 1.52 [10:26:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.38, 1.75, 1.52 [10:28:49] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.97, 1.49, 1.46 [10:40:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.68, 3.29 [10:42:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.10, 3.19, 3.16 [10:43:41] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.12, 1.63, 1.95 [10:45:41] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.21, 2.04, 2.07 [10:49:41] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.31, 1.65, 1.90 [10:55:41] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.38, 1.89, 1.90 [10:57:41] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.07, 1.61, 1.79 [11:01:41] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.03, 1.53, 1.70 [11:03:41] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.50, 1.54, 1.69 [11:07:41] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.39, 1.97, 1.85 [11:11:42] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.07, 1.53, 1.69 [11:25:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.80, 3.42, 3.21 [11:27:12] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.50, 4.06, 3.21 [11:27:41] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.53, 1.76, 1.59 [11:29:05] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.74, 3.10, 3.14 [11:29:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.37, 3.70, 3.18 [11:29:41] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.74, 1.75, 1.61 [11:31:41] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.41, 1.68, 1.60 [11:33:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.01, 3.45, 3.31 [11:35:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.92, 3.41 [11:36:59] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.84, 3.12, 3.17 [11:37:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.24, 3.04, 3.20 [11:41:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.79, 3.51 [11:43:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.13, 3.34, 3.36 [11:45:30] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.18, 1.78, 1.44 [11:47:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.60, 3.48 [11:49:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.58, 3.14, 3.32 [11:49:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.88, 1.42, 1.38 [11:52:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.18, 3.01 [11:53:33] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.71, 2.18, 1.82 [11:54:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.28, 2.96, 2.96 [11:55:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.71, 3.53, 3.48 [11:57:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.13, 3.10, 3.32 [12:03:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.57, 3.42 [12:05:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 3.20, 3.31 [12:05:28] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.44, 1.80, 1.90 [12:19:20] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.83, 1.38, 1.63 [12:25:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.62, 3.25 [12:27:16] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.09, 2.10, 1.81 [12:29:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 3.84, 3.43 [12:29:15] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.02, 1.67, 1.69 [12:30:12] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.85, 3.70, 3.32 [12:33:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.09, 3.99, 3.53 [12:36:04] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.62, 3.27, 3.27 [12:37:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.05, 3.73, 3.55 [12:39:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.83, 3.60 [12:40:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.96, 3.48, 3.02 [12:43:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.32, 3.81, 3.69 [12:44:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.03, 2.85, 2.87 [12:47:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.79, 3.66 [12:51:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.96, 3.54, 3.59 [12:54:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.12, 3.85, 3.36 [12:54:59] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.64, 1.74, 1.45 [12:57:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.85, 3.67 [12:57:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.58, 3.19 [12:58:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 4.00, 3.58 [12:58:57] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.46, 1.69, 1.51 [12:59:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.17, 3.15, 3.43 [12:59:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.18, 3.01, 3.03 [13:01:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.16, 3.00, 3.33 [13:05:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.73, 3.57 [13:08:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.99, 3.93, 3.63 [13:09:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 3.88, 3.63 [13:10:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.76, 3.41, 3.47 [13:12:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.75, 3.59 [13:14:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.66, 3.61, 3.55 [13:16:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.21, 3.88, 3.66 [13:18:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.66, 3.60 [13:20:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.03, 1.52, 1.32 [13:22:43] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.54, 1.54, 1.36 [13:25:22] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [13:26:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.55, 2.99, 3.32 [13:26:41] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.64, 1.92, 1.54 [13:27:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.54, 3.83 [13:28:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.22, 1.65, 1.49 [13:33:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.31, 3.58 [13:33:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.02, 3.86, 3.58 [13:35:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.88, 3.94, 3.65 [13:37:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.12, 4.47, 3.89 [13:39:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.64, 3.18 [13:45:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.67, 3.38, 3.24 [13:47:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.91, 3.47, 3.87 [13:54:54] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:55:04] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 3.94, 3.89 [13:56:26] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.25, 1.93, 1.64 [13:56:48] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.55, 3.36, 3.01 [13:58:25] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.28, 1.78, 1.63 [13:58:42] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.56, 3.06, 2.94 [13:58:59] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.47, 3.79, 3.84 [14:02:23] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.38, 1.61, 1.60 [14:02:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.98, 3.88, 3.81 [14:03:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.61, 4.37, 3.67 [14:08:18] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.24, 1.75, 1.62 [14:10:17] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.44, 1.58, 1.57 [14:11:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.90, 3.83 [14:14:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.04, 3.87, 3.98 [14:18:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.96, 3.99, 3.97 [14:19:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 3.79, 3.74 [14:21:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.63, 3.35, 3.59 [14:22:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.53, 3.81, 3.73 [14:24:26] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [14:37:00] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.87, 1.68, 1.53 [14:38:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.64, 3.60, 3.88 [14:38:59] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.48, 1.64, 1.54 [14:40:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.94, 3.93, 3.95 [14:44:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.57, 3.81 [14:45:54] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.77, 2.68, 2.14 [14:49:52] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.40, 1.95, 1.95 [14:53:50] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.11, 1.36, 1.70 [14:54:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.74, 4.14, 3.95 [14:57:47] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 2.00, 1.95, 1.87 [14:59:48] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.59, 2.20, 1.97 [15:00:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.78, 3.69, 3.86 [15:01:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.28, 1.84, 1.86 [15:04:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.60, 4.86, 4.22 [15:11:41] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.83, 1.27, 1.59 [15:34:04] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.84, 1.95, 1.46 [15:36:32] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.93, 1.69, 1.54 [15:37:52] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.61, 2.09, 1.60 [15:39:45] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.98, 1.71, 1.52 [15:40:30] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.27, 2.11, 1.76 [15:41:39] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.80, 1.42, 1.44 [15:42:29] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.41, 1.89, 1.72 [15:44:28] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.96, 1.66, 1.67 [15:48:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.22, 3.97 [15:50:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.29, 3.80, 4.10 [15:52:23] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.28, 1.73, 1.71 [15:54:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.43, 3.90 [15:54:22] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.11, 1.44, 1.60 [16:00:28] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 9.10, 6.70, 4.94 [16:02:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.79, 3.86 [16:03:33] RhinosF1: I know. Sorry. I was testing something there yesterday. [16:04:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.96, 3.93 [16:10:07] [02mw-config] 07Universal-Omega closed pull request 03#4752: Enable MirahezeSurvey on betawiki only - 13https://github.com/miraheze/mw-config/pull/4752 [16:10:08] [url] Page not found · GitHub · GitHub | github.com [16:10:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/96d51cbec42e...39a73af6e8ae [16:10:09] [url] Comparing 96d51cbec42e...39a73af6e8ae · miraheze/mw-config · GitHub | github.com [16:10:10] [02miraheze/mw-config] 07AgentIsai 0339a73af - Enable MirahezeSurvey on betawiki only (#4752) [16:10:28] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 5.87, 7.76, 6.65 [16:10:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.46, 4.00 [16:11:11] miraheze/mw-config - Universal-Omega the build passed. [16:12:09] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.66, 2.92, 3.37 [16:14:28] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 5.80, 6.75, 6.49 [16:14:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 3.82, 4.01 [16:16:11] PROBLEM - clockupwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for clockupwiki.org could not be found [16:18:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.59, 3.53 [16:19:41] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.43, 1.91, 1.37 [16:21:40] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.20, 1.60, 1.32 [16:22:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.99, 4.11, 3.74 [16:23:07] !log [@mwtask111] starting deploy of {'config': True} to all [16:23:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:19] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 11s [16:23:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:26:01] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.61, 3.55, 2.30 [16:26:22] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/addthis.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php addthis --disable (START) [16:26:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:28:18] [02puppet] 07Universal-Omega opened pull request 03#2666: T7876: remove `s7.addthis.com` from CSP - 13https://github.com/miraheze/puppet/pull/2666 [16:28:18] [url] Page not found · GitHub · GitHub | github.com [16:28:29] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:29:14] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/addthis.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php addthis --disable (END - exit=0) [16:29:15] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/gettingstarted.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php gettingstarted --disable (START) [16:29:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:29:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:30:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.14, 3.62, 3.96 [16:31:15] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/gettingstarted.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php gettingstarted --disable (END - exit=0) [16:31:16] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/headerfooter.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php headerfooter --disable (START) [16:31:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:31:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:33:20] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/headerfooter.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php headerfooter --disable (END - exit=0) [16:33:21] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/magicnumberedheadings.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php magicnumberedheadings --disable (START) [16:33:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:33:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:33:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.86, 3.80, 3.85 [16:34:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.25, 3.58, 3.83 [16:35:03] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/magicnumberedheadings.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php magicnumberedheadings --disable (END - exit=0) [16:35:04] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/fancyboxthumbs.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php fancyboxthumbs --disable (START) [16:35:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:35:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:43] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/fancyboxthumbs.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php fancyboxthumbs --disable (END - exit=0) [16:36:44] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/foreground.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php foreground --disable (START) [16:36:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:55] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.83, 1.77, 1.95 [16:38:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.69, 3.60, 3.79 [16:39:24] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/foreground.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php foreground --disable (END - exit=0) [16:39:25] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/pivot.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php pivot --disable (START) [16:39:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:39:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:40:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.38, 4.22, 3.97 [16:41:53] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.05, 1.66, 1.84 [16:42:21] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/pivot.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php pivot --disable (END - exit=0) [16:42:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:42:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.06, 3.83, 3.86 [16:43:52] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.29, 1.53, 1.77 [16:45:51] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.93, 1.36, 1.69 [16:45:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 3.56, 3.68 [16:46:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.68, 4.05, 3.92 [16:46:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/23793b6e360e...08303c32e4ba [16:46:53] [url] Comparing 23793b6e360e...08303c32e4ba · miraheze/mw-config · GitHub | github.com [16:46:54] [02miraheze/mw-config] 07Universal-Omega 0308303c3 - Update ExtensionsTest.php [16:46:56] [02mw-config] 07Universal-Omega synchronize pull request 03#4749: Remove extensions - 13https://github.com/miraheze/mw-config/pull/4749 [16:46:56] [url] Page not found · GitHub · GitHub | github.com [16:47:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.25, 3.17, 3.53 [16:48:02] miraheze/mw-config - Universal-Omega the build passed. [16:48:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.20, 3.34, 3.68 [16:48:53] [02mw-config] 07Universal-Omega closed pull request 03#4749: Remove extensions - 13https://github.com/miraheze/mw-config/pull/4749 [16:48:53] ... [16:48:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/mw-config/compare/39a73af6e8ae...9616ab5c6eba [16:48:55] [url] Comparing 39a73af6e8ae...9616ab5c6eba · miraheze/mw-config · GitHub | github.com [16:48:56] [02miraheze/mw-config] 07Universal-Omega 039616ab5 - Remove extensions (#4749) [16:48:57] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [16:48:59] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [16:48:59] [url] Page not found · GitHub · GitHub | github.com [16:49:00] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [16:49:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:49:16] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 16s [16:49:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:49:55] miraheze/mw-config - Universal-Omega the build passed. [16:50:10] !log [universalomega@mwtask111] starting deploy of {'l10n': True} to all [16:50:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:51:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.62, 2.54, 3.20 [16:53:06] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [16:53:07] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 0s [16:53:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:53:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:53:39] !log [universalomega@test101] starting deploy of {'l10n': True} to all [16:53:41] !log [universalomega@test101] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [16:53:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.65, 10.35, 8.99 [16:53:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:53:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:47] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.61, 2.79, 1.75 [16:56:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.30, 4.07, 3.80 [16:56:55] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 785 bytes in 0.405 second response time [16:57:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.90, 12.18, 10.02 [16:58:01] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [16:58:40] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.76, 1.76, 1.56 [16:58:41] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.89, 1.92, 1.77 [16:59:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.72, 11.31, 9.97 [17:00:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.30, 1.27, 1.41 [17:00:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.45, 4.48, 3.77 [17:01:24] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.94, 10.15, 9.70 [17:02:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.96, 1.45, 1.62 [17:02:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.83, 3.62 [17:02:56] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 32103 bytes in 0.701 second response time [17:03:14] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.24, 1.87, 1.56 [17:05:12] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.86, 1.48, 1.45 [17:06:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.40, 3.14, 3.39 [17:09:33] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.61, 1.73, 1.69 [17:10:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.82, 3.57, 3.50 [17:10:56] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 785 bytes in 0.928 second response time [17:11:09] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:11:32] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.22, 1.49, 1.60 [17:12:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.27, 10.64, 9.90 [17:14:36] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.41, 3.98, 3.64 [17:14:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.54, 10.00, 9.77 [17:14:56] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 32320 bytes in 0.895 second response time [17:17:26] !log [universalomega@mwtask111] finished deploy of {'l10n': True} to all - SUCCESS in 1636s [17:17:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:18:28] !log [universalomega@test101] starting deploy of {'l10n': True} to all [17:18:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:19:41] [02miraheze/mw-config] 07Universal-Omega pushed 037 commits to 03Universal-Omega-patch-4 [+0/-0/±16] 13https://github.com/miraheze/mw-config/compare/08284a51cf77...050a5a7e74f9 [17:19:41] [url] Comparing 08284a51cf77...050a5a7e74f9 · miraheze/mw-config · GitHub | github.com [17:19:42] [02miraheze/mw-config] 07Universal-Omega 03050a5a7 - Merge branch 'master' into Universal-Omega-patch-4 [17:19:44] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [17:19:44] [url] Page not found · GitHub · GitHub | github.com [17:20:13] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/050a5a7e74f9...e0daa70fe85f [17:20:14] [url] Comparing 050a5a7e74f9...e0daa70fe85f · miraheze/mw-config · GitHub | github.com [17:20:15] [02miraheze/mw-config] 07github-actions 03e0daa70 - CI: lint code to MediaWiki standards [17:20:16] [02mw-config] 07github-actions[bot] synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [17:20:17] [url] Page not found · GitHub · GitHub | github.com [17:20:49] miraheze/mw-config - Universal-Omega the build passed. [17:22:22] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.15, 2.39, 1.54 [17:23:25] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.54, 2.08, 1.70 [17:24:49] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.011 second response time [17:24:57] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.013 second response time [17:25:11] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.252 second response time [17:25:39] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.011 second response time [17:25:45] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.236 second response time [17:26:23] PROBLEM - mail121 webmail.miraheze.org HTTPS on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:26:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 4.00, 3.97 [17:26:38] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 11.16, 6.43, 4.09 [17:28:30] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.75, 4.05, 4.00 [17:29:22] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.91, 1.81, 1.77 [17:30:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.70, 3.88 [17:33:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e0daa70fe85f...d04e99fe9c93 [17:33:12] [url] Comparing e0daa70fe85f...d04e99fe9c93 · miraheze/mw-config · GitHub | github.com [17:33:13] [02miraheze/mw-config] 07Universal-Omega 03d04e99f - Update LocalSettings.php [17:33:15] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [17:33:15] [url] Page not found · GitHub · GitHub | github.com [17:33:19] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.60, 1.96, 1.83 [17:34:25] miraheze/mw-config - Universal-Omega the build passed. [17:34:37] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.42, 4.23, 4.50 [17:35:18] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.25, 1.76, 1.77 [17:36:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.81, 3.93, 3.83 [17:38:22] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.45, 1.86, 1.84 [17:38:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.74, 3.77 [17:38:38] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 11.09, 6.85, 5.37 [17:39:16] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.10, 1.79, 1.77 [17:39:38] !log [universalomega@test101] finished deploy of {'l10n': True} to all - SUCCESS in 1269s [17:39:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:42:22] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 0.19, 1.11, 1.55 [17:43:15] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.79, 1.71, 1.73 [17:44:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.63, 4.07, 3.86 [17:45:13] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.40, 1.58, 1.68 [17:50:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.81, 9.92, 9.21 [17:51:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.40, 9.72, 8.69 [17:52:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.19, 10.57, 9.53 [17:54:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.26, 3.49, 3.84 [17:56:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.48, 2.06, 1.79 [17:58:05] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.63, 1.84, 1.74 [17:58:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.98, 4.51, 4.10 [17:58:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.19, 10.72, 10.08 [17:59:04] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:59:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.85, 10.07, 9.37 [18:00:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.43, 1.64, 1.68 [18:02:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.67, 8.94, 9.56 [18:03:30] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.90, 2.17, 1.43 [18:04:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.64, 3.89 [18:05:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.83, 1.66, 1.33 [18:08:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.23, 3.74, 3.83 [18:09:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d04e99fe9c93...568c520f621f [18:09:45] [url] Comparing d04e99fe9c93...568c520f621f · miraheze/mw-config · GitHub | github.com [18:09:46] [02miraheze/mw-config] 07Universal-Omega 03568c520 - Update MirahezeFunctions.php [18:09:48] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [18:09:48] [url] Page not found · GitHub · GitHub | github.com [18:10:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.57, 3.36, 3.69 [18:11:00] miraheze/mw-config - Universal-Omega the build passed. [18:12:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.87, 3.39, 3.97 [18:14:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.13, 4.05, 4.14 [18:14:50] PROBLEM - wiki.chippyapps.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.chippyapps.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:15:49] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [18:15:49] [url] Page not found · GitHub · GitHub | github.com [18:15:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/568c520f621f...45dedc941485 [18:15:51] [url] Comparing 568c520f621f...45dedc941485 · miraheze/mw-config · GitHub | github.com [18:15:52] [02miraheze/mw-config] 07Universal-Omega 0345dedc9 - Update LocalSettings.php [18:16:56] miraheze/mw-config - Universal-Omega the build has errored. [18:17:53] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.81, 1.95, 1.72 [18:17:58] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.30, 4.37, 2.60 [18:18:07] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.74, 4.03, 3.84 [18:18:14] PROBLEM - matomo101 APT on matomo101 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [18:18:20] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [18:18:21] [url] Page not found · GitHub · GitHub | github.com [18:18:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/45dedc941485...ee072496f442 [18:18:22] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.76, 1.27 [18:18:22] [url] Comparing 45dedc941485...ee072496f442 · miraheze/mw-config · GitHub | github.com [18:18:23] [02miraheze/mw-config] 07Universal-Omega 03ee07249 - Update MirahezeFunctions.php [18:18:55] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 314 bytes in 0.045 second response time [18:19:10] PROBLEM - test101 Puppet on test101 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 29 minutes ago with 0 failures [18:19:29] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [18:19:29] [url] Page not found · GitHub · GitHub | github.com [18:19:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ee072496f442...e941c35e1915 [18:19:31] miraheze/mw-config - Universal-Omega the build has errored. [18:19:31] [url] Comparing ee072496f442...e941c35e1915 · miraheze/mw-config · GitHub | github.com [18:19:32] [02miraheze/mw-config] 07Universal-Omega 03e941c35 - Update LocalSettings.php [18:19:52] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.39, 1.70, 1.65 [18:20:02] !log [@test101] starting deploy of {'config': True} to all [18:20:03] !log [@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [18:20:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.53, 3.67 [18:20:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:20:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:20:14] PROBLEM - matomo101 APT on matomo101 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [18:20:27] miraheze/mw-config - Universal-Omega the build has errored. [18:20:37] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [18:20:37] [url] Page not found · GitHub · GitHub | github.com [18:20:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e941c35e1915...88f680163cba [18:20:39] [url] Comparing e941c35e1915...88f680163cba · miraheze/mw-config · GitHub | github.com [18:20:40] [02miraheze/mw-config] 07Universal-Omega 0388f6801 - Update LocalSettings.php [18:21:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.10, 2.09, 1.57 [18:21:39] miraheze/mw-config - Universal-Omega the build passed. [18:22:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.64, 1.80, 1.40 [18:23:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.23, 1.77, 1.51 [18:24:28] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.23, 1.78, 1.29 [18:24:34] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [18:24:34] [url] Page not found · GitHub · GitHub | github.com [18:24:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/88f680163cba...b27b79c02a88 [18:24:36] [url] Comparing 88f680163cba...b27b79c02a88 · miraheze/mw-config · GitHub | github.com [18:24:37] [02miraheze/mw-config] 07Universal-Omega 03b27b79c - Update MirahezeFunctions.php [18:24:59] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.24, 1.56, 1.46 [18:25:33] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.45, 1.74, 1.96 [18:25:48] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.72, 2.05, 1.79 [18:25:54] miraheze/mw-config - Universal-Omega the build passed. [18:26:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 4.27, 3.93 [18:26:27] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.13, 1.53, 1.26 [18:26:56] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 32887 bytes in 1.530 second response time [18:28:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.85, 3.68, 3.75 [18:28:36] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [18:29:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.30, 1.71, 1.72 [18:30:22] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.22, 1.94, 1.62 [18:31:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.20, 1.20, 1.64 [18:32:22] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.89, 1.64 [18:32:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+1/-0/±0] 13https://github.com/miraheze/mw-config/compare/b27b79c02a88...5c58fdf6b5ae [18:32:44] [url] Comparing b27b79c02a88...5c58fdf6b5ae · miraheze/mw-config · GitHub | github.com [18:32:45] [02miraheze/mw-config] 07Universal-Omega 035c58fdf - Create StaticSiteConfiguration.php [18:32:47] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [18:32:47] [url] Page not found · GitHub · GitHub | github.com [18:33:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5c58fdf6b5ae...bb9fb29f3bc3 [18:33:17] [url] Comparing 5c58fdf6b5ae...bb9fb29f3bc3 · miraheze/mw-config · GitHub | github.com [18:33:18] [02miraheze/mw-config] 07Universal-Omega 03bb9fb29 - Update MirahezeFunctions.php [18:33:20] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [18:33:20] [url] Page not found · GitHub · GitHub | github.com [18:33:43] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.88, 1.52, 1.66 [18:33:59] miraheze/mw-config - Universal-Omega the build passed. [18:33:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.71, 2.88, 3.34 [18:34:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bb9fb29f3bc3...ef6adbe86528 [18:34:17] [url] Comparing bb9fb29f3bc3...ef6adbe86528 · miraheze/mw-config · GitHub | github.com [18:34:18] [02miraheze/mw-config] 07Universal-Omega 03ef6adbe - Update LocalSettings.php [18:34:19] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [18:34:20] [url] Page not found · GitHub · GitHub | github.com [18:34:25] miraheze/mw-config - Universal-Omega the build passed. [18:34:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.37, 3.30, 4.00 [18:35:28] miraheze/mw-config - Universal-Omega the build passed. [18:36:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ef6adbe86528...a58f6a933ec5 [18:36:33] [url] Comparing ef6adbe86528...a58f6a933ec5 · miraheze/mw-config · GitHub | github.com [18:36:34] [02miraheze/mw-config] 07Universal-Omega 03a58f6a9 - Update MirahezeFunctions.php [18:36:36] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [18:36:36] [url] Page not found · GitHub · GitHub | github.com [18:37:38] miraheze/mw-config - Universal-Omega the build passed. [18:38:22] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 0.45, 1.32, 1.49 [18:38:55] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 314 bytes in 0.040 second response time [18:39:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a58f6a933ec5...705a734217bf [18:39:57] [url] Comparing a58f6a933ec5...705a734217bf · miraheze/mw-config · GitHub | github.com [18:39:58] [02miraheze/mw-config] 07Universal-Omega 03705a734 - Update MirahezeFunctions.php [18:40:00] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [18:40:00] [url] Page not found · GitHub · GitHub | github.com [18:41:09] miraheze/mw-config - Universal-Omega the build passed. [18:42:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.73, 3.68, 3.86 [18:48:11] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4167 bytes in 0.053 second response time [18:48:27] here [18:48:35] Yep [18:48:44] Reception123, paladox: ping [18:49:08] Icinga says too many connections [18:49:15] That's bad [18:49:49] CosmicAlpha: you around? [18:50:11] I think everything dead [18:50:15] RhinosF1: Yes. [18:50:23] CosmicAlpha: I think we've lost the misc DB [18:50:33] Not sure what's backing it up [18:50:37] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [18:50:47] PROBLEM - db112 MariaDB on db112 is CRITICAL: Too many connections [18:50:49] icinga-miraheze: we know [18:51:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.57, 9.72, 9.34 [18:51:18] Reception123, paladox: can one of you run show proccesslist and work out what's filling it up [18:51:46] RhinosF1: Hmm... Nothing I can do about that though. But weird. [18:52:12] CosmicAlpha: as long as I'm not missing anything [18:53:50] CosmicAlpha: can we ping Owen + void your try and get someone [18:54:38] to not your [18:56:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.50, 3.31 [18:56:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.02, 9.59, 9.44 [18:58:08] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:58:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.01, 3.40, 3.30 [18:59:49] .op [18:59:49] Attempting to OP... [19:01:05] RECOVERY - mail121 webmail.miraheze.org HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 200 OK - 6654 bytes in 0.064 second response time [19:01:20] Voidwalker: hi db121 is saying too many connections [19:01:28] Nope [19:01:33] 112 [19:01:37] I hate repeat numbers [19:01:48] ok, will login and check [19:02:32] * RhinosF1 is far too good at mixing repeat numbers up [19:03:30] mariadb appears to be shutting down, but not immediately clear why [19:04:09] Voidwalker: I assume it's crashed [19:04:10] [discord] Phab is down [19:04:15] we know [19:04:16] [discord] A Troublesome Encounter! [19:04:17] [discord] Woe! This request had its journey cut short by unexpected circumstances (Can Not Connect to MySQL). [19:04:29] We're clearly talking about DB issues [19:04:36] And I said misc services are down [19:04:37] [discord] Oh [19:04:39] [discord] my bad [19:04:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.94, 3.87, 3.49 [19:04:50] Voidwalker: can we get any logs [19:04:58] To show what's being running [19:05:03] Or what shut it down [19:05:42] mariadb service is currently closing, again, not clear why [19:06:33] Voidwalker: what do the MariaDB logs say? [19:06:37] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 0.11, 2.07, 5.45 [19:06:38] Wherever they are now [19:06:48] oh, disk space is at 100% [19:06:51] Does graylog work without a DB if it's there? [19:06:56] Oh that's crap [19:07:04] How is disk space 100% [19:07:14] That VM is not even that old [19:07:32] I think Owen will have to ring John [19:07:37] To get approval [19:08:37] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.23, 1.47, 4.81 [19:09:33] I'm going to see what I can poke [19:10:20] Voidwalker: maybe clear binlogs? [19:11:07] pywiki is 77G [19:11:37] Voidwalker: no wiki should be on db112 [19:11:44] it's a misc db [19:12:01] *pywik [19:12:08] piwik* [19:12:13] That's Matomo [19:13:49] yeah [19:14:38] my bad [19:14:45] I'm not quite sure what's best here [19:15:31] Binlogs are probably best bet [19:15:41] But they can't be that big [19:16:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.88, 3.83 [19:18:56] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 31825 bytes in 0.700 second response time [19:20:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.03, 4.13, 3.91 [19:22:22] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [19:22:22] [url] Page not found · GitHub · GitHub | github.com [19:22:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/705a734217bf...de0721b8b878 [19:22:24] [url] Comparing 705a734217bf...de0721b8b878 · miraheze/mw-config · GitHub | github.com [19:22:25] [02miraheze/mw-config] 07Universal-Omega 03de0721b - Update LocalSettings.php [19:23:11] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/de0721b8b878...94d8a0537fbc [19:23:11] [url] Comparing de0721b8b878...94d8a0537fbc · miraheze/mw-config · GitHub | github.com [19:23:12] [02miraheze/mw-config] 07github-actions 0394d8a05 - CI: lint code to MediaWiki standards [19:23:14] [02mw-config] 07github-actions[bot] synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [19:23:14] [url] Page not found · GitHub · GitHub | github.com [19:23:26] hmm, if I can get mysql back up, I can possibly try deleting old matomo logs to see how much space that would free [19:23:51] miraheze/mw-config - Universal-Omega the build passed. [19:27:11] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.85, 1.71, 1.59 [19:27:40] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [19:29:09] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.70, 1.60, 1.56 [19:32:35] !log manually removed mysql binlogs on db112 [19:32:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:35:47] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99038 bytes in 7.840 second response time [19:35:48] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 99046 bytes in 3.824 second response time [19:36:02] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 99046 bytes in 0.786 second response time [19:36:11] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 18991 bytes in 0.256 second response time [19:36:37] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.164 second response time [19:36:39] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.244 second response time [19:36:45] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 99077 bytes in 0.222 second response time [19:36:47] RECOVERY - db112 MariaDB on db112 is OK: Uptime: 75 Threads: 11 Questions: 67327 Slow queries: 785 Opens: 279 Open tables: 273 Queries per second avg: 897.693 [19:37:29] !log restarted mariadb on db112 [19:37:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:44:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.66, 3.67, 3.96 [19:46:01] Voidwalker: look at since yesterday morning [19:46:03] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=db112.miraheze.org&var-port=9100&viewPanel=152&from=now-7d&to=now-1m [19:46:23] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.77, 3.96 [19:46:28] More space was requested for db112 but cloud11 doesn't have enough disk available for the amount of space requested [19:46:38] [02puppet] 07Reception123 closed pull request 03#2666: T7876: remove `s7.addthis.com` from CSP - 13https://github.com/miraheze/puppet/pull/2666 [19:46:38] [url] Page not found · GitHub · GitHub | github.com [19:46:39] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3552e802e78c...07f52a84af74 [19:46:40] [url] Comparing 3552e802e78c...07f52a84af74 · miraheze/puppet · GitHub | github.com [19:46:41] [02miraheze/puppet] 07Universal-Omega 0307f52a8 - T7876: remove `s7.addthis.com` from CSP (#2666) [19:48:02] Voidwalker: https://grafana.miraheze.org/d/MQWgroiiz/mariadb-overview?orgId=1&refresh=10s&var-datasource=Prometheus&var-job=All&var-instance=db112.miraheze.org:9104&from=now-3h&to=now&viewPanel=92 is not good [19:48:16] We could have easily predicted issues [19:48:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.59, 3.87 [19:50:20] I guess that's a monitoring gap [19:50:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.56, 3.78, 3.89 [19:50:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.50, 1.92, 1.38 [19:52:12] [Grafana] !sre FIRING: There has been a rise in the MediaWiki exception rate https://grafana.miraheze.org/d/GtxbP1Xnk [19:52:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.92, 3.62, 3.84 [19:52:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.16, 1.59, 1.32 [19:53:43] PROBLEM - prometheus101 Current Load on prometheus101 is CRITICAL: CRITICAL - load average: 5.60, 3.21, 1.51 [19:54:08] Voidwalker: db101 timed out [19:55:20] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [19:55:20] [url] Page not found · GitHub · GitHub | github.com [19:55:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/94d8a0537fbc...f0bfc7a0baec [19:55:22] [url] Comparing 94d8a0537fbc...f0bfc7a0baec · miraheze/mw-config · GitHub | github.com [19:55:23] [02miraheze/mw-config] 07Universal-Omega 03f0bfc7a - Update MirahezeFunctions.php [19:56:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.44, 4.04, 3.94 [19:56:20] miraheze/mw-config - Universal-Omega the build passed. [19:56:23] RhinosF1, not seeing anything on 101 [19:56:40] Voidwalker: https://phabricator.miraheze.org/T9375 too [19:56:41] [url] ⚓ T9375 db112 outage follow ups | phabricator.miraheze.org [19:56:43] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.67, 1.75, 1.42 [19:56:53] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.20, 2.03, 1.59 [19:56:56] Looks back now Voidwalker [19:57:06] temporary blip my guess [19:57:11] Yep [19:57:12] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [19:58:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.57, 3.61, 3.80 [19:58:42] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.27, 1.55, 1.38 [19:58:51] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.30, 1.76, 1.54 [19:59:44] !log manually running matomo archive script (trying to squeeze out what disk space I can) [19:59:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:55] Voidwalker: anything else I'm missing on that task [20:00:20] also matomo seems to believe that the archive script was last ran over three days ago, despite the archive log suggesting otherwise [20:00:35] Right [20:00:49] What does the log suggest Voidwalker [20:00:50] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.04, 1.48, 1.47 [20:02:12] [Grafana] !sre RESOLVED: MediaWiki Exception Rate https://grafana.miraheze.org/d/GtxbP1Xnk [20:02:38] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.88, 2.00, 1.62 [20:03:48] It wouldn't be the first time something happens with that archive script [20:03:53] I remember having to manually run it a few times too [20:04:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.88, 3.87, 3.76 [20:04:31] not sure (running the script overwrote the log for the archive script), but it had an entry for today that appears to have completed its task at 19:00 [20:04:37] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.01, 1.59, 1.51 [20:04:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.04, 9.43, 8.33 [20:05:03] Voidwalker: I doubt it would have worked at 7 [20:05:24] The DB stopped working properly at 18:20 ish [20:05:31] true, but the log indicates it was on the last task since 15:00 [20:06:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f0bfc7a0baec...cad3ba37992b [20:06:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.54, 3.64 [20:06:14] [url] Comparing f0bfc7a0baec...cad3ba37992b · miraheze/mw-config · GitHub | github.com [20:06:15] [02miraheze/mw-config] 07Universal-Omega 03cad3ba3 - Update LocalSettings.php [20:06:16] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [20:06:16] [url] Page not found · GitHub · GitHub | github.com [20:06:38] I'll monitor the script for now and see what happens [20:06:43] That's when disk went full and connections started piling up [20:06:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.88, 9.42, 8.48 [20:07:32] miraheze/mw-config - Universal-Omega the build passed. [20:08:22] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 4.76, 2.87, 1.69 [20:10:08] hmm, disk usage is creeping up again, I may just drop the records from the DB if it looks like the script is too slow [20:11:41] PROBLEM - prometheus101 Current Load on prometheus101 is WARNING: WARNING - load average: 0.38, 1.01, 1.82 [20:12:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.89, 3.75 [20:13:41] RECOVERY - prometheus101 Current Load on prometheus101 is OK: OK - load average: 0.64, 0.85, 1.66 [20:16:09] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.62, 3.68 [20:18:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.77, 3.42, 3.86 [20:20:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cad3ba37992b...94084d11804b [20:20:42] [url] Comparing cad3ba37992b...94084d11804b · miraheze/mw-config · GitHub | github.com [20:20:43] [02miraheze/mw-config] 07Universal-Omega 0394084d1 - Update LocalSettings.php [20:20:44] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [20:20:45] [url] Page not found · GitHub · GitHub | github.com [20:21:47] miraheze/mw-config - Universal-Omega the build passed. [20:22:06] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.19, 3.05, 3.37 [20:22:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.52, 3.74, 3.88 [20:24:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.91, 3.99, 3.97 [20:26:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.46, 3.47 [20:26:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/94084d11804b...f0d951754373 [20:26:34] [url] Comparing 94084d11804b...f0d951754373 · miraheze/mw-config · GitHub | github.com [20:26:35] [02miraheze/mw-config] 07Universal-Omega 03f0d9517 - Update MirahezeFunctions.php [20:26:37] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [20:26:37] [url] Page not found · GitHub · GitHub | github.com [20:27:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f0d951754373...9344ae372f68 [20:27:42] [url] Comparing f0d951754373...9344ae372f68 · miraheze/mw-config · GitHub | github.com [20:27:43] [02miraheze/mw-config] 07Universal-Omega 039344ae3 - Update MirahezeFunctions.php [20:27:43] miraheze/mw-config - Universal-Omega the build passed. [20:27:44] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [20:27:45] [url] Page not found · GitHub · GitHub | github.com [20:28:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.78, 3.97, 3.65 [20:28:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.25, 4.12, 4.04 [20:28:44] miraheze/mw-config - Universal-Omega the build passed. [20:30:38] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.86, 2.45, 1.84 [20:32:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.04, 3.81, 3.96 [20:34:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.43, 3.56, 3.69 [20:34:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.42, 1.85, 1.73 [20:38:34] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.12, 1.55, 1.64 [20:41:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.86, 3.73 [20:44:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 3.40, 3.59 [20:45:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 4.00, 3.84 [20:47:27] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.90, 1.71, 1.62 [20:47:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.68, 4.04, 3.88 [20:48:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.62, 3.59, 3.67 [20:49:26] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.45, 1.92, 1.70 [20:49:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.66, 3.75 [20:51:25] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 2.00, 1.87, 1.71 [20:52:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.67, 4.56, 3.99 [20:53:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.03, 4.18, 3.91 [20:56:03] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.85, 1.54, 1.29 [20:57:22] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.17, 1.80, 1.73 [20:57:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.59, 3.74 [20:58:02] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.65, 1.23, 1.20 [20:58:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.97, 3.93 [20:59:21] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.97, 1.46, 1.61 [21:00:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.36, 10.43, 9.60 [21:02:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.25, 9.72, 9.45 [21:03:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.30, 3.52 [21:04:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.71, 3.63, 3.74 [21:05:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.29, 3.49 [21:06:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.71, 3.16, 3.55 [21:13:41] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.14, 4.30, 3.80 [21:15:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.52, 3.90, 3.71 [21:18:22] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.43, 1.59, 1.97 [21:18:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.14, 4.14, 3.79 [21:19:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 4.00, 3.78 [21:20:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.10, 3.95, 3.78 [21:21:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.30, 3.67, 3.68 [21:24:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.17, 4.38, 3.94 [21:26:22] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.40, 1.33, 1.68 [21:27:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.13, 4.01, 3.78 [21:29:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9344ae372f68...9643305acad8 [21:29:48] [url] Comparing 9344ae372f68...9643305acad8 · miraheze/mw-config · GitHub | github.com [21:29:49] [02miraheze/mw-config] 07Universal-Omega 039643305 - Update MirahezeFunctions.php [21:29:50] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [21:29:51] [url] Page not found · GitHub · GitHub | github.com [21:30:52] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.98, 1.77, 1.47 [21:30:56] miraheze/mw-config - Universal-Omega the build passed. [21:31:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.36, 3.46, 3.65 [21:32:22] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 4.64, 2.77, 2.12 [21:32:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.35, 3.46, 3.73 [21:32:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.46, 1.39 [21:33:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.58, 3.66 [21:36:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.03, 4.01, 3.86 [21:38:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.76, 3.59, 3.73 [21:39:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.42, 3.57 [21:41:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.43, 4.06, 3.76 [21:42:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 9.12, 5.09, 4.19 [21:45:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.88, 3.80 [21:47:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.74, 3.74 [21:51:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.74, 3.75, 3.79 [21:52:53] !log reboot test101 [21:53:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:29] PROBLEM - test101 JobRunner Service on test101 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [21:56:22] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.22, 1.17, 0.55 [21:56:28] RECOVERY - test101 JobRunner Service on test101 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [21:58:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.75, 3.60, 3.93 [21:59:02] !log killed matomo archive script, manually running purge in screen [21:59:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:00:22] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.03, 1.92, 0.99 [22:01:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.12, 3.52, 3.62 [22:03:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.48, 3.60 [22:04:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.26, 4.16, 4.00 [22:05:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 3.91, 3.74 [22:07:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.78, 3.72 [22:08:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.82, 3.94 [22:10:22] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.98, 1.55 [22:12:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9643305acad8...ceb53830ecdb [22:12:14] [url] Comparing 9643305acad8...ceb53830ecdb · miraheze/mw-config · GitHub | github.com [22:12:15] [02miraheze/mw-config] 07Universal-Omega 03ceb5383 - Update MirahezeFunctions.php [22:12:17] [02mw-config] 07Universal-Omega synchronize pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [22:12:17] [url] Page not found · GitHub · GitHub | github.com [22:12:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.20, 4.05, 3.96 [22:13:19] miraheze/mw-config - Universal-Omega the build passed. [22:16:22] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.06, 1.94, 1.67 [22:16:38] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:45] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:18:22] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.90, 1.96, 1.72 [22:18:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.29, 3.48, 3.78 [22:18:41] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 99069 bytes in 7.236 second response time [22:19:35] [02mw-config] 07Universal-Omega edited pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [22:19:36] [url] Page not found · GitHub · GitHub | github.com [22:20:28] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 9.88, 7.22, 4.85 [22:20:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.65, 4.32, 4.05 [22:22:58] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 99069 bytes in 3.732 second response time [22:23:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.57, 4.48, 3.94 [22:24:11] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.36, 1.94, 1.40 [22:24:22] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.45, 2.16, 1.86 [22:24:28] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 5.43, 7.32, 5.54 [22:25:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.84, 3.97, 3.80 [22:26:10] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.11, 1.60, 1.34 [22:26:28] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 4.75, 6.60, 5.50 [22:26:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.66, 3.88 [22:28:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.24, 1.77, 1.78 [22:28:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.32, 4.48, 4.14 [22:31:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.78, 4.20, 3.86 [22:32:22] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.25, 1.54, 1.68 [22:32:58] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.37, 4.70, 4.19 [22:33:36] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-7/±1] 13https://github.com/miraheze/mediawiki/compare/95c4fc9309e6...6f02b18070fd [22:33:37] [url] Comparing 95c4fc9309e6...6f02b18070fd · miraheze/mediawiki · GitHub | github.com [22:33:37] [02miraheze/mediawiki] 07Universal-Omega 036f02b18 - Remove extensions [22:34:51] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/6f02b18070fd...a641e08906da [22:34:52] [url] Comparing 6f02b18070fd...a641e08906da · miraheze/mediawiki · GitHub | github.com [22:34:53] [02miraheze/mediawiki] 07Universal-Omega 03a641e08 - Update .branches.json [22:34:53] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.91, 5.19, 4.41 [22:35:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.73, 3.59, 3.70 [22:35:58] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:36:48] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.64, 5.15, 4.48 [22:37:56] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 99069 bytes in 2.496 second response time [22:38:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.25, 3.73, 3.99 [22:38:44] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 4.39, 4.80, 4.43 [22:39:35] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.10, 2.87, 3.37 [22:42:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.47, 4.79, 4.31 [22:43:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.43, 3.51 [22:44:17] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/c8814d08dde1 [22:44:19] [02miraheze/ssl] 07Universal-Omega 03c8814d0 - Change clockupwiki.org to en.clockupwiki.org [22:44:20] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [22:44:20] [url] Page not found · GitHub · GitHub | github.com [22:44:22] [02ssl] 07Universal-Omega opened pull request 03#546: Change clockupwiki.org to en.clockupwiki.org - 13https://github.com/miraheze/ssl/pull/546 [22:44:22] [url] Page not found · GitHub · GitHub | github.com [22:44:47] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/c8814d08dde1...a5d2320b40dc [22:44:47] [url] Comparing c8814d08dde1...a5d2320b40dc · miraheze/ssl · GitHub | github.com [22:44:48] [02miraheze/ssl] 07Universal-Omega 03a5d2320 - Add cert [22:44:50] [02ssl] 07Universal-Omega synchronize pull request 03#546: Change clockupwiki.org to en.clockupwiki.org - 13https://github.com/miraheze/ssl/pull/546 [22:44:50] [url] Page not found · GitHub · GitHub | github.com [22:45:15] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a5d2320b40dc...ab204fc137ac [22:45:15] [url] Comparing a5d2320b40dc...ab204fc137ac · miraheze/ssl · GitHub | github.com [22:45:16] [02miraheze/ssl] 07Universal-Omega 03ab204fc - Update certs.yaml [22:45:18] [02ssl] 07Universal-Omega synchronize pull request 03#546: Change clockupwiki.org to en.clockupwiki.org - 13https://github.com/miraheze/ssl/pull/546 [22:45:18] [url] Page not found · GitHub · GitHub | github.com [22:46:24] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+1/-1/±1] 13https://github.com/miraheze/ssl/compare/29292c84ab50...316a2dc0cc43 [22:46:24] [url] Comparing 29292c84ab50...316a2dc0cc43 · miraheze/ssl · GitHub | github.com [22:46:25] [02miraheze/ssl] 07Universal-Omega 03316a2dc - Change clockupwiki.org to en.clockupwiki.org (#546) [22:46:27] [02ssl] 07Universal-Omega closed pull request 03#546: Change clockupwiki.org to en.clockupwiki.org - 13https://github.com/miraheze/ssl/pull/546 [22:46:27] [url] Page not found · GitHub · GitHub | github.com [22:46:28] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [22:46:29] ... [22:46:30] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [22:47:35] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.52, 3.13, 3.39 [22:47:51] [02dns] 07Universal-Omega opened pull request 03#288: Remove clockupwiki.org zone - 13https://github.com/miraheze/dns/pull/288 [22:47:51] [url] Page not found · GitHub · GitHub | github.com [22:50:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.74, 1.62, 1.21 [22:52:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.41, 1.18 [22:53:26] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/clockupwiki.org.crt] [22:54:37] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.82, 4.87, 4.35 [22:55:25] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [22:56:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.49, 3.83 [22:58:01] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/4a46ef6b23a3 [22:58:02] [02miraheze/ssl] 07Universal-Omega 034a46ef6 - Remove wiki.ripto.gq cert [22:58:04] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [22:58:04] [url] Page not found · GitHub · GitHub | github.com [22:58:05] [02ssl] 07Universal-Omega opened pull request 03#547: Remove wiki.ripto.gq cert - 13https://github.com/miraheze/ssl/pull/547 [22:58:06] ... [22:58:30] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4a46ef6b23a3...dffa34560f8f [22:58:31] [url] Comparing 4a46ef6b23a3...dffa34560f8f · miraheze/ssl · GitHub | github.com [22:58:32] [02miraheze/ssl] 07Universal-Omega 03dffa345 - Update certs.yaml [22:58:34] [02ssl] 07Universal-Omega synchronize pull request 03#547: Remove wiki.ripto.gq cert - 13https://github.com/miraheze/ssl/pull/547 [22:58:34] [url] Page not found · GitHub · GitHub | github.com [23:00:45] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 10.26, 8.21, 6.04 [23:01:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.07, 2.63, 3.36 [23:02:18] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [23:02:19] [url] Page not found · GitHub · GitHub | github.com [23:02:20] [02ssl] 07Universal-Omega closed pull request 03#547: Remove wiki.ripto.gq cert - 13https://github.com/miraheze/ssl/pull/547 [23:02:20] ... [23:02:21] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [23:02:23] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/316a2dc0cc43...9ae71bea8b80 [23:02:23] [url] Comparing 316a2dc0cc43...9ae71bea8b80 · miraheze/ssl · GitHub | github.com [23:02:24] [02miraheze/ssl] 07Universal-Omega 039ae71be - Remove wiki.ripto.gq cert (#547) [23:03:05] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:08] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:41] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:42] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:00] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.430 second response time [23:05:12] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 8.152 second response time [23:05:44] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 99046 bytes in 7.451 second response time [23:05:44] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 99077 bytes in 7.443 second response time [23:06:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.58, 3.95 [23:08:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.86, 3.99, 4.07 [23:09:33] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.014 second response time [23:10:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.56, 3.11, 3.74 [23:10:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.17, 2.25, 1.72 [23:11:35] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 99077 bytes in 7.711 second response time [23:15:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.96, 2.03, 1.49 [23:16:39] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:16:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.09, 3.41, 3.09 [23:18:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.02, 3.35, 3.11 [23:18:41] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 99069 bytes in 1.751 second response time [23:19:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.52, 2.00, 1.62 [23:20:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.69, 3.80, 3.78 [23:21:13] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.97, 1.68, 1.55 [23:21:50] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.75, 2.36, 1.59 [23:22:53] !log [@mwtask111] starting deploy of {'config': True} to all [23:22:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:23:50] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.37, 1.90, 1.51 [23:24:12] [02mw-config] 07Universal-Omega closed pull request 03#4746: Improvements to SiteConfiguration - 13https://github.com/miraheze/mw-config/pull/4746 [23:24:13] [url] Page not found · GitHub · GitHub | github.com [23:24:14] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [23:24:14] [url] Page not found · GitHub · GitHub | github.com [23:24:15] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [23:24:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±8] 13https://github.com/miraheze/mw-config/compare/9616ab5c6eba...b39915619e02 [23:24:17] [url] Comparing 9616ab5c6eba...b39915619e02 · miraheze/mw-config · GitHub | github.com [23:24:18] [02miraheze/mw-config] 07Universal-Omega 03b399156 - Improvements to SiteConfiguration (#4746) [23:24:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.12, 4.73, 3.72 [23:25:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.02, 4.45, 3.65 [23:25:20] miraheze/mw-config - Universal-Omega the build passed. [23:25:50] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.96, 1.58, 1.44 [23:26:41] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:26:48] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:26:53] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [23:27:12] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:27:16] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [23:27:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.65, 10.42, 8.07 [23:27:30] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:27:33] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:27:39] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:27:40] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:27:45] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:27:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:09] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 52s [23:28:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.29, 10.65, 8.54 [23:28:45] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:29:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:16] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:31:35] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [23:31:37] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [23:31:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.49, 1.78, 1.89 [23:31:38] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:31:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 15.66, 13.18, 9.83 [23:32:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:32:25] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 8.57, 6.71, 4.71 [23:33:23] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 9.567 second response time [23:33:31] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 8.435 second response time [23:33:36] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.47, 2.08, 2.00 [23:33:43] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 6.919 second response time [23:33:47] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23887 bytes in 8.389 second response time [23:33:47] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 7.033 second response time [23:33:57] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [23:34:00] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 5.141 second response time [23:34:02] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 7.579 second response time [23:34:14] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 7.263 second response time [23:35:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b39915619e02...edf7d5e5345a [23:35:01] [url] Comparing b39915619e02...edf7d5e5345a · miraheze/mw-config · GitHub | github.com [23:35:02] [02miraheze/mw-config] 07Universal-Omega 03edf7d5e - Re-enable Maps on 1.38 [23:35:54] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:36:10] miraheze/mw-config - Universal-Omega the build passed. [23:37:55] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:38:06] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:38:28] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:39:41] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 8.579 second response time [23:39:43] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23887 bytes in 8.232 second response time [23:39:57] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 7.690 second response time [23:41:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.15, 11.95, 10.85 [23:43:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.59, 12.49, 11.18 [23:44:04] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:05] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:05] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:47:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.57, 11.95, 11.20 [23:48:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.92, 11.99, 11.58 [23:49:32] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:43] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:49] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:06] !log [@test101] starting deploy of {'config': True} to all [23:50:07] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [23:50:57] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:51:00] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:51:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.28, 3.72, 3.99 [23:52:09] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgUseCategoryBrowser --sourcelist=false --remove [23:52:14] [02mw-config] 07Universal-Omega closed pull request 03#4748: Remove `$wgUseCategoryBrowser` - 13https://github.com/miraheze/mw-config/pull/4748 [23:52:15] [url] Page not found · GitHub · GitHub | github.com [23:52:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/edf7d5e5345a...a940c1f3a813 [23:52:16] [url] Comparing edf7d5e5345a...a940c1f3a813 · miraheze/mw-config · GitHub | github.com [23:52:17] [02miraheze/mw-config] 07Universal-Omega 03a940c1f - Remove `$wgUseCategoryBrowser` (#4748) [23:52:19] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 [23:52:20] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://github.com/miraheze/mw-config [23:52:21] [url] Page not found · GitHub · GitHub | github.com [23:52:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.91, 11.98, 11.61 [23:52:57] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.70, 11.39, 11.04 [23:53:24] miraheze/mw-config - Universal-Omega the build passed. [23:53:39] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 4.16, 6.85, 7.12 [23:53:41] !log [@mwtask111] starting deploy of {'config': True} to all [23:53:55] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.025 second response time [23:54:04] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.317 second response time [23:54:07] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 26s [23:54:11] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.028 second response time [23:54:12] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23887 bytes in 0.429 second response time [23:54:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.76, 11.09, 11.36 [23:54:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.92, 11.64, 11.18 [23:55:07] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 8.481 second response time [23:55:11] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 8.776 second response time [23:55:24] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.40, 3.97, 4.02 [23:55:26] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.312 second response time [23:55:55] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 8.597 second response time [23:56:49] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.73, 12.45, 11.54 [23:57:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.29, 3.87 [23:57:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.84, 3.96 [23:58:21] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 9.477 second response time [23:59:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.73, 3.79, 3.98 [23:59:18] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.39, 4.30, 4.10 [23:59:27] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 3.83, 5.39, 6.42