[00:03:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.61, 9.96, 9.89 [00:06:20] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 405 failures. Last run 5 minutes ago with 405 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/puppet],File[/etc/update-motd.d/97-last-puppet-run],File[/etc/ferm/conf.d/02_main],File[/etc/sudoers] [00:06:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.57, 9.57, 9.74 [00:08:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.14, 9.15, 9.85 [00:11:21] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.39, 9.89, 9.39 [00:12:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.06, 10.31, 9.96 [00:17:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.56, 9.82, 9.83 [00:18:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.32, 3.24, 2.76 [00:18:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.70, 9.97, 9.97 [00:20:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.44, 9.12, 9.25 [00:20:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.23, 9.26, 9.63 [00:21:09] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.90, 9.46, 9.91 [00:21:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.48, 2.70, 2.62 [00:28:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.13, 9.43, 9.55 [00:30:31] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.40, 10.36, 9.91 [00:30:55] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:31:42] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [00:32:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.10, 8.58, 9.16 [00:34:04] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.62, 9.65, 9.54 [00:34:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.49, 9.63, 9.74 [00:35:38] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:37:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.91, 9.23, 9.39 [00:37:41] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:37:45] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.20, 9.76, 9.76 [00:39:14] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures [00:39:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.32, 9.40, 9.23 [00:40:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.81, 9.43, 9.02 [00:41:03] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [00:41:04] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.24, 9.95, 9.67 [00:41:45] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22202 bytes in 1.169 second response time [00:43:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.17, 9.19, 9.03 [00:45:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.29, 9.61, 9.41 [00:46:52] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.75, 9.55, 9.18 [00:50:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.64, 8.50, 8.83 [00:50:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.92, 9.26, 9.66 [00:53:06] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.71, 2.13, 1.78 [00:53:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.68, 9.07, 9.76 [00:54:00] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.59, 9.75, 9.75 [00:55:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.43, 9.66, 9.46 [00:56:07] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.90, 1.54, 1.60 [00:58:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.85, 9.52, 9.43 [00:59:43] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.47, 9.17, 9.46 [01:02:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.83, 9.35, 9.50 [01:02:33] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.30, 7.43, 8.22 [01:10:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.43, 9.59, 9.39 [01:10:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:12:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.11, 9.93, 8.99 [01:13:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.42, 9.14, 9.30 [01:15:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.61, 8.57, 8.64 [01:16:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:17:26] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.52, 9.20, 9.19 [01:18:50] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.41, 7.91, 8.40 [01:20:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.31, 8.67, 8.98 [01:20:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.77, 9.05, 9.71 [01:22:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.57, 9.39, 9.20 [01:23:53] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.01, 10.81, 10.26 [01:25:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.68, 8.69, 8.96 [01:25:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.28, 8.91, 8.64 [01:26:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.30, 9.78, 9.98 [01:30:37] [02puppet] 07Universal-Omega opened pull request 03#2373: Increase memcached memory to 15GB - 13https://github.com/miraheze/puppet/pull/2373 [01:31:00] [02puppet] 07Universal-Omega synchronize pull request 03#2373: Increase memcached memory to 15GB - 13https://github.com/miraheze/puppet/pull/2373 [01:32:50] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.22, 9.57, 9.81 [01:37:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.77, 8.30, 8.48 [01:37:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.90, 8.18, 8.41 [01:38:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.49, 9.30, 9.77 [01:40:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.09, 9.66, 9.25 [01:41:34] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.08, 1.10, 1.99 [01:41:39] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.51, 9.71, 9.83 [01:44:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.01, 9.19, 8.78 [01:44:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.13, 8.89, 8.74 [01:46:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.74, 9.97, 9.58 [01:47:46] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.05, 1.07, 1.67 [01:49:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.35, 10.18, 9.70 [01:52:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.87, 9.92, 9.69 [02:05:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.86, 9.46, 9.71 [02:10:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.91, 9.94, 9.55 [02:11:46] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.71, 10.64, 9.83 [02:14:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.79, 10.05, 9.79 [02:16:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.79, 9.73, 9.62 [02:19:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.99, 9.94, 9.71 [02:19:55] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.48, 9.26, 9.58 [02:22:54] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.59, 10.20, 9.85 [02:26:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.83, 9.22, 9.55 [02:29:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.09, 9.51, 9.58 [02:32:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.25, 8.34, 9.10 [02:32:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.28, 9.35, 9.74 [02:35:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.56, 10.23, 10.04 [02:36:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.18, 8.44, 9.56 [02:39:04] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.96, 9.02, 9.55 [02:41:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.52, 10.13, 9.65 [02:42:06] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.38, 9.25, 9.55 [02:45:07] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.44, 9.32, 9.49 [02:47:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.97, 9.50, 9.55 [02:51:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.54, 8.72, 9.26 [02:55:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.78, 8.58, 9.75 [02:56:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.13, 11.22, 10.20 [03:01:10] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.59, 9.58, 9.78 [03:02:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.33, 9.92, 9.95 [03:04:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.70, 9.34, 9.67 [03:08:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.39, 9.88, 9.81 [03:09:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.38, 9.91, 9.77 [03:11:53] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.03, 9.81, 9.35 [03:12:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.93, 9.60, 9.69 [03:13:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f01e0fd1a525...440be3c410b9 [03:13:30] [02miraheze/mw-config] 07Universal-Omega 03440be3c - Update GlobalCache.php [03:13:32] [02mw-config] 07Universal-Omega synchronize pull request 03#4418: Enable sidebar cache - 13https://github.com/miraheze/mw-config/pull/4418 [03:14:36] miraheze/mw-config - Universal-Omega the build passed. [03:14:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.56, 8.91, 9.10 [03:15:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.70, 10.21, 9.88 [03:16:11] [02mw-config] 07Universal-Omega closed pull request 03#4418: Enable sidebar cache - 13https://github.com/miraheze/mw-config/pull/4418 [03:16:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/df90aa884de7...018cc8d81fc9 [03:16:14] [02miraheze/mw-config] 07Universal-Omega 03018cc8d - Enable sidebar cache (#4418) [03:16:16] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 [03:16:17] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config [03:17:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/018cc8d81fc9...8d74c76896c2 [03:17:03] [02miraheze/mw-config] 07Universal-Omega 038d74c76 - Decrease luasandbox cpuLimit to 4 [03:17:21] miraheze/mw-config - Universal-Omega the build passed. [03:17:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.75, 9.02, 9.78 [03:17:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.35, 9.29, 9.64 [03:18:05] miraheze/mw-config - Universal-Omega the build passed. [03:18:36] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.08, 2.37, 1.65 [03:18:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/e1b3597ea7ac [03:19:00] [02miraheze/mw-config] 07Universal-Omega 03e1b3597 - Move $wgRevisionCacheExpiry to GlobalCache.php [03:19:01] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config [03:19:03] [02mw-config] 07Universal-Omega opened pull request 03#4423: Move $wgRevisionCacheExpiry to GlobalCache.php - 13https://github.com/miraheze/mw-config/pull/4423 [03:20:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e1b3597ea7ac...9c1fd2391f3e [03:20:09] [02miraheze/mw-config] 07Universal-Omega 039c1fd23 - Update GlobalCache.php [03:20:10] miraheze/mw-config - Universal-Omega the build passed. [03:20:10] [02mw-config] 07Universal-Omega synchronize pull request 03#4423: Move $wgRevisionCacheExpiry to GlobalCache.php - 13https://github.com/miraheze/mw-config/pull/4423 [03:20:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.43, 10.28, 10.18 [03:20:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9c1fd2391f3e...ed3607c76548 [03:20:53] [02miraheze/mw-config] 07Universal-Omega 03ed3607c - Update GlobalCache.php [03:20:54] [02mw-config] 07Universal-Omega synchronize pull request 03#4423: Move $wgRevisionCacheExpiry to GlobalCache.php - 13https://github.com/miraheze/mw-config/pull/4423 [03:21:06] [02mw-config] 07Universal-Omega edited pull request 03#4423: Move $wgRevisionCacheExpiry to GlobalCache.php and increase - 13https://github.com/miraheze/mw-config/pull/4423 [03:21:14] miraheze/mw-config - Universal-Omega the build passed. [03:21:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/8d74c76896c2...0be9b2465ba2 [03:21:28] [02miraheze/mw-config] 07Universal-Omega 030be9b24 - Move $wgRevisionCacheExpiry to GlobalCache.php and increase (#4423) [03:21:29] [02mw-config] 07Universal-Omega closed pull request 03#4423: Move $wgRevisionCacheExpiry to GlobalCache.php and increase - 13https://github.com/miraheze/mw-config/pull/4423 [03:21:31] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 [03:21:32] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config [03:21:53] miraheze/mw-config - Universal-Omega the build passed. [03:22:39] miraheze/mw-config - Universal-Omega the build passed. [03:23:00] !log [@mwtask111] starting deploy of {'config': True} to scsvg [03:23:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:13] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 12s [03:23:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.77, 9.90, 9.29 [03:26:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.46, 9.43, 9.40 [03:27:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.18, 9.80, 9.99 [03:29:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.46, 9.76, 9.95 [03:29:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.63, 9.29, 9.33 [03:30:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.67, 9.56, 9.44 [03:32:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.65, 10.32, 10.12 [03:32:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.29, 9.68, 9.44 [03:33:45] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.17, 10.16, 10.05 [03:36:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.16, 9.57, 9.46 [03:36:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.30, 9.60, 9.84 [03:39:10] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.70, 9.56, 9.45 [03:39:43] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.29, 10.10, 10.00 [03:42:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.31, 9.96, 9.65 [03:46:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0be9b2465ba2...8e47fd6fae3f [03:46:35] [02miraheze/mw-config] 07Universal-Omega 038e47fd6 - Increase $wgRevisionCacheExpiry to 7 days [03:47:37] miraheze/mw-config - Universal-Omega the build passed. [03:53:34] !log [@mwtask111] starting deploy of {'config': True} to scsvg [03:53:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:53:43] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 9s [03:53:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:53:54] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.82, 9.97, 9.99 [03:53:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.95, 9.53, 9.74 [03:56:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.70, 9.55, 9.95 [03:56:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.75, 10.27, 9.97 [03:59:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.17, 10.42, 10.21 [03:59:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.09, 10.05, 10.00 [04:06:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.08, 8.92, 9.65 [04:06:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.22, 9.18, 9.75 [04:06:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.31, 9.51, 9.95 [04:09:15] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 9.16, 9.97, 10.10 [04:11:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.39, 8.56, 9.64 [04:12:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.16, 8.84, 9.65 [04:12:06] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.22, 9.34, 9.59 [04:14:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.01, 9.40, 9.59 [04:15:06] User reported on Discord issues while searching on `lgbtawiki`. It appears all searches result in a database error: [1aedb3ed051209dad1f9e7bd] 2022-01-21 04:14:08: Fatal exception of type "Wikimedia\Rdbms\DBQueryError" [04:17:19] Likely do to the textindex being off. [04:17:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.08, 9.24, 9.46 [04:20:05] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.36, 10.37, 10.15 [04:20:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.48, 8.81, 9.28 [04:22:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.31, 9.40, 9.81 [04:25:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.40, 4.10, 3.16 [04:25:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.55, 10.33, 10.10 [04:32:29] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.33, 7.48, 8.28 [04:32:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.31, 9.46, 9.25 [04:35:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.49, 9.14, 9.16 [04:37:06] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.60, 0.89, 1.77 [04:40:07] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.57, 0.70, 1.53 [04:41:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.59, 9.10, 9.86 [04:43:56] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.78, 7.28, 8.24 [04:44:10] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.27, 9.70, 9.94 [04:44:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.55, 10.21, 9.66 [04:47:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.00, 9.79, 9.60 [04:50:24] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.09, 9.25, 8.69 [04:53:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.94, 9.28, 8.80 [04:53:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.98, 8.59, 8.55 [04:55:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.37, 9.08, 9.79 [04:56:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.10, 9.32, 9.31 [04:56:21] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.17, 9.14, 8.76 [04:57:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.35, 9.48, 9.81 [04:59:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.16, 9.19, 8.88 [04:59:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.48, 8.54, 9.02 [05:02:04] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.26, 10.03, 9.24 [05:02:04] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.01, 8.29, 8.50 [05:04:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.19, 9.13, 9.75 [05:04:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.03, 8.49, 8.78 [05:05:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.25, 7.57, 8.43 [05:06:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.04, 9.02, 9.51 [05:09:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.28, 9.11, 9.43 [05:11:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.57, 9.52, 8.96 [05:11:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.46, 9.07, 8.76 [05:12:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.05, 8.61, 9.16 [05:14:25] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.42, 9.97, 9.20 [05:15:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.76, 9.62, 9.42 [05:16:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.03, 9.74, 9.21 [05:16:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:16:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.61, 8.60, 9.73 [05:17:13] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.37, 9.53, 9.15 [05:17:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.98, 8.75, 8.75 [05:18:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.72, 9.58, 9.26 [05:18:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.88, 8.94, 9.20 [05:18:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.81, 9.28, 9.16 [05:23:07] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.15, 9.06, 9.02 [05:25:54] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.80, 8.64, 8.84 [05:26:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.64, 8.99, 9.00 [05:29:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.07, 9.66, 9.67 [05:29:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.18, 8.91, 8.96 [05:30:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.32, 9.38, 9.34 [05:30:53] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.03, 9.12, 9.48 [05:31:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.44, 8.97, 8.79 [05:31:49] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.53, 9.41, 8.99 [05:32:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.08, 9.36, 9.12 [05:33:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.37, 9.38, 9.54 [05:34:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.81, 9.40, 8.98 [05:34:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.07, 8.92, 8.88 [05:35:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.02, 9.42, 9.21 [05:36:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.92, 10.76, 10.05 [05:37:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.45, 9.53, 9.58 [05:38:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.74, 9.55, 9.68 [05:39:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.62, 9.69, 9.76 [05:40:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.51, 9.21, 9.43 [05:40:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.19, 8.67, 8.70 [05:41:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:43:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.67, 9.39, 9.00 [05:43:42] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.09, 9.27, 9.06 [05:45:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:45:56] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.26, 9.41, 9.39 [05:48:51] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.23, 1.39, 1.89 [05:49:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.51, 9.28, 9.23 [05:50:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:50:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.87, 10.10, 9.76 [05:51:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.40, 9.50, 9.15 [05:52:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.29, 9.37, 9.25 [05:52:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.28, 10.28, 9.76 [05:54:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.69, 9.17, 9.42 [05:54:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.16, 7.97, 8.39 [05:55:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.46, 8.08, 8.77 [05:55:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.00, 9.48, 9.52 [05:57:11] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.30, 9.34, 9.39 [05:57:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:00:26] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.06, 1.28, 1.62 [06:00:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.17, 9.87, 9.72 [06:01:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.26, 9.76, 9.53 [06:02:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.19, 9.92, 9.73 [06:03:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.85, 9.63, 9.19 [06:03:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.89, 8.87, 9.22 [06:04:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.17, 9.64, 9.89 [06:06:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.07, 9.60, 9.26 [06:09:41] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.74, 2.21, 1.87 [06:10:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.69, 9.65, 9.73 [06:12:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.32, 7.46, 8.31 [06:13:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.96, 9.71, 9.74 [06:17:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:18:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.16, 9.67, 9.69 [06:19:35] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.26, 10.13, 9.55 [06:20:42] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.97, 9.58, 9.18 [06:23:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.22, 9.69, 9.05 [06:24:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:29:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.13, 9.90, 9.72 [06:29:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.13, 9.29, 9.18 [06:29:32] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=ftpwwiki --username-prefix wikia:futurepolandball --report 1 --no-updates /home/reception/futurepolandball_pages_full.xml (START) [06:29:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:30:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.76, 9.96, 9.71 [06:32:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.34, 10.33, 9.85 [06:35:31] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f482270a3abf...ed684e4a16cf [06:35:33] [02miraheze/puppet] 07Reception123 03ed684e4 - add ED25519 key for myself (reception) [06:37:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.84, 9.31, 9.69 [06:47:20] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=ftpwwiki --username-prefix wikia:futurepolandball --report 1 --no-updates /home/reception/futurepolandball_pages_full.xml (END - exit=0) [06:47:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:47:40] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=ftpwwiki (START) [06:47:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:50:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.47, 7.82, 8.32 [06:50:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.49, 9.88, 9.48 [06:51:02] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=ftpwwiki (END - exit=256) [06:51:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:53:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.12, 9.89, 9.56 [06:54:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:56:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.91, 3.21, 2.59 [06:56:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.01, 9.13, 9.97 [06:56:31] Reception123: FYI rebuildtextindex failed there so search is broken now. https://ftpw.miraheze.org/w/index.php?search=Test&title=Special%3ASearch&go=Go [06:56:34] [url] Permission error - Future Polandball Wiki | ftpw.miraheze.org [06:56:39] CosmicAlpha: yeah, running that now [06:56:44] CosmicAlpha: it's always that annoying too long error [06:56:49] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.74, 8.93, 8.58 [06:57:14] CosmicAlpha: it also failed with rebuildtextindex.php individually now so there's not much that can be done then :( [06:58:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.86, 8.21, 8.46 [06:58:28] Reception123: it won't ever work if it fails rebuildall. Rebuildall actually runs rebuildtextindex, so if it fails there so will the script individually. You can fix it the way we've done it before, manually insert the indexes. [06:59:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.87, 2.97 [06:59:05] oh [06:59:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.06, 9.97, 9.61 [06:59:22] CosmicAlpha: so just ALTER TABLE searchindex ADD FULLTEXT INDEX si_title (si_title), ADD FULLTEXT INDEX si_text (si_text); ? [06:59:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.60, 8.29, 8.39 [07:00:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:00:45] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.70, 9.46, 8.62 [07:01:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.05, 3.63, 3.03 [07:01:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.83, 8.97, 9.28 [07:02:52] same issue on `lgbtawiki` after their import finished [07:03:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.34, 8.60, 8.44 [07:04:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.40, 2.84, 2.81 [07:04:47] Reception123: I think so, but can't confirm with 100% certainty bit if not it won't do damage just fail. [07:04:59] !log ALTER TABLE searchindex ADD FULLTEXT si_text (si_text); (ftpwwiki) [07:05:03] !log ALTER TABLE searchindex ADD FULLTEXT si_text (si_text); (lgbtawiki) [07:05:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:05:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:05:17] Thanks! [07:05:58] Don't forget si_title also though I think. [07:09:11] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.15, 8.44, 8.41 [07:13:39] !log granted crat/sysop to my test account on betawiki using createAndPromote.php [07:13:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:14:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.65, 8.79, 8.28 [07:16:06] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.22, 9.42, 9.35 [07:18:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.63, 8.70, 8.57 [07:18:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.29, 10.06, 9.26 [07:18:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.58, 8.72, 8.36 [07:20:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:21:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.99, 9.07, 9.06 [07:21:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.33, 8.40, 8.29 [07:21:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.10, 8.81, 9.13 [07:24:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.24, 9.24, 9.21 [07:25:54] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.00, 8.10, 8.24 [07:27:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.96, 9.62, 9.37 [07:27:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:29:40] [02mw-config] 07Reception123 commented on pull request 03#4411: Add PropertySuggester to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4411#issuecomment-1018252024 [07:29:44] [02mw-config] 07Reception123 closed pull request 03#4411: Add PropertySuggester to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4411 [07:29:45] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/8e47fd6fae3f...5a7ebeddd9f7 [07:29:47] [02miraheze/mw-config] 07lens0021 035a7ebed - Add PropertySuggester to ManageWikiExtensions (#4411) [07:30:17] !log [reception@mwtask111] starting deploy of {'config': True, 'l10nupdate': True, 'l10n': True, 'extensionlist': True} to scsvg [07:30:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:30:46] miraheze/mw-config - Reception123 the build passed. [07:32:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.06, 10.06, 9.57 [07:34:53] 12:05 AM <+ CosmicAlpha> Don't forget si_title also though I think. [07:34:57] Reception123: ^ [07:35:35] CosmicAlpha: oh, I checked SAL and when you did it last on subcultureswiki you only did what I did just now [07:35:50] though I guess that was missed [07:36:07] I thought I did both... I could've forgotten to log the other... [07:36:59] !log ALTER TABLE searchindex ADD FULLTEXT INDEX si_title (si_title) (ftpwwiki) [07:37:03] !log ALTER TABLE searchindex ADD FULLTEXT INDEX si_title (si_title) (lgbtawiki) [07:37:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:37:07] > 19:31 Universal Omega: ALTER TABLE searchindex ADD FULLTEXT si_title (si_title); on subcultureswiki [07:37:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:37:12] Reception123: ^ [07:37:15] I did log it [07:37:59] ah, I thought all those three were the same command but only the first two were [07:38:03] title and text look so similar :P [07:38:12] Though for some reason logged si_text twice there... [07:39:45] I wonder why? [07:40:16] Maybe because it was when there was 2 log bots here... though I thought we cleaned that up. Guess not. [07:40:44] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.68, 8.21, 8.50 [07:40:51] ah yeah, perhaps [07:41:10] but then why didn't title log twice? perhaps most were cleaned but not the text one [07:41:28] It could've been cleaned and just text was missed. [07:41:43] CosmicAlpha: also what's the difference between --l18n and --l18nupdate for deploy-mediawiki? [07:42:34] I18nupdate is from upstream. Its on a cron, LocalisationUpdate extension. [07:42:49] oh [07:43:37] !log [reception@mwtask111] finished deploy of {'config': True, 'l10nupdate': True, 'l10n': True, 'extensionlist': True} to scsvg - FAIL: [0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] in 799s [07:43:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:43:50] !log [reception@mwtask111] starting deploy of {'config': True, 'l10n': True, 'extensionlist': True} to scsvg [07:43:52] --l10nupdate runs the script to update l10n from git likes runs nightly [07:43:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:43:59] --l10n rebuilds l10n [07:44:20] I see [07:44:23] Reception123: that's alot of zeros... [07:44:29] CosmicAlpha: 0 is good [07:44:38] not sure why there's a 2 there [07:44:39] 0 means success [07:44:41] I know... it was a statement. [07:44:45] The 2 is a failure [07:44:55] Reception123: whichever command failed exited with code 2 [07:45:09] yeah I accidentally used --l18nupdate as I wasn't sure if both were needed now [07:45:25] No [07:45:30] yeah, I realise now [07:45:35] No human ever needs to use the update one [07:47:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.09, 9.66, 9.92 [07:49:13] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.98, 10.57, 9.80 [07:50:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.32, 10.38, 10.16 [07:52:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.67, 9.94, 9.71 [07:53:45] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_37 [+1/-0/±1] 13https://github.com/miraheze/mediawiki/compare/7cead6e887af...9e0c35907009 [07:53:47] [02miraheze/mediawiki] 07Reception123 039e0c359 - Install QuickInstantCommons (per T8626) [07:53:53] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.12, 9.89, 8.82 [07:54:17] Reception123: we should make that work with MH commons too [07:54:28] yeah that's a good idea [07:54:51] It's under advanced config [07:54:55] It should make sense [07:55:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.82, 9.29, 9.83 [07:56:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.91, 9.52, 8.83 [07:57:45] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.03, 9.32, 9.45 [07:58:15] !log [reception@mwtask111] finished deploy of {'config': True, 'l10n': True, 'extensionlist': True} to scsvg - SUCCESS in 865s [07:58:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:58:20] finally [07:58:25] [02mw-config] 07Reception123 closed pull request 03#4410: Set PropertySuggesterInitialSuggestions for horimiyawiki - 13https://github.com/miraheze/mw-config/pull/4410 [07:58:27] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5a7ebeddd9f7...3b74242661fa [07:58:28] [02miraheze/mw-config] 07lens0021 033b74242 - Set PropertySuggesterInitialSuggestions for horimiyawiki (#4410) [07:58:34] !log [reception@mwtask111] starting deploy of {'config': True} to scsvg [07:58:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:58:43] !log [reception@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 8s [07:58:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.58, 10.05, 10.01 [07:58:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:59:25] !log [@mwtask111] starting deploy of {'config': True} to scsvg [07:59:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:59:35] miraheze/mw-config - Reception123 the build passed. [07:59:36] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 10s [07:59:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:00:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.86, 9.37, 9.46 [08:01:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.52, 9.69, 9.90 [08:04:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.35, 10.36, 10.12 [08:09:02] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.45, 7.60, 8.48 [08:09:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.52, 9.41, 9.86 [08:11:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.04, 8.72, 8.76 [08:11:42] RhinosF1: I was going to configure that also. [08:12:15] It should be a very simple change. [08:12:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:12:53] Also was for the shared uploads config. [08:13:32] CosmicAlpha: thank you! [08:13:36] You are amazing [08:13:50] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.25, 7.72, 8.38 [08:20:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:21:12] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.18, 9.27, 9.37 [08:23:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.12, 9.14, 8.83 [08:23:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.03, 8.87, 8.74 [08:24:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.84, 9.26, 9.33 [08:24:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.47, 8.30, 7.99 [08:25:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:25:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.02, 9.43, 8.99 [08:26:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 10.39, 9.74 [08:28:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:28:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.42, 8.97, 8.91 [08:29:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.60, 9.81, 9.62 [08:29:53] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.01, 7.65, 7.84 [08:32:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.77, 9.92, 8.97 [08:35:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.21, 10.52, 9.84 [08:35:33] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ed684e4a16cf...814ebf15a088 [08:35:34] [02miraheze/puppet] 07Reception123 03814ebf1 - rm 2nd key for me (reception) [08:35:37] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.82, 8.25, 8.50 [08:36:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.02, 8.51, 8.25 [08:36:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.96, 9.25, 8.54 [08:38:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.13, 9.82, 9.68 [08:38:52] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.40, 7.90, 8.04 [08:40:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.68, 7.93, 8.40 [08:42:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.50, 7.82, 8.50 [08:43:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:44:43] [02puppet] 07Universal-Omega opened pull request 03#2374: MediaWiki: include nodejs everywhere - 13https://github.com/miraheze/puppet/pull/2374 [08:45:09] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.27, 8.18, 8.42 [08:45:17] [02puppet] 07Universal-Omega synchronize pull request 03#2374: MediaWiki: include nodejs everywhere - 13https://github.com/miraheze/puppet/pull/2374 [08:48:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.38, 8.47, 8.11 [08:49:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.62, 10.26, 9.27 [08:49:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:50:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.16, 8.43, 8.16 [08:51:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.92, 9.28, 9.05 [08:54:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:57:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:57:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.30, 9.45, 9.25 [08:59:53] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.85, 7.84, 8.43 [09:00:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.40, 8.69, 8.38 [09:00:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.53, 8.63, 8.98 [09:02:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:02:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.79, 8.22, 8.28 [09:03:51] [02puppet] 07Reception123 closed pull request 03#2374: MediaWiki: include nodejs everywhere - 13https://github.com/miraheze/puppet/pull/2374 [09:03:52] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/814ebf15a088...0cb20926bc32 [09:03:54] [02miraheze/puppet] 07Universal-Omega 030cb2092 - MediaWiki: include nodejs everywhere (#2374) [09:12:34] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:16:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:19:06] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.40, 10.60, 9.31 [09:21:27] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:21:40] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:22:26] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:22:32] PROBLEM - mwtask111 ferm_active on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:23:04] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:23:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.95, 10.45, 9.03 [09:23:58] PROBLEM - mwtask111 HTTPS on mwtask111 is CRITICAL: connect to address 2a10:6740::6:208 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [09:24:46] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [09:24:54] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [09:29:52] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [09:29:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.38, 9.80, 9.27 [09:30:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.97, 8.40, 8.01 [09:31:24] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0cb20926bc32...73e7c630cb8a [09:31:25] [02miraheze/puppet] 07Reception123 0373e7c63 - rm img.shields.io from CSP whitelist (per T7899) [09:31:33] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:31:37] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:31:49] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:33:17] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:34:46] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [09:34:48] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 1.201 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [09:34:59] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [09:36:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:37:57] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:38:52] Reception123: what happened on task [09:39:00] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:39:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.89, 9.11, 9.94 [09:40:03] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.57, 8.12, 8.10 [09:41:16] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [09:42:04] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [09:42:15] RhinosF1: let me see if I can find anything useful, I've not been doing anything [09:42:27] except deleteBatch which got killed :( [09:42:47] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:43:02] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:45:49] RhinosF1: OOM at 9:34,9:36,9:37,9:40, 9:43 [09:46:03] and 22,29 [09:46:29] then my script was killed with [09:46:31] https://www.irccloud.com/pastebin/joHo1iye/ [09:47:11] and it's really slow now [09:47:42] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=mwtask111.miraheze.org&var-port=9100&viewPanel=281 yikes [09:48:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.00, 9.01, 8.57 [09:48:21] paladox: any idea how there could be such high load? ^ [09:48:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.48, 9.59, 9.71 [09:49:29] Nope [09:49:39] You’ll need to look at htop [09:49:39] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:49:47] Maybe reboot it [09:49:53] CPU is at 100% [09:50:21] Reception123: I do. The 2 jobs that got moved to mwtask111. I told you they were using 100% of resources on the MW servers. So I moved them to mwtask111, at least it doesn't cause production issues. [09:50:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:50:37] CosmicAlpha: oh [09:51:25] though it's so slow now [09:51:49] CPU was continually at 100% on MW servers. Even the WMF have a jobrunner unique to those 2 jobs. [09:52:34] They can run ffmpeg for hours. [09:52:55] CosmicAlpha: how did we handle that before with the old infra? [09:53:08] It was the cause of alot of the issues on the old infra. [09:53:55] CosmicAlpha: hmm, though why did it suddenly spike now? [09:54:08] after 9:30 UTC [09:54:30] Usage. What's htop saying? I could be wrong about the cause, but I do think so. [09:54:56] https://usercontent.irccloud-cdn.com/file/bdNBzBnG/image.png [09:55:17] CosmicAlpha: ^ [09:55:34] ffmpeg does seem to be a part of it but I don't get why it only started causing issues recently [09:56:13] Because if was just moved to high intensity = mwtask111 running it. [09:56:42] It was causing 100% CPU usage for hours on the MW servers before. [09:57:08] CosmicAlpha: that was yesterday though [09:57:34] We need another task server for jobs, ideally just for those 2 like the WMF. We just don't have the resources I don't think. Reception123: well like traffic, it spikes depending on usage. [09:57:53] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:57:57] hmm yeah, we need to figure something out [09:58:03] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:58:17] CosmicAlpha: what exactly is that job for? [09:58:36] Reception123: the absolute only option is disabling TimedMediaHandler. We can not move it back to production servers. That causes user issues. [09:58:39] TimedMediaHandler [09:59:20] We might be able to limit it and kill it sooner, but then it makes TMH near useless. [09:59:59] We just do not have the resources for it. I saw some docs somewhere that even said it requires very high resources and is recommended to place it on its on jobrunner. [10:00:05] CosmicAlpha: it'd be nice if we could somehow limit it and display some warning saying "Too many people are using this feature right now, please try later." [10:00:35] https://www.irccloud.com/pastebin/fs5Pk5R3/ [10:00:39] ^ CosmicAlpha can't we configure these? [10:01:15] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:01:16] Reception123: that's not possible. Not with jobs. And that's what I'm saying, the limits we'd need to set with that would make TMH useless. [10:01:27] oh [10:02:04] that's problematic. Ideally for me would be to only allow a set number of users to watch a video using TMH at once [10:02:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.53, 9.82, 9.92 [10:02:14] though I get that would be too complicated to do [10:04:44] PROBLEM - mwtask111 Disk Space on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:04:52] CosmicAlpha: wow the deleteBatch script is so slow now that it can barely even say which pages it needs to skip for being nonexistent [10:04:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.37, 7.85, 8.38 [10:05:12] Maybe.... $wgWaitTimeForTranscodeReset [10:05:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:06:20] But we should experiment with $wgEnableTranscode, and see how much functionality TimedMediaHandler is actually depend on transcoding. I assume quite a bit, but we really just don't have the kind of resources for it. [10:06:34] yeah, we need to try some experiments with those configs [10:06:45] as we just can't have 48 load [10:07:11] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [10:07:14] RECOVERY - mwtask111 ferm_active on mwtask111 is OK: OK ferm input default policy is set [10:07:16] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [10:07:20] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 2.859 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [10:07:57] RECOVERY - mwtask111 Disk Space on mwtask111 is OK: DISK OK - free space: / 55782 MB (84% inode=91%); [10:07:59] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [10:08:06] RECOVERY - mwtask111 NTP time on mwtask111 is OK: NTP OK: Offset -0.007327139378 secs [10:08:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:08:39] CosmicAlpha: I don't know what just happened but the script is working really fast now all of a sudden [10:08:43] maybe everyone stopped watching the videos? [10:09:04] Reception123: yeah but moving to task wasn't really avoidable after realising it was causing such issues on the MW servers. [10:09:06] htop shows everything decreased now [10:09:15] Probably the high usage one stopped. [10:09:34] It just takes one person, one job sometimes. [10:09:34] all of them stopped [10:09:58] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [10:10:00] yeah I think it was two people [10:10:03] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet last ran 1 hour ago [10:10:22] Reception123: maybe it was killed. [10:10:50] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [10:10:51] possibly yeah [10:11:04] or it was a single user who opened multiple tabs maybe and they closed them [10:11:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.89, 10.34, 9.95 [10:11:29] but yeah everything's back to normal now [10:12:15] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22202 bytes in 3.600 second response time [10:12:53] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:12:54] RECOVERY - mwtask111 HTTPS on mwtask111 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 553 bytes in 0.014 second response time [10:12:58] Reception123: Yeah... the WMF have a jobrunner 5 times the resources of mwtask111, thats only task is the transcoding jobs. [10:13:03] CosmicAlpha: but yeah, if those were taking place on mw* I can see why we were having performance issues [10:13:17] so either way it's much better we have it on mwtask111 only [10:13:47] Yes. That's why I moved it. I think it was the source of some issues on old infrastructure also. [10:14:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.23, 9.37, 9.68 [10:15:40] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteArchivedRevisions.php --wiki=polandballwiki --delete (END - exit=0) [10:15:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:17:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.20, 9.69, 9.73 [10:18:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:19:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.37, 9.20, 9.57 [10:20:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.24, 9.78, 9.12 [10:21:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:22:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.28, 8.99, 9.37 [10:22:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.85, 9.73, 9.20 [10:25:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.58, 8.69, 9.18 [10:31:02] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.98, 7.71, 8.41 [10:33:47] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.84, 7.37, 8.29 [10:36:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:46:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:51:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:53:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.42, 8.55, 8.44 [10:55:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.49, 8.01, 8.25 [11:01:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.76, 8.55, 8.23 [11:01:15] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.77, 0.44, 1.87 [11:03:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.88, 7.85, 8.01 [11:03:55] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.16, 0.34, 1.60 [11:09:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.56, 9.09, 8.71 [11:12:24] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.13, 7.80, 8.28 [11:15:44] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.51, 1.69, 1.54 [11:18:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.38, 9.11, 8.66 [11:19:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.59, 8.53, 8.01 [11:19:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:19:41] [02puppet] 07Universal-Omega reopened pull request 03#2351: ssl-acme: proxy through bastion - 13https://github.com/miraheze/puppet/pull/2351 [11:22:57] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.42, 11.04, 9.37 [11:24:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:24:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.88, 8.77, 8.68 [11:25:22] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.32, 8.45, 8.13 [11:25:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.50, 9.76, 9.14 [11:27:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:27:40] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/9e0c35907009...69eae6b3d48b [11:27:42] [02miraheze/mediawiki] 07Reception123 0369eae6b - Update ShortDescription ext. [11:27:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.59, 9.58, 8.98 [11:28:23] [02mediawiki] 07Reception123 closed pull request 03#4987: Bump extensions/Cargo from `b1d6c6e` to `e33a7fd` - 13https://github.com/miraheze/mediawiki/pull/4987 [11:28:24] [02mediawiki] 07Reception123 deleted branch 03dependabot/submodules/REL1_37/extensions/Cargo-e33a7fd - 13https://github.com/miraheze/mediawiki [11:28:26] [02miraheze/mediawiki] 07Reception123 deleted branch 03dependabot/submodules/REL1_37/extensions/Cargo-e33a7fd [11:28:27] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/69eae6b3d48b...a5eb611d7a72 [11:28:29] [02miraheze/mediawiki] 07dependabot[bot] 03a5eb611 - Bump extensions/Cargo from `b1d6c6e` to `e33a7fd` (#4987) [11:28:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.18, 10.29, 9.40 [11:29:34] [02mediawiki] 07Reception123 closed pull request 03#5006: Bump extensions/Translate from `266785e` to `12e795e` - 13https://github.com/miraheze/mediawiki/pull/5006 [11:29:36] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/a5eb611d7a72...e6b1209fb984 [11:29:37] [02miraheze/mediawiki] 07dependabot[bot] 03e6b1209 - Bump extensions/Translate from `266785e` to `12e795e` (#5006) [11:29:39] [02miraheze/mediawiki] 07Reception123 deleted branch 03dependabot/submodules/REL1_37/extensions/Translate-12e795e [11:29:40] [02mediawiki] 07Reception123 deleted branch 03dependabot/submodules/REL1_37/extensions/Translate-12e795e - 13https://github.com/miraheze/mediawiki [11:31:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.24, 9.18, 9.12 [11:32:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:32:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.99, 8.72, 8.43 [11:33:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.70, 9.13, 8.44 [11:34:07] [02mediawiki] 07Reception123 closed pull request 03#5002: Bump skins/Evelution from `833020a` to `d492062` - 13https://github.com/miraheze/mediawiki/pull/5002 [11:34:09] [02mediawiki] 07Reception123 deleted branch 03dependabot/submodules/REL1_37/skins/Evelution-d492062 - 13https://github.com/miraheze/mediawiki [11:34:10] [02miraheze/mediawiki] 07Reception123 deleted branch 03dependabot/submodules/REL1_37/skins/Evelution-d492062 [11:34:12] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/e6b1209fb984...7316faf58575 [11:34:13] [02miraheze/mediawiki] 07dependabot[bot] 037316faf - Bump skins/Evelution from `833020a` to `d492062` (#5002) [11:34:18] [02mediawiki] 07Reception123 closed pull request 03#4993: Bump extensions/CreateWiki from `5698d35` to `2b0f27d` - 13https://github.com/miraheze/mediawiki/pull/4993 [11:34:19] [02miraheze/mediawiki] 07Reception123 deleted branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-2b0f27d [11:34:21] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/7316faf58575...b974055ea353 [11:34:22] [02miraheze/mediawiki] 07dependabot[bot] 03b974055 - Bump extensions/CreateWiki from `5698d35` to `2b0f27d` (#4993) [11:34:24] [02mediawiki] 07Reception123 deleted branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-2b0f27d - 13https://github.com/miraheze/mediawiki [11:34:52] !log [reception@mwtask111] starting deploy of {'world': True, 'l10n': True} to scsvg [11:34:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:36:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.05, 7.84, 8.06 [11:36:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.95, 9.71, 9.54 [11:39:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:40:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.76, 7.75, 8.50 [11:41:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.05, 8.07, 8.39 [11:46:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.04, 8.58, 8.21 [11:47:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.94, 8.49, 8.62 [11:47:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.81, 9.76, 9.58 [11:49:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.73, 8.28, 8.17 [11:49:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:51:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.76, 9.64, 9.10 [11:54:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.08, 9.41, 9.17 [11:58:21] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:58:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:58:43] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:59:06] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:59:16] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:43] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:59:55] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:00:08] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:00:13] PROBLEM - mwtask111 ferm_active on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:00:16] PROBLEM - mwtask111 Disk Space on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:00:36] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:55] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:01:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.59, 7.96, 8.44 [12:01:05] PROBLEM - mwtask111 HTTPS on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:44] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [12:02:37] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [12:02:38] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [12:03:14] RECOVERY - mwtask111 NTP time on mwtask111 is OK: NTP OK: Offset 0.001060634851 secs [12:03:24] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [12:03:49] RECOVERY - mwtask111 Disk Space on mwtask111 is OK: DISK OK - free space: / 55756 MB (84% inode=91%); [12:04:35] RECOVERY - mwtask111 HTTPS on mwtask111 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 553 bytes in 3.134 second response time [12:11:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.83, 9.07, 9.80 [12:11:27] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:43] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:12:02] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.63, 8.83, 8.71 [12:12:20] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:12:42] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:12:48] PROBLEM - mwtask111 Disk Space on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:13:12] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [12:14:20] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 50 minutes ago with 0 failures [12:14:22] RECOVERY - mwtask111 ferm_active on mwtask111 is OK: OK ferm input default policy is set [12:14:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 9.95, 9.95 [12:14:46] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22202 bytes in 2.446 second response time [12:14:48] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [12:14:55] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 0.211 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [12:15:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.75, 8.86, 8.75 [12:15:32] !log [reception@mwtask111] finished deploy of {'world': True, 'l10n': True} to scsvg - FAIL: [0, 0, 0, 0, 0, 35072, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] in 2440s [12:15:33] RECOVERY - mwtask111 NTP time on mwtask111 is OK: NTP OK: Offset 0.001417189837 secs [12:15:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:15:48] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [12:15:48] !log [reception@mwtask111] starting deploy of {'world': True, 'l10n': True} to scsvg [12:15:49] RECOVERY - mwtask111 Disk Space on mwtask111 is OK: DISK OK - free space: / 55735 MB (84% inode=91%); [12:15:51] 2440s?? [12:15:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:17:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.40, 8.53, 8.00 [12:17:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.25, 9.62, 9.84 [12:17:47] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [12:17:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.71, 8.86, 8.50 [12:20:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.12, 8.11, 7.91 [12:20:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.41, 8.09, 8.29 [12:24:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.61, 9.40, 8.66 [12:27:01] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.96, 8.48, 8.44 [12:28:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:31:42] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.22, 7.66, 8.35 [12:33:30] !log [reception@mwtask111] finished deploy of {'world': True, 'l10n': True} to scsvg - SUCCESS in 1062s [12:33:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:34:21] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 9.39, 9.31 [12:36:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:37:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.75, 9.57, 9.43 [12:41:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:45:38] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.66, 9.64, 9.40 [12:46:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.86, 8.87, 8.43 [12:49:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.95, 10.01, 8.89 [12:50:15] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.14, 9.45, 8.64 [12:50:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:51:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.33, 9.43, 9.41 [12:53:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.63, 9.73, 8.89 [12:54:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.12, 9.37, 8.96 [12:56:48] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.07, 9.88, 9.67 [12:58:36] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.69, 7.90, 8.35 [12:59:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.53, 9.12, 9.43 [13:00:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:03:14] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.55, 7.72, 8.38 [13:05:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.35, 8.92, 9.19 [13:05:11] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.08, 0.12, 1.71 [13:08:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.67, 8.13, 8.82 [13:08:01] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.14, 0.12, 1.44 [13:08:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:09:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.58, 8.65, 8.38 [13:10:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.92, 9.44, 9.21 [13:12:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.53, 9.34, 8.68 [13:12:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.24, 8.41, 8.27 [13:15:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.48, 9.13, 8.74 [13:18:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.35, 9.12, 8.52 [13:18:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:23:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.97, 8.97, 8.68 [13:24:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:24:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.46, 8.91, 9.21 [13:26:35] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.21, 8.32, 8.45 [13:27:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.23, 9.55, 9.42 [13:32:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.96, 9.28, 8.98 [13:33:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.39, 8.70, 8.61 [13:35:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.03, 9.23, 9.03 [13:35:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.34, 8.62, 8.36 [13:36:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.91, 8.64, 8.13 [13:38:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.54, 7.59, 8.00 [13:38:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.63, 8.10, 8.46 [13:41:38] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.62, 7.97, 8.04 [13:45:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.53, 9.29, 8.95 [13:46:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.33, 8.93, 8.80 [13:47:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.59, 9.46, 9.04 [13:48:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.27, 8.85, 8.43 [13:49:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.13, 8.62, 8.70 [13:50:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.95, 9.76, 9.96 [13:50:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.82, 9.52, 9.20 [13:51:00] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.04, 8.41, 8.33 [13:52:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.11, 7.80, 8.39 [13:54:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:57:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.40, 8.72, 8.52 [13:58:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.90, 10.28, 10.03 [13:58:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.07, 9.39, 9.14 [14:00:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.16, 8.15, 8.32 [14:01:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.98, 9.44, 9.72 [14:04:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.01, 10.24, 9.93 [14:04:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.27, 8.95, 9.02 [14:04:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.39, 8.83, 8.43 [14:05:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:06:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.72, 9.48, 8.91 [14:07:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.18, 9.80, 9.86 [14:07:50] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.84, 8.16, 8.25 [14:10:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:12:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.67, 10.63, 10.18 [14:13:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.79, 9.49, 9.16 [14:13:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:14:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.97, 8.78, 8.49 [14:15:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.35, 10.30, 9.16 [14:15:23] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.97, 9.63, 9.08 [14:16:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.66, 9.06, 9.03 [14:17:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.30, 9.25, 8.36 [14:18:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.26, 9.70, 9.26 [14:20:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.24, 9.24, 8.52 [14:21:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.39, 9.70, 9.36 [14:22:10] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.14, 9.95, 9.44 [14:23:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.58, 10.25, 9.03 [14:24:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.04, 9.61, 9.36 [14:24:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.18, 9.83, 9.48 [14:25:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.59, 9.29, 9.26 [14:26:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.50, 8.12, 8.47 [14:27:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.82, 9.85, 9.53 [14:29:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.82, 9.64, 9.21 [14:30:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.67, 9.52, 9.42 [14:30:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.74, 10.02, 9.63 [14:33:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.08, 11.02, 10.01 [14:33:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.67, 8.29, 9.01 [14:33:54] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.36, 7.18, 8.25 [14:35:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.83, 7.56, 8.37 [14:40:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.33, 9.04, 8.70 [14:42:18] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.25, 9.66, 9.30 [14:42:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.46, 9.36, 8.94 [14:43:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.01, 9.55, 8.87 [14:46:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.46, 9.65, 9.11 [14:48:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.86, 9.91, 9.60 [14:49:52] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.15, 11.25, 9.77 [14:51:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.32, 7.89, 8.36 [14:52:41] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.03, 1.86, 1.35 [14:55:03] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.58, 7.84, 8.41 [14:55:33] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.27, 1.59, 1.33 [14:55:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.92, 9.16, 9.35 [14:58:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.92, 10.28, 9.72 [14:58:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:01:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.51, 9.40, 9.48 [15:03:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:04:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.77, 9.05, 8.69 [15:05:34] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.94, 8.10, 8.49 [15:11:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.85, 9.20, 9.83 [15:13:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:13:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.56, 8.28, 8.47 [15:17:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.75, 8.86, 9.93 [15:18:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.17, 8.93, 8.94 [15:19:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:20:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.05, 8.84, 8.66 [15:20:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.73, 10.81, 10.47 [15:21:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.67, 9.25, 9.10 [15:21:44] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.01, 9.52, 8.90 [15:23:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.92, 8.40, 8.52 [15:24:40] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.05, 8.50, 8.63 [15:26:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.30, 9.09, 8.74 [15:26:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.27, 9.14, 9.22 [15:27:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.36, 10.56, 9.63 [15:29:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.07, 9.29, 8.91 [15:29:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.88, 9.95, 9.56 [15:29:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.78, 3.58, 2.88 [15:30:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.73, 9.73, 9.45 [15:30:43] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.03, 10.63, 9.48 [15:31:33] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.56, 9.72, 8.73 [15:32:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.51, 2.98 [15:33:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.13, 9.39, 9.23 [15:34:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:34:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.57, 8.68, 8.50 [15:37:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.96, 8.22, 8.34 [15:38:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.04, 9.60, 9.34 [15:38:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.19, 7.83, 8.34 [15:38:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.55, 2.97, 2.91 [15:40:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:41:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.46, 9.07, 9.18 [15:45:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.49, 9.09, 8.89 [15:47:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.80, 9.92, 9.40 [15:53:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.84, 8.89, 8.54 [15:56:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.96, 9.81, 9.56 [15:56:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.87, 9.46, 9.41 [15:58:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.32, 9.81, 9.58 [15:58:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.47, 9.78, 9.32 [15:58:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.06, 9.17, 8.66 [15:59:10] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.83, 8.22, 8.37 [15:59:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.31, 10.05, 9.63 [16:01:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.99, 9.10, 9.36 [16:01:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.95, 9.56, 9.33 [16:05:10] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.17, 9.83, 9.83 [16:06:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.78, 9.51, 8.88 [16:07:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.05, 10.24, 9.66 [16:07:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.08, 9.77, 9.49 [16:08:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.40, 9.81, 9.81 [16:09:10] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.97, 9.68, 9.10 [16:10:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.15, 9.49, 9.43 [16:10:53] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.16, 9.78, 9.18 [16:13:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.85, 9.75, 9.62 [16:13:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.89, 9.73, 9.26 [16:13:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.91, 9.82, 9.85 [16:16:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.19, 10.57, 9.96 [16:16:40] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.30, 10.33, 10.01 [16:19:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.22, 9.33, 9.27 [16:22:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.51, 9.72, 9.46 [16:22:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.23, 9.49, 9.75 [16:25:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:26:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.23, 9.32, 9.07 [16:29:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.08, 9.03, 8.99 [16:31:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:33:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.61, 9.35, 9.88 [16:33:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.00, 10.63, 9.95 [16:34:08] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.27, 7.81, 8.37 [16:38:42] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.38, 10.13, 10.02 [16:41:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.65, 10.13, 9.44 [16:41:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:42:22] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.19, 7.78, 8.43 [16:44:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.57, 9.72, 9.42 [16:44:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:45:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.60, 9.59, 9.70 [16:48:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.71, 10.18, 9.92 [16:50:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.93, 9.86, 10.00 [16:53:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.54, 9.99, 10.02 [16:53:06] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.43, 10.26, 9.61 [16:54:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.57, 9.71, 9.87 [16:55:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.16, 9.33, 9.75 [16:56:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.90, 9.03, 9.23 [16:56:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.87, 9.23, 8.69 [16:59:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.79, 8.90, 8.62 [17:01:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.09, 9.53, 9.63 [17:04:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.95, 9.04, 9.41 [17:05:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.50, 7.60, 8.44 [17:08:09] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.75, 8.14, 8.43 [17:09:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.42, 3.74, 2.56 [17:10:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.62, 9.75, 9.57 [17:14:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.85, 9.16, 8.87 [17:17:53] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.76, 1.53, 1.92 [17:17:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.33, 8.59, 8.59 [17:23:47] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.68, 1.12, 1.62 [17:25:17] [02puppet] 07Universal-Omega synchronize pull request 03#2351: ssl-acme: proxy through bastion - 13https://github.com/miraheze/puppet/pull/2351 [17:26:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.96, 9.91, 9.26 [17:29:54] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.40, 9.50, 9.06 [17:30:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.73, 9.59, 9.05 [17:31:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.98, 8.58, 8.42 [17:32:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.12, 9.23, 9.15 [17:32:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.38, 9.56, 9.20 [17:33:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.85, 9.59, 9.13 [17:33:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 3.50, 3.09 [17:34:30] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.59, 8.17, 8.29 [17:35:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.47, 10.12, 9.50 [17:36:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.88, 2.76, 2.87 [17:36:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.01, 8.87, 9.67 [17:38:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.02, 8.95, 9.14 [17:41:46] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.42, 9.52, 8.73 [17:42:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.61, 9.95, 9.46 [17:44:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.30, 9.90, 9.45 [17:44:40] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.22, 9.09, 8.68 [17:45:10] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.22, 9.24, 9.28 [17:48:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.94, 9.94, 9.58 [17:50:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.23, 9.00, 9.28 [17:52:53] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.66, 10.05, 9.59 [17:56:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.80, 8.09, 8.44 [17:56:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.39, 9.45, 9.66 [17:56:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.39, 9.85, 9.46 [17:58:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.13, 9.73, 9.62 [17:59:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:59:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.34, 10.75, 10.10 [17:59:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.12, 9.56, 9.43 [18:01:28] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.61, 10.11, 9.75 [18:05:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.77, 9.28, 9.72 [18:05:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:07:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.12, 8.69, 9.33 [18:10:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:10:52] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.28, 10.07, 9.53 [18:11:13] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.53, 10.84, 10.17 [18:11:40] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.47, 10.03, 9.55 [18:13:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.31, 9.26, 9.27 [18:13:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.57, 9.88, 9.55 [18:13:54] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.66, 9.87, 8.97 [18:14:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:14:58] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.22, 2.17, 1.47 [18:15:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.21, 8.88, 9.13 [18:16:50] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.32, 10.52, 9.87 [18:16:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.56, 9.31, 8.90 [18:17:57] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.49, 1.37, 1.28 [18:19:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.82, 9.97, 9.80 [18:19:53] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.66, 9.59, 9.07 [18:20:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.75, 9.05, 9.41 [18:22:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.07, 10.31, 9.98 [18:22:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.38, 8.79, 8.87 [18:25:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.52, 9.68, 9.81 [18:25:51] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.93, 7.79, 8.49 [18:29:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:35:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.02, 7.46, 8.30 [18:36:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.33, 7.31, 8.25 [18:36:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:37:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.57, 9.21, 9.75 [18:40:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.06, 9.40, 9.72 [18:43:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.22, 9.14, 9.54 [18:43:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.30, 9.18, 9.18 [18:46:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.33, 9.64, 9.66 [18:46:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.17, 8.82, 9.09 [18:51:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.76, 9.37, 8.82 [18:52:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.75, 9.77, 8.92 [18:54:07] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.15, 8.33, 8.50 [18:55:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.26, 9.64, 9.94 [18:55:54] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.95, 7.34, 8.39 [18:59:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.44, 8.78, 8.35 [19:02:30] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.64, 8.02, 8.11 [19:03:41] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.53, 1.99, 1.45 [19:04:40] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.23, 9.48, 9.50 [19:09:21] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.23, 1.25, 1.36 [19:15:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.14, 9.00, 8.50 [19:16:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.78, 9.45, 9.27 [19:16:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.91, 9.93, 9.46 [19:16:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:17:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.31, 8.95, 8.55 [19:21:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:23:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.78, 9.36, 8.81 [19:24:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.06, 9.74, 9.58 [19:26:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.85, 8.83, 8.70 [19:26:46] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.36, 9.06, 8.71 [19:27:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.74, 9.26, 8.77 [19:27:18] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.00, 10.48, 9.91 [19:27:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.62, 9.57, 9.62 [19:29:40] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.47, 9.16, 8.84 [19:30:26] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.06, 10.01, 9.76 [19:33:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.95, 9.52, 9.70 [19:33:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.19, 9.76, 9.76 [19:36:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:38:29] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.62, 7.79, 8.32 [19:41:42] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.00, 9.78, 9.58 [19:41:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.29, 10.04, 9.16 [19:42:04] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.14, 8.97, 9.21 [19:42:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:44:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.06, 9.06, 9.34 [19:44:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.48, 9.49, 9.12 [19:46:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.05, 9.32, 9.03 [19:47:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.10, 9.94, 9.36 [19:49:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.77, 8.47, 8.75 [19:50:11] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.88, 9.33, 8.89 [19:50:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.12, 9.46, 9.46 [19:50:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.60, 8.99, 9.07 [19:52:33] [02puppet] 07Universal-Omega synchronize pull request 03#2373: Increase memcached memory to 15GB - 13https://github.com/miraheze/puppet/pull/2373 [19:52:46] [02puppet] 07Universal-Omega synchronize pull request 03#2373: Increase memcached memory to 15GB - 13https://github.com/miraheze/puppet/pull/2373 [19:52:57] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.99, 9.96, 9.54 [19:53:09] [02puppet] 07Universal-Omega edited pull request 03#2373: Increase memcached memory to 14.5GB - 13https://github.com/miraheze/puppet/pull/2373 [19:55:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.64, 9.02, 8.82 [19:55:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.69, 9.92, 9.64 [19:56:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.54, 1.20 [19:58:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.33, 8.84, 8.80 [19:58:45] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.16, 9.94, 9.67 [19:59:27] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.63, 1.16, 1.11 [19:59:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.08, 9.59, 9.30 [20:01:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.62, 9.52, 9.57 [20:02:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 9.39, 10.11, 9.78 [20:02:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.38, 9.12, 9.18 [20:04:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.85, 10.41, 9.38 [20:04:39] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.14, 7.57, 8.27 [20:04:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.26, 8.84, 9.37 [20:06:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/585d9c780de2 [20:06:04] [02miraheze/mw-config] 07Universal-Omega 03585d9c7 - Update GlobalExtensions.php [20:06:06] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config [20:06:14] [02mw-config] 07Universal-Omega opened pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [20:07:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/585d9c780de2...5816a2702ba6 [20:07:09] [02miraheze/mw-config] 07Universal-Omega 035816a27 - Update LocalSettings.php [20:07:10] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [20:07:21] miraheze/mw-config - Universal-Omega the build passed. [20:08:10] miraheze/mw-config - Universal-Omega the build passed. [20:08:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5816a2702ba6...53f949575d7e [20:08:34] [02miraheze/mw-config] 07Universal-Omega 0353f9495 - Update ManageWikiSettings.php [20:08:36] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [20:09:35] miraheze/mw-config - Universal-Omega the build passed. [20:09:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.31, 9.56, 9.30 [20:11:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.58, 9.65, 9.27 [20:11:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.24, 9.62, 8.86 [20:12:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.81, 10.32, 9.62 [20:15:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.42, 10.00, 9.63 [20:17:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.10, 8.99, 9.19 [20:17:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.36, 9.17, 8.96 [20:17:57] [02mw-config] 07RhinosF1 reviewed pull request 03#4424 commit - 13https://github.com/miraheze/mw-config/pull/4424#discussion_r789968402 [20:19:03] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.43, 10.61, 9.68 [20:21:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.50, 3.59, 3.17 [20:22:20] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.13, 9.26, 8.90 [20:24:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.22, 3.44, 3.18 [20:24:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.44, 9.98, 9.76 [20:25:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/53f949575d7e...972be6d7e372 [20:25:55] [02miraheze/mw-config] 07Universal-Omega 03972be6d - Update Defines.php [20:25:57] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [20:26:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.93, 9.43, 9.16 [20:27:10] miraheze/mw-config - Universal-Omega the build passed. [20:27:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.58, 9.55, 9.63 [20:28:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.12, 9.41, 9.12 [20:29:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.30, 9.34, 9.18 [20:30:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.06, 3.50, 3.22 [20:30:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/972be6d7e372...c067a7a82bbe [20:30:19] [02miraheze/mw-config] 07Universal-Omega 03c067a7a - Update LocalWiki.php [20:30:21] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [20:30:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.15, 9.81, 9.72 [20:31:08] [02mw-config] 07Universal-Omega reviewed pull request 03#4424 commit - 13https://github.com/miraheze/mw-config/pull/4424#discussion_r789975502 [20:31:31] RhinosF1: ^ [20:32:08] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.45, 7.86, 8.44 [20:32:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.99, 11.97, 10.22 [20:32:30] miraheze/mw-config - Universal-Omega the build passed. [20:33:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.49, 2.64, 2.94 [20:33:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.00, 9.01, 9.43 [20:36:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.44, 9.27, 9.76 [20:39:21] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.11, 8.85, 8.87 [20:44:52] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.62, 10.84, 9.37 [20:46:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.77, 9.65, 9.90 [20:47:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.43, 10.30, 9.84 [20:48:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.81, 9.74, 9.41 [20:48:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.17, 10.13, 9.89 [20:50:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.29, 9.23, 9.20 [20:53:02] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.61, 1.96, 1.37 [20:54:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.13, 9.92, 9.91 [20:56:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.04, 1.53, 1.31 [20:59:58] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.32, 9.69, 9.14 [21:00:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.59, 9.06, 9.05 [21:02:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.69, 9.35, 9.87 [21:02:59] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.37, 8.15, 8.72 [21:03:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.69, 8.40, 8.75 [21:03:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.46, 11.28, 10.19 [21:05:28] [02puppet] 07Universal-Omega synchronize pull request 03#2373: Increase memcached memory to 14.5GB - 13https://github.com/miraheze/puppet/pull/2373 [21:05:44] [02puppet] 07Universal-Omega synchronize pull request 03#2373: Increase memcached memory to 14.5GB - 13https://github.com/miraheze/puppet/pull/2373 [21:05:51] [02puppet] 07Universal-Omega edited pull request 03#2373: Increase memcached memory to 11GB - 13https://github.com/miraheze/puppet/pull/2373 [21:06:04] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.02, 7.70, 8.45 [21:06:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.88, 9.77, 9.79 [21:11:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.36, 9.86, 9.81 [21:14:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.56, 9.31, 9.60 [21:15:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.60, 9.26, 9.39 [21:17:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.66, 9.73, 9.69 [21:17:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.39, 9.57, 9.04 [21:18:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.06, 9.36, 9.42 [21:19:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.36, 9.44, 9.03 [21:20:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.76, 8.82, 8.84 [21:21:05] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.29, 9.62, 9.50 [21:24:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.92, 9.29, 9.39 [21:26:10] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.01, 9.38, 9.20 [21:29:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.26, 9.01, 9.09 [21:31:17] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.61, 10.31, 9.55 [21:32:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:33:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.85, 9.11, 9.12 [21:34:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.76, 9.65, 9.43 [21:36:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:37:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.10, 4.14, 3.49 [21:38:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.62, 7.69, 8.48 [21:39:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.01, 9.53, 9.42 [21:40:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.86, 3.50 [21:41:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.24, 1.98, 1.70 [21:43:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 4.06, 3.64 [21:43:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.36, 9.77, 9.98 [21:44:20] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.24, 2.28, 1.85 [21:45:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.07, 9.61, 9.09 [21:46:46] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.81, 1.71, 1.39 [21:48:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.34, 9.62, 9.46 [21:48:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.35, 9.37, 9.06 [21:49:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.86, 10.43, 10.15 [21:49:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.97, 3.75 [21:49:45] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.20, 1.45, 1.34 [21:51:25] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.35, 8.03, 8.43 [21:53:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.64, 3.28 [21:54:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.42, 9.68, 9.19 [21:54:56] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.05, 8.97, 8.89 [21:55:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.80, 4.42, 3.98 [21:56:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.20, 3.73, 3.36 [21:58:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.58, 8.64, 8.77 [21:58:42] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.22, 9.07, 8.80 [21:59:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.38, 3.27 [22:00:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.53, 3.51, 3.25 [22:00:50] [02puppet] 07Universal-Omega opened pull request 03#2375: MediaWiki: Add StructuredNavigation composer - 13https://github.com/miraheze/puppet/pull/2375 [22:01:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.22, 9.49, 9.09 [22:01:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.94, 8.78, 8.76 [22:01:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.83, 3.85 [22:02:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.90, 2.76, 3.06 [22:02:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.13, 1.65, 1.96 [22:03:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.68, 2.74, 3.00 [22:03:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.52, 9.66, 9.96 [22:03:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.57, 8.77, 8.89 [22:04:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.88, 8.58, 9.56 [22:04:37] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.60, 7.56, 8.31 [22:06:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.44, 9.95, 9.78 [22:06:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.03, 9.06, 9.55 [22:08:33] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.32, 1.30, 1.68 [22:09:16] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.05, 10.36, 10.15 [22:09:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.62, 11.75, 10.52 [22:09:47] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.66, 9.36, 9.01 [22:09:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.84, 9.01, 9.42 [22:10:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 3.49, 3.55 [22:12:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.14, 10.28, 9.85 [22:14:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.51, 9.23, 8.78 [22:16:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.69, 9.42, 9.97 [22:16:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.66, 8.98, 9.43 [22:18:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.21, 9.97, 9.09 [22:18:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.10, 9.20, 9.76 [22:19:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.24, 9.49, 9.54 [22:24:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.67, 9.53, 9.27 [22:24:24] [02mw-config] 07Universal-Omega closed pull request 03#4404: Set $wgMainStash for betaheze - 13https://github.com/miraheze/mw-config/pull/4404 [22:24:27] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7 - 13https://github.com/miraheze/mw-config [22:24:28] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7 [22:24:54] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.70, 9.53, 9.89 [22:30:59] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.91, 10.41, 10.06 [22:31:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.63, 3.32 [22:32:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.95, 3.87, 3.81 [22:33:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.46, 9.89, 9.70 [22:34:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.81, 3.45 [22:35:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.97, 4.60, 4.12 [22:35:40] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/d8c779b74bab [22:35:42] [02miraheze/ManageWiki] 07Universal-Omega 03d8c779b - Support namespaces without accompanying talk namespaces [22:35:43] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [22:35:45] [02ManageWiki] 07Universal-Omega opened pull request 03#325: Support namespaces without accompanying talk namespaces - 13https://github.com/miraheze/ManageWiki/pull/325 [22:35:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7930c44aa74d...4947b21f632a [22:35:55] [02miraheze/mw-config] 07Universal-Omega 034947b21 - Update ManageWikiExtensions.php [22:35:56] [02mw-config] 07Universal-Omega synchronize pull request 03#4402: Fix StructuredDiscussions' namespaces - 13https://github.com/miraheze/mw-config/pull/4402 [22:36:57] miraheze/mw-config - Universal-Omega the build passed. [22:36:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.84, 9.37, 9.58 [22:37:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.26, 9.77, 9.97 [22:38:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 3.84, 3.50 [22:38:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.49, 3.74, 3.50 [22:39:47] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.35, 7.82, 8.50 [22:40:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.14, 8.94, 9.81 [22:40:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.01, 9.52, 9.33 [22:41:06] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/d8c779b74bab...940b86ecddbd [22:41:08] [02miraheze/ManageWiki] 07Universal-Omega 03940b86e - Update ManageWikiFormFactoryBuilder.php [22:41:09] [02ManageWiki] 07Universal-Omega synchronize pull request 03#325: Support namespaces without accompanying talk namespaces - 13https://github.com/miraheze/ManageWiki/pull/325 [22:41:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.94, 3.08, 3.25 [22:42:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.42, 3.08, 3.65 [22:42:10] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.11, 2.61, 3.11 [22:43:18] miraheze/ManageWiki - Universal-Omega the build passed. [22:43:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.76, 10.20, 10.21 [22:46:17] miraheze/ManageWiki - Universal-Omega the build passed. [22:46:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.38, 9.47, 9.41 [22:47:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.14, 8.69, 8.69 [22:48:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.48, 3.70, 3.68 [22:49:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.70, 10.39, 9.87 [22:53:37] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.03, 9.53, 9.02 [22:57:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.74, 2.88, 3.27 [22:59:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.43, 9.09, 8.99 [23:00:01] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:02] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.48, 9.38, 9.82 [23:04:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.67, 9.26, 9.95 [23:04:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.95, 9.14, 9.72 [23:09:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.83, 8.02, 8.50 [23:10:07] [02ManageWiki] 07Universal-Omega closed pull request 03#325: Support namespaces without accompanying talk namespaces - 13https://github.com/miraheze/ManageWiki/pull/325 [23:10:08] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/be249c927463...33d0aa91145f [23:10:10] [02miraheze/ManageWiki] 07Universal-Omega 0333d0aa9 - Support namespaces without accompanying talk namespaces (#325) [23:10:11] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [23:10:13] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [23:10:41] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.88, 10.91, 10.27 [23:12:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.78, 10.05, 9.80 [23:13:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.30, 9.07, 9.35 [23:14:47] [02mw-config] 07Universal-Omega closed pull request 03#4402: Fix StructuredDiscussions' namespaces - 13https://github.com/miraheze/mw-config/pull/4402 [23:14:49] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [23:14:50] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3b74242661fa...dc9700932868 [23:14:52] [02miraheze/mw-config] 07Universal-Omega 03dc97009 - Fix StructuredDiscussions' namespaces (#4402) [23:14:53] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [23:15:05] miraheze/ManageWiki - Universal-Omega the build passed. [23:15:58] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.03, 9.78, 9.16 [23:16:01] miraheze/mw-config - Universal-Omega the build passed. [23:18:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.25, 9.36, 9.63 [23:18:09] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1087s [23:18:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.62, 8.66, 8.84 [23:20:46] !log [@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 1244s [23:21:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:20] !log [@mwtask111] starting deploy of {'config': True} to scsvg [23:22:33] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 12s [23:22:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.36, 9.49, 9.48 [23:28:04] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.51, 9.41, 9.02 [23:30:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.00, 9.77, 9.60 [23:31:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.84, 4.22, 3.47 [23:34:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.23, 3.33, 3.25 [23:36:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/dc9700932868...a60fd9101e02 [23:36:47] [02miraheze/mw-config] 07Universal-Omega 03a60fd91 - Re-enable ShortDescription [23:37:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.68, 9.01, 9.18 [23:37:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.22, 9.03, 9.99 [23:37:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.65, 8.62, 9.83 [23:37:45] miraheze/mw-config - Universal-Omega the build passed. [23:40:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.35, 9.27, 9.91 [23:45:10] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.41, 10.21, 9.80 [23:48:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.01, 9.56, 9.61 [23:51:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.41, 9.23, 9.55 [23:51:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.23, 9.13, 9.45 [23:51:55] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.76, 7.43, 8.33 [23:52:15] !log [@mwtask111] starting deploy of {'config': True} to scsvg [23:52:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:52:23] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 7s [23:52:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:54:42] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.05, 8.78, 9.10 [23:57:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.68, 9.51, 9.50 [23:57:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.38, 8.31, 8.84 [23:59:15] [02puppet] 07Universal-Omega opened pull request 03#2376: CSP: remove `mc-uc.netease.com` (per T7910) - 13https://github.com/miraheze/puppet/pull/2376