[10:37:51] lunch [12:35:35] Hi everyone. @Trey314159, @gehel and @ebernhardson helped me with explaining some of the more advanced features of wikipedia search. I presented my little wiki search engine (uses solr) and it went really well. Wanted to thank everyone. [12:40:04] I'm going to try and get this to be a dense vector search next. I did get it to the point where I was doing some NLP-NER to extract locations, dates, organizations, and people. It worked pretty well. Next iteration is going to focus on scaling the indexing to speed it up - it took days to get through all the processing of the docs on an 8 core machine. After that, I might have some q's about how [12:40:04] you're doing your training models if that's OK. After scaling this, I want to use the dense vector search features instead of BM25 matching. [12:47:52] kristianrickert: congrats! glad to hear your presentation went well [12:51:17] kristianrickert: glad to hear as well! You're always welcomed to ask questions here, or to join one of our office hours: https://etherpad.wikimedia.org/p/Search_Platform_Office_Hours [13:11:44] o/ [13:49:40] kristianrickert: Good news on the presentation! Congrats! [13:57:32] I'll def save my q's for the office hours too. If I ever post it here when the office hours aren't going on, there's def no rush to answer. I have a ton of scaffolding coding to do first before I need to ask anymore q's. But I have ONE quick question: [14:01:03] there's a version ID with the article in the dump - right? I'm caching a lot of calculations. So when I parse a new dump, I want to make sure I'm purging/updating the calculated extractions right - it seems like if I take a doc id + version id would be a perfect key instead of calculating a hash. [15:01:58] ryankemper: retrospective: https://meet.google.com/eki-rafx-cxi [15:30:58] gehel no rush but if there are any docs on the team API thing, let us know! Sounds interesting [16:13:06] randomly interesting, enwiki_(content|general) is 764G of primary indices, once exported to hive the compressed parquet source is ~148G. But it's not 100% working yet, still needs some iteration [16:13:52] vs just under 100G for the dumps we already make [16:17:43] ebernhardson: related to this, was using ebernhardson.cirrus2hive_v3 and found that page_id seems null [16:18:11] Joseph suggested using avro for this kind of dataset btw [16:23:30] dcausse: yes i saw that yesterday but don't yet understand how :S i do something like `hit['_source']['page_id'] = hit['_id']; yield hit['_source']` so it shouldn't ever be null :( I have to probably work it out in the pyspark shell and figure out whats going wrong on a small sample [16:23:46] ok, switching to avro should be as simple as a couple lines in the create table statement [16:26:45] (and then any oddities of serialization, i had to recursively walk things and change [] into None to get parquet working. Who knows what oddities avro will enforce :) [16:28:00] upside is this is reasonably quick, it ran from 23:04 to 2:36, so only 3.5 hours to do a full dump using 72 parallel shard queries [16:40:16] can't seem to read the whole table "Container killed by YARN for exceeding memory limits" wondering if it's because of parquet [16:41:45] hmm, interesting. The partitions themselves should be pretty small, on average ~60M per file with a few ~250M files [16:43:24] if in python, i often extend spark.executor.memoryOverhead, sometimes just loading things into python blows out the memory limits. If everything is being done in java-side (spark functions only) then it's harder to say, in theory spark functions should all be limited to heap and OOM instead of getting killed by yarn [16:43:55] I'm using a python notebook indeed [16:44:10] wondering if I can tune that from the notebook directly [16:44:39] not sure, i always used a bare notebook and use `import findspark; findspark.init('/usr/lib/spark2')` and then use SparkSession.builder to set config [16:45:29] if spark is already initialized you can't change those config options anymore, while yarn can spin up different sized instances for a single application spark requires it all to be pre-defined and constant [16:50:21] ok trying with spark.executor.memory: 8g and spark.executor.memoryOverhead:4g (was spark.executor.memory:4g initially) [16:52:52] default memory overhead is pretty low, i think 240M [16:53:19] oh, actually it's max(memory*10%, 384M) [16:54:18] at one time i wrote up a thing to inject into the java side that repeated the kind of checks yarn did and reported it to the logs with details...but i don't remember when or where i did that :S [16:54:40] back when mjolnir was regularly blowing memory limits [17:04:19] meh still no luck... [17:04:32] using 22/10/13 16:56:53 WARN Utils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf. [17:04:36] [Stage 1:> (70 + 256) / 4570]22/10/13 16:57:17 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_59 ! [17:04:38] 22/10/13 16:57:17 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_155 ! [17:04:40] 22/10/13 16:57:17 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_201 ! [17:04:42] [Stage 1:> (72 + 256) / 4570]22/10/13 16:57:17 ERROR YarnScheduler: Lost executor 85 on an-worker1136.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:04:44] 22/10/13 16:57:17 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 85 for reason Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:04:46] 22/10/13 16:57:17 WARN TaskSetManager: Lost task 252.0 in stage 1.0 (TID 1258, an-worker1136.eqiad.wmnet, executor 85): ExecutorLostFailure (executor 85 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:04:48] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:04:50] 22/10/13 16:57:17 WARN TaskSetManager: Lost task 242.0 in stage 1.0 (TID 1039, an-worker1136.eqiad.wmnet, executor 85): ExecutorLostFailure (executor 85 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:04:52] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:04:54] 22/10/13 16:57:17 WARN TaskSetManager: Lost task 326.0 in stage 1.0 (TID 1277, an-worker1136.eqiad.wmnet, executor 85): ExecutorLostFailure (executor 85 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:04:56] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:04:58] 22/10/13 16:57:17 WARN TaskSetManager: Lost task 318.0 in stage 1.0 (TID 1271, an-worker1136.eqiad.wmnet, executor 85): ExecutorLostFailure (executor 85 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:05:00] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:05:02] [Stage 1:=> (106 + 252) / 4570]22/10/13 16:57:20 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_140 ! [17:05:04] 22/10/13 16:57:20 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_990 ! [17:05:06] 22/10/13 16:57:20 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_92 ! [17:05:08] 22/10/13 16:57:20 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_747 ! [17:05:10] 22/10/13 16:57:20 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_100 ! [17:05:12] 22/10/13 16:57:20 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_801 ! [17:05:14] 22/10/13 16:57:20 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_106 ! [17:05:16] 22/10/13 16:57:20 ERROR YarnScheduler: Lost executor 76 on an-worker1119.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:05:18] 22/10/13 16:57:20 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 76 for reason Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:05:20] 22/10/13 16:57:20 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 69 for reason Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:05:22] 22/10/13 16:57:20 WARN TaskSetManager: Lost task 854.0 in stage 1.0 (TID 1282, an-worker1119.eqiad.wmnet, executor 76): ExecutorLostFailure (executor 76 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:05:24] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:05:26] 22/10/13 16:57:20 WARN TaskSetManager: Lost task 1120.0 in stage 1.0 (TID 1339, an-worker1119.eqiad.wmnet, executor 76): ExecutorLostFailure (executor 76 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:05:28] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:05:30] 22/10/13 16:57:20 WARN TaskSetManager: Lost task 1079.0 in stage 1.0 (TID 1329, an-worker1119.eqiad.wmnet, executor 76): ExecutorLostFailure (executor 76 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:05:32] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:05:34] 22/10/13 16:57:20 WARN TaskSetManager: Lost task 907.0 in stage 1.0 (TID 1289, an-worker1119.eqiad.wmnet, executor 76): ExecutorLostFailure (executor 76 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:05:36] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:05:38] 22/10/13 16:57:20 ERROR YarnScheduler: Lost executor 69 on an-worker1131.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:05:40] 22/10/13 16:57:20 WARN TaskSetManager: Lost task 121.0 in stage 1.0 (TID 1031, an-worker1131.eqiad.wmnet, executor 69): ExecutorLostFailure (executor 69 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:05:42] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:05:44] 22/10/13 16:57:20 WARN TaskSetManager: Lost task 130.0 in stage 1.0 (TID 1273, an-worker1131.eqiad.wmnet, executor 69): ExecutorLostFailure (executor 69 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:05:46] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:05:48] 22/10/13 16:57:20 WARN TaskSetManager: Lost task 157.0 in stage 1.0 (TID 1299, an-worker1131.eqiad.wmnet, executor 69): ExecutorLostFailure (executor 69 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:05:50] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:05:52] 22/10/13 16:57:20 WARN TaskSetManager: Lost task 132.0 in stage 1.0 (TID 1286, an-worker1131.eqiad.wmnet, executor 69): ExecutorLostFailure (executor 69 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:05:54] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:05:56] 22/10/13 16:57:20 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_109 ! [17:05:58] 22/10/13 16:57:20 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_75 ! [17:06:00] 22/10/13 16:57:20 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_86 ! [17:06:02] [Stage 1:=> (112 + 244) / 4570]22/10/13 16:57:20 WARN TransportChannelHandler: Exception in connection from /10.64.36.7:59146 [17:06:04] java.io.IOException: Connection reset by peer [17:06:06] at sun.nio.ch.FileDispatcherImpl.read0(Native Method) [17:06:08] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) [17:06:10] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) [17:06:12] at sun.nio.ch.IOUtil.read(IOUtil.java:192) [17:06:14] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) [17:06:16] at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) [17:06:18] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) [17:06:20] at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343) [17:06:22] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) [17:06:24] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [17:06:26] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [17:06:28] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [17:06:30] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [17:06:32] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [17:06:34] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) [17:06:36] at java.lang.Thread.run(Thread.java:750) [17:06:38] [Stage 1:=> (169 + 248) / 4570]22/10/13 16:57:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_107 ! [17:06:40] 22/10/13 16:57:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_7 ! [17:06:42] 22/10/13 16:57:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_111 ! [17:06:44] 22/10/13 16:57:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_4 ! [17:06:46] 22/10/13 16:57:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_6 ! [17:06:46] huh [17:06:48] 22/10/13 16:57:23 ERROR YarnScheduler: Lost executor 108 on an-worker1132.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:06:50] 22/10/13 16:57:23 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 108 for reason Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:06:52] 22/10/13 16:57:23 WARN TaskSetManager: Lost task 209.0 in stage 1.0 (TID 1297, an-worker1132.eqiad.wmnet, executor 108): ExecutorLostFailure (executor 108 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:06:54] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:06:56] 22/10/13 16:57:23 WARN TaskSetManager: Lost task 130.1 in stage 1.0 (TID 1399, an-worker1132.eqiad.wmnet, executor 108): ExecutorLostFailure (executor 108 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:06:58] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:07:00] 22/10/13 16:57:23 WARN TaskSetManager: Lost task 234.0 in stage 1.0 (TID 1313, an-worker1132.eqiad.wmnet, executor 108): ExecutorLostFailure (executor 108 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:07:02] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:07:04] 22/10/13 16:57:23 WARN TaskSetManager: Lost task 211.0 in stage 1.0 (TID 1298, an-worker1132.eqiad.wmnet, executor 108): ExecutorLostFailure (executor 108 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:07:06] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:07:08] [Stage 1:==> (198 + 252) / 4570]22/10/13 16:57:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_793 ! [17:07:10] 22/10/13 16:57:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_250 ! [17:07:12] 22/10/13 16:57:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3 ! [17:07:14] 22/10/13 16:57:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_611 ! [17:07:16] 22/10/13 16:57:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_654 ! [17:07:18] 22/10/13 16:57:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_165 ! [17:07:20] 22/10/13 16:57:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_19 ! [17:07:22] 22/10/13 16:57:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_402 ! [17:07:24] 22/10/13 16:57:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_561 ! [17:07:26] 22/10/13 16:57:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_34 ! [17:07:28] 22/10/13 16:57:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_725 ! [17:07:30] 22/10/13 16:57:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_105 ! [17:07:32] 22/10/13 16:57:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_337 ! [17:07:34] 22/10/13 16:57:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_391 ! [17:07:36] 22/10/13 16:57:25 ERROR YarnScheduler: Lost executor 66 on an-worker1110.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:07:38] 22/10/13 16:57:25 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 66 for reason Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:07:40] 22/10/13 16:57:25 WARN TaskSetManager: Lost task 714.0 in stage 1.0 (TID 1345, an-worker1110.eqiad.wmnet, executor 66): ExecutorLostFailure (executor 66 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:07:42] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:07:44] 22/10/13 16:57:25 WARN TaskSetManager: Lost task 754.0 in stage 1.0 (TID 1374, an-worker1110.eqiad.wmnet, executor 66): ExecutorLostFailure (executor 66 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:07:46] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:07:48] 22/10/13 16:57:25 WARN TaskSetManager: Lost task 794.0 in stage 1.0 (TID 1424, an-worker1110.eqiad.wmnet, executor 66): ExecutorLostFailure (executor 66 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:07:50] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:07:52] 22/10/13 16:57:25 WARN TaskSetManager: Lost task 606.0 in stage 1.0 (TID 1316, an-worker1110.eqiad.wmnet, executor 66): ExecutorLostFailure (executor 66 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:07:54] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:07:56] [Stage 1:======> (579 + 252) / 4570]22/10/13 16:57:43 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_280 ! [17:07:58] 22/10/13 16:57:43 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_439 ! [17:08:00] 22/10/13 16:57:43 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_588 ! [17:08:02] 22/10/13 16:57:43 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_857 ! [17:08:04] 22/10/13 16:57:43 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_507 ! [17:08:06] 22/10/13 16:57:43 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_624 ! [17:08:08] 22/10/13 16:57:43 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_317 ! [17:08:10] 22/10/13 16:57:43 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_809 ! [17:08:11] paste of doom :) [17:08:12] 22/10/13 16:57:43 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_487 ! [17:08:14] 22/10/13 16:57:43 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_555 ! [17:08:16] 22/10/13 16:57:43 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_332 ! [17:08:18] 22/10/13 16:57:43 ERROR YarnScheduler: Lost executor 113 on analytics1071.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:08:20] 22/10/13 16:57:43 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 113 for reason Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:08:22] 22/10/13 16:57:43 WARN TaskSetManager: Lost task 960.0 in stage 1.0 (TID 1707, analytics1071.eqiad.wmnet, executor 113): ExecutorLostFailure (executor 113 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:08:24] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:08:26] 22/10/13 16:57:43 WARN TaskSetManager: Lost task 959.0 in stage 1.0 (TID 1700, analytics1071.eqiad.wmnet, executor 113): ExecutorLostFailure (executor 113 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:08:28] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:08:30] 22/10/13 16:57:43 WARN TaskSetManager: Lost task 860.0 in stage 1.0 (TID 1684, analytics1071.eqiad.wmnet, executor 113): ExecutorLostFailure (executor 113 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:08:32] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:08:34] 22/10/13 16:57:43 WARN TaskSetManager: Lost task 1083.0 in stage 1.0 (TID 1768, analytics1071.eqiad.wmnet, executor 113): ExecutorLostFailure (executor 113 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:08:36] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:08:38] [Stage 1:=======> (703 + 238) / 4570]22/10/13 16:57:49 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_594 ! [17:08:40] 22/10/13 16:57:49 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_493 ! [17:08:42] 22/10/13 16:57:49 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1057 ! [17:08:44] 22/10/13 16:57:49 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_683 ! [17:08:46] 22/10/13 16:57:49 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_948 ! [17:08:48] 22/10/13 16:57:49 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_148 ! [17:08:50] 22/10/13 16:57:49 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_150 ! [17:08:52] 22/10/13 16:57:49 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_596 ! [17:08:54] 22/10/13 16:57:49 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_769 ! [17:08:56] 22/10/13 16:57:49 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_151 ! [17:08:58] 22/10/13 16:57:49 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_110 ! [17:09:00] 22/10/13 16:57:49 ERROR YarnScheduler: Lost executor 99 on an-worker1129.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:09:02] 22/10/13 16:57:49 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 99 for reason Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:09:04] 22/10/13 16:57:49 WARN TaskSetManager: Lost task 1341.0 in stage 1.0 (TID 1871, an-worker1129.eqiad.wmnet, executor 99): ExecutorLostFailure (executor 99 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:09:06] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:09:08] 22/10/13 16:57:49 WARN TaskSetManager: Lost task 1241.0 in stage 1.0 (TID 1849, an-worker1129.eqiad.wmnet, executor 99): ExecutorLostFailure (executor 99 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:09:10] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:09:12] 22/10/13 16:57:49 WARN TaskSetManager: Lost task 1189.0 in stage 1.0 (TID 1777, an-worker1129.eqiad.wmnet, executor 99): ExecutorLostFailure (executor 99 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:09:15] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:09:16] 22/10/13 16:57:49 WARN TaskSetManager: Lost task 1203.0 in stage 1.0 (TID 1794, an-worker1129.eqiad.wmnet, executor 99): ExecutorLostFailure (executor 99 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:09:18] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:09:20] [Stage 1:========> (769 + 230) / 4570]22/10/13 16:57:52 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 98 for reason Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:09:22] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:09:24] 22/10/13 16:57:52 ERROR YarnScheduler: Lost executor 98 on an-worker1129.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:09:26] 22/10/13 16:57:52 WARN TaskSetManager: Lost task 1329.0 in stage 1.0 (TID 1859, an-worker1129.eqiad.wmnet, executor 98): ExecutorLostFailure (executor 98 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:09:28] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:09:30] 22/10/13 16:57:52 WARN TaskSetManager: Lost task 1189.1 in stage 1.0 (TID 1967, an-worker1129.eqiad.wmnet, executor 98): ExecutorLostFailure (executor 98 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:09:32] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:09:34] 22/10/13 16:57:52 WARN TaskSetManager: Lost task 1241.1 in stage 1.0 (TID 1972, an-worker1129.eqiad.wmnet, executor 98): ExecutorLostFailure (executor 98 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:09:36] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:09:38] 22/10/13 16:57:52 WARN TaskSetManager: Lost task 1370.0 in stage 1.0 (TID 1927, an-worker1129.eqiad.wmnet, executor 98): ExecutorLostFailure (executor 98 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:09:40] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:09:42] 22/10/13 16:57:52 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_188 ! [17:09:44] 22/10/13 16:57:52 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_612 ! [17:09:46] 22/10/13 16:57:52 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_186 ! [17:09:48] 22/10/13 16:57:52 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1076 ! [17:09:50] it's only listing 4k partitions, but the input should be more like 23-25k partitions, i wonder if spark is making some bad decisions about how to merge multiple partitions together and loading too many large rows at once [17:09:50] 22/10/13 16:57:52 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_291 ! [17:09:52] 22/10/13 16:57:52 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_343 ! [17:09:54] 22/10/13 16:57:52 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_369 ! [17:09:56] 22/10/13 16:57:52 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_709 ! [17:09:58] 22/10/13 16:57:52 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_648 ! [17:10:00] 22/10/13 16:57:52 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_803 ! [17:10:04] 22/10/13 16:57:52 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_794 ! [17:10:06] 22/10/13 16:57:52 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1190 ! [17:10:08] 22/10/13 16:57:52 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_187 ! [17:10:10] 22/10/13 16:57:52 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_205 ! [17:10:12] 22/10/13 16:57:52 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_397 ! [17:10:14] [Stage 1:========> (775 + 226) / 4570]22/10/13 16:57:52 WARN TransportChannelHandler: Exception in connection from /10.64.5.13:57874 [17:10:16] java.io.IOException: Connection reset by peer [17:10:18] at sun.nio.ch.FileDispatcherImpl.read0(Native Method) [17:10:20] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) [17:10:22] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) [17:10:24] at sun.nio.ch.IOUtil.read(IOUtil.java:192) [17:10:26] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) [17:10:28] at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) [17:10:30] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) [17:10:32] at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343) [17:10:34] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) [17:10:36] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [17:10:39] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [17:10:41] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [17:10:43] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [17:10:45] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [17:10:47] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) [17:10:49] at java.lang.Thread.run(Thread.java:750) [17:10:51] [Stage 1:==========> (955 + 215) / 4570]22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_718 ! [17:10:53] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_506 ! [17:10:55] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1111 ! [17:10:57] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_63 ! [17:10:59] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_642 ! [17:11:01] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_720 ! [17:11:03] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1471 ! [17:11:07] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_79 ! [17:11:10] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_241 ! [17:11:12] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1329 ! [17:11:14] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1144 ! [17:11:16] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_846 ! [17:11:18] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_542 ! [17:11:20] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1273 ! [17:11:22] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1418 ! [17:11:24] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_464 ! [17:11:26] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_355 ! [17:11:28] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_66 ! [17:11:30] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_444 ! [17:11:32] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_853 ! [17:11:34] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_67 ! [17:11:36] 22/10/13 16:58:01 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_678 ! [17:11:38] 22/10/13 16:58:01 ERROR YarnScheduler: Lost executor 100 on an-worker1129.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:11:40] 22/10/13 16:58:01 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 100 for reason Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:11:42] 22/10/13 16:58:01 WARN TaskSetManager: Lost task 1602.0 in stage 1.0 (TID 2182, an-worker1129.eqiad.wmnet, executor 100): ExecutorLostFailure (executor 100 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:11:44] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:11:46] 22/10/13 16:58:01 WARN TaskSetManager: Lost task 1379.0 in stage 1.0 (TID 1945, an-worker1129.eqiad.wmnet, executor 100): ExecutorLostFailure (executor 100 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:11:48] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:11:50] 22/10/13 16:58:01 WARN TaskSetManager: Lost task 1378.0 in stage 1.0 (TID 1936, an-worker1129.eqiad.wmnet, executor 100): ExecutorLostFailure (executor 100 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:11:52] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:11:54] 22/10/13 16:58:01 WARN TaskSetManager: Lost task 1571.0 in stage 1.0 (TID 2178, an-worker1129.eqiad.wmnet, executor 100): ExecutorLostFailure (executor 100 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:11:56] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:11:59] [Stage 1:=============> (1182 + 197) / 4570]22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1640 ! [17:12:01] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1251 ! [17:12:03] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2094 ! [17:12:05] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1631 ! [17:12:07] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3760 ! [17:12:11] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1711 ! [17:12:13] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2039 ! [17:12:15] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2675 ! [17:12:17] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_226 ! [17:12:19] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_58 ! [17:12:21] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2374 ! [17:12:23] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2108 ! [17:12:25] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_902 ! [17:12:27] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2541 ! [17:12:29] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2344 ! [17:12:31] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_997 ! [17:12:33] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3334 ! [17:12:35] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3263 ! [17:12:37] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1897 ! [17:12:39] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3626 ! [17:12:42] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2630 ! [17:12:44] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_949 ! [17:12:46] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2069 ! [17:12:48] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1039 ! [17:12:50] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1778 ! [17:12:52] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2503 ! [17:12:54] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_196 ! [17:12:56] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2641 ! [17:12:58] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1327 ! [17:13:00] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_335 ! [17:13:02] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2807 ! [17:13:04] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3239 ! [17:13:06] 22/10/13 16:58:15 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 74 for reason Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:13:08] 22/10/13 16:58:15 ERROR YarnScheduler: Lost executor 74 on an-worker1109.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:13:10] 22/10/13 16:58:15 WARN TaskSetManager: Lost task 1833.0 in stage 1.0 (TID 2326, an-worker1109.eqiad.wmnet, executor 74): ExecutorLostFailure (executor 74 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:13:14] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:13:16] 22/10/13 16:58:15 WARN TaskSetManager: Lost task 1902.0 in stage 1.0 (TID 2398, an-worker1109.eqiad.wmnet, executor 74): ExecutorLostFailure (executor 74 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:13:18] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:13:20] 22/10/13 16:58:15 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 68 for reason Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:13:22] 22/10/13 16:58:15 WARN TaskSetManager: Lost task 1767.0 in stage 1.0 (TID 2277, an-worker1109.eqiad.wmnet, executor 74): ExecutorLostFailure (executor 74 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:13:24] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:13:27] 22/10/13 16:58:15 WARN TaskSetManager: Lost task 1770.0 in stage 1.0 (TID 2297, an-worker1109.eqiad.wmnet, executor 74): ExecutorLostFailure (executor 74 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:13:29] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:13:31] 22/10/13 16:58:15 ERROR YarnScheduler: Lost executor 68 on an-worker1116.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:13:33] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1012 ! [17:13:35] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_692 ! [17:13:37] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1639 ! [17:13:39] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_827 ! [17:13:41] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1694 ! [17:13:43] 22/10/13 16:58:15 WARN TaskSetManager: Lost task 4181.0 in stage 1.0 (TID 2388, an-worker1116.eqiad.wmnet, executor 68): ExecutorLostFailure (executor 68 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:13:45] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:13:47] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1719 ! [17:13:49] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1266 ! [17:13:51] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1647 ! [17:13:53] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1732 ! [17:13:55] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_95 ! [17:13:57] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1232 ! [17:13:59] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_734 ! [17:14:01] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1662 ! [17:14:03] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_787 ! [17:14:05] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_813 ! [17:14:07] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_890 ! [17:14:10] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_957 ! [17:14:12] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_502 ! [17:14:14] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1636 ! [17:14:18] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_625 ! [17:14:20] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_950 ! [17:14:22] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1683 ! [17:14:24] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1530 ! [17:14:26] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1785 ! [17:14:28] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_722 ! [17:14:30] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_856 ! [17:14:32] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_873 ! [17:14:34] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_992 ! [17:14:36] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_589 ! [17:14:38] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1721 ! [17:14:40] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1492 ! [17:14:42] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1615 ! [17:14:44] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_198 ! [17:14:46] 22/10/13 16:58:15 WARN TaskSetManager: Lost task 3724.0 in stage 1.0 (TID 2259, an-worker1116.eqiad.wmnet, executor 68): ExecutorLostFailure (executor 68 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:14:48] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:14:50] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_843 ! [17:14:52] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1342 ! [17:14:54] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1608 ! [17:14:56] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1617 ! [17:14:58] 22/10/13 16:58:15 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1252 ! [17:15:00] [Stage 1:=============> (1188 + 192) / 4570]22/10/13 16:58:16 WARN TransportChannelHandler: Exception in connection from /10.64.36.141:55426 [17:15:02] java.io.IOException: Connection reset by peer [17:15:04] at sun.nio.ch.FileDispatcherImpl.read0(Native Method) [17:15:06] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) [17:15:08] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) [17:15:10] at sun.nio.ch.IOUtil.read(IOUtil.java:192) [17:15:13] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) [17:15:15] at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) [17:15:17] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) [17:15:21] at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343) [17:15:23] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) [17:15:25] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [17:15:27] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [17:15:29] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [17:15:31] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [17:15:33] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [17:15:35] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) [17:15:37] at java.lang.Thread.run(Thread.java:750) [17:15:39] [Stage 1:==============> (1304 + 193) / 4570]22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_133 ! [17:15:41] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_716 ! [17:15:43] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_595 ! [17:15:45] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_450 ! [17:15:47] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_529 ! [17:15:49] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_81 ! [17:15:51] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_209 ! [17:15:53] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_581 ! [17:15:55] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_222 ! [17:15:57] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_571 ! [17:15:59] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_835 ! [17:16:01] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_52 ! [17:16:04] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_424 ! [17:16:06] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_509 ! [17:16:08] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1902 ! [17:16:10] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_374 ! [17:16:12] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_514 ! [17:16:14] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_568 ! [17:16:16] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_395 ! [17:16:18] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_143 ! [17:16:20] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_134 ! [17:16:24] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_679 ! [17:16:26] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2 ! [17:16:28] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_586 ! [17:16:30] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_442 ! [17:16:32] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1083 ! [17:16:34] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_57 ! [17:16:36] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_287 ! [17:16:38] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_479 ! [17:16:40] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_592 ! [17:16:42] 22/10/13 16:58:23 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_750 ! [17:16:45] 22/10/13 16:58:23 ERROR YarnScheduler: Lost executor 131 on an-worker1145.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:16:47] 22/10/13 16:58:23 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 131 for reason Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:16:49] 22/10/13 16:58:23 WARN TaskSetManager: Lost task 942.0 in stage 1.0 (TID 2526, an-worker1145.eqiad.wmnet, executor 131): ExecutorLostFailure (executor 131 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:16:51] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:16:53] 22/10/13 16:58:23 WARN TaskSetManager: Lost task 762.0 in stage 1.0 (TID 2424, an-worker1145.eqiad.wmnet, executor 131): ExecutorLostFailure (executor 131 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:16:55] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:16:57] 22/10/13 16:58:23 WARN TaskSetManager: Lost task 839.0 in stage 1.0 (TID 2513, an-worker1145.eqiad.wmnet, executor 131): ExecutorLostFailure (executor 131 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:16:59] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:17:01] 22/10/13 16:58:23 WARN TaskSetManager: Lost task 784.0 in stage 1.0 (TID 2441, an-worker1145.eqiad.wmnet, executor 131): ExecutorLostFailure (executor 131 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:17:03] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:17:05] [Stage 1:==============> (1312 + 188) / 4570]22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_214 ! [17:17:07] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1755 ! [17:17:09] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_657 ! [17:17:12] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1402 ! [17:17:13] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1435 ! [17:17:15] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_898 ! [17:17:17] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_836 ! [17:17:19] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_517 ! [17:17:21] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_41 ! [17:17:23] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_178 ! [17:17:28] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_781 ! [17:17:30] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_739 ! [17:17:32] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_267 ! [17:17:34] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_36 ! [17:17:36] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_407 ! [17:17:38] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_21 ! [17:17:40] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_804 ! [17:17:42] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_883 ! [17:17:44] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_855 ! [17:17:46] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1137 ! [17:17:48] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_786 ! [17:17:50] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_354 ! [17:17:52] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1559 ! [17:17:54] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1368 ! [17:17:56] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_973 ! [17:17:59] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_600 ! [17:18:01] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_891 ! [17:18:03] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_408 ! [17:18:05] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1214 ! [17:18:07] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1328 ! [17:18:09] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_260 ! [17:18:11] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1143 ! [17:18:13] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1626 ! [17:18:15] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_162 ! [17:18:17] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1248 ! [17:18:19] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_61 ! [17:18:21] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3381 ! [17:18:23] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3991 ! [17:18:25] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_512 ! [17:18:27] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2866 ! [17:18:31] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1166 ! [17:18:33] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1955 ! [17:18:35] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2367 ! [17:18:37] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1797 ! [17:18:39] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_816 ! [17:18:41] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1084 ! [17:18:43] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1160 ! [17:18:45] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3573 ! [17:18:47] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3588 ! [17:18:49] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3764 ! [17:18:51] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_744 ! [17:18:54] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_590 ! [17:18:56] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2077 ! [17:18:58] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3179 ! [17:19:00] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2137 ! [17:19:02] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_469 ! [17:19:04] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2189 ! [17:19:06] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3357 ! [17:19:08] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2255 ! [17:19:10] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2576 ! [17:19:12] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1441 ! [17:19:14] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_629 ! [17:19:16] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2395 ! [17:19:18] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_946 ! [17:19:20] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3023 ! [17:19:22] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3191 ! [17:19:24] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1998 ! [17:19:26] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1187 ! [17:19:28] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2606 ! [17:19:30] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1146 ! [17:19:35] 22/10/13 16:58:24 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1681 ! [17:19:37] 22/10/13 16:58:24 ERROR YarnScheduler: Lost executor 84 on an-worker1084.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:19:39] 22/10/13 16:58:24 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 84 for reason Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:19:41] 22/10/13 16:58:24 WARN TaskSetManager: Lost task 1773.0 in stage 1.0 (TID 2523, an-worker1084.eqiad.wmnet, executor 84): ExecutorLostFailure (executor 84 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:19:43] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:19:45] 22/10/13 16:58:24 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 70 for reason Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:19:47] 22/10/13 16:58:24 WARN TaskSetManager: Lost task 1762.0 in stage 1.0 (TID 2517, an-worker1084.eqiad.wmnet, executor 84): ExecutorLostFailure (executor 84 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:19:49] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:19:51] 22/10/13 16:58:24 WARN TaskSetManager: Lost task 1591.0 in stage 1.0 (TID 2331, an-worker1084.eqiad.wmnet, executor 84): ExecutorLostFailure (executor 84 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:19:53] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:19:55] 22/10/13 16:58:24 WARN TaskSetManager: Lost task 1638.0 in stage 1.0 (TID 2393, an-worker1084.eqiad.wmnet, executor 84): ExecutorLostFailure (executor 84 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:19:57] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:19:59] 22/10/13 16:58:24 ERROR YarnScheduler: Lost executor 70 on an-worker1116.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:20:01] 22/10/13 16:58:24 WARN TaskSetManager: Lost task 4181.1 in stage 1.0 (TID 2410, an-worker1116.eqiad.wmnet, executor 70): ExecutorLostFailure (executor 70 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:20:03] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:20:05] 22/10/13 16:58:24 WARN TaskSetManager: Lost task 3724.1 in stage 1.0 (TID 2409, an-worker1116.eqiad.wmnet, executor 70): ExecutorLostFailure (executor 70 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:20:07] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:20:09] [Stage 1:===============> (1394 + 192) / 4570]22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1810 ! [17:20:11] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1350 ! [17:20:13] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2538 ! [17:20:15] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1208 ! [17:20:17] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1917 ! [17:20:19] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2154 ! [17:20:21] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1746 ! [17:20:23] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1831 ! [17:20:25] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1971 ! [17:20:27] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1417 ! [17:20:29] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1298 ! [17:20:32] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2208 ! [17:20:34] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2065 ! [17:20:38] 22/10/13 16:58:31 ERROR YarnScheduler: Lost executor 135 on an-worker1147.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:20:40] 22/10/13 16:58:31 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 135 for reason Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:20:42] 22/10/13 16:58:31 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 103 for reason Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:20:44] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 2672.0 in stage 1.0 (TID 2555, an-worker1147.eqiad.wmnet, executor 135): ExecutorLostFailure (executor 135 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:20:46] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:20:48] 22/10/13 16:58:31 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 81 for reason Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:20:50] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 2712.0 in stage 1.0 (TID 2569, an-worker1147.eqiad.wmnet, executor 135): ExecutorLostFailure (executor 135 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:20:52] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:20:54] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 2872.0 in stage 1.0 (TID 2590, an-worker1147.eqiad.wmnet, executor 135): ExecutorLostFailure (executor 135 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:20:56] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:20:58] 22/10/13 16:58:31 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 73 for reason Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:21:00] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 2310.0 in stage 1.0 (TID 2473, an-worker1147.eqiad.wmnet, executor 135): ExecutorLostFailure (executor 135 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:21:02] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:21:04] 22/10/13 16:58:31 ERROR YarnScheduler: Lost executor 103 on an-worker1118.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:21:06] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 2242.0 in stage 1.0 (TID 2487, an-worker1118.eqiad.wmnet, executor 103): ExecutorLostFailure (executor 103 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:21:09] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:21:10] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 2244.0 in stage 1.0 (TID 2490, an-worker1118.eqiad.wmnet, executor 103): ExecutorLostFailure (executor 103 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:21:13] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:21:15] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 2358.0 in stage 1.0 (TID 2612, an-worker1118.eqiad.wmnet, executor 103): ExecutorLostFailure (executor 103 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:21:17] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:21:19] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 2341.0 in stage 1.0 (TID 2567, an-worker1118.eqiad.wmnet, executor 103): ExecutorLostFailure (executor 103 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:21:21] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:21:23] 22/10/13 16:58:31 ERROR YarnScheduler: Lost executor 81 on analytics1072.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:21:25] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_932 ! [17:21:27] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_331 ! [17:21:29] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 3806.0 in stage 1.0 (TID 2601, analytics1072.eqiad.wmnet, executor 81): ExecutorLostFailure (executor 81 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:21:31] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:21:33] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1712 ! [17:21:35] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1088 ! [17:21:37] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1525 ! [17:21:41] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1891 ! [17:21:43] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1245 ! [17:21:45] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1973 ! [17:21:47] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1576 ! [17:21:49] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_778 ! [17:21:51] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2214 ! [17:21:53] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2034 ! [17:21:55] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_65 ! [17:21:57] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_60 ! [17:21:59] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_46 ! [17:22:02] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1345 ! [17:22:04] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_454 ! [17:22:06] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_406 ! [17:22:08] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_998 ! [17:22:10] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_845 ! [17:22:12] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1165 ! [17:22:14] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_45 ! [17:22:16] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1209 ! [17:22:18] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 2762.0 in stage 1.0 (TID 2562, analytics1072.eqiad.wmnet, executor 81): ExecutorLostFailure (executor 81 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:22:20] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:22:22] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 2741.0 in stage 1.0 (TID 2552, analytics1072.eqiad.wmnet, executor 81): ExecutorLostFailure (executor 81 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:22:24] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:22:26] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 2855.0 in stage 1.0 (TID 2563, analytics1072.eqiad.wmnet, executor 81): ExecutorLostFailure (executor 81 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:22:28] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:22:30] 22/10/13 16:58:31 ERROR YarnScheduler: Lost executor 73 on an-worker1123.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:22:32] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1664 ! [17:22:34] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2476 ! [17:22:36] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2215 ! [17:22:38] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_441 ! [17:22:40] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 2179.0 in stage 1.0 (TID 2443, an-worker1123.eqiad.wmnet, executor 73): ExecutorLostFailure (executor 73 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:22:44] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:22:46] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_23 ! [17:22:48] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1109 ! [17:22:50] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2081 ! [17:22:52] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1876 ! [17:22:55] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2397 ! [17:22:57] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_935 ! [17:22:59] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 2209.0 in stage 1.0 (TID 2457, an-worker1123.eqiad.wmnet, executor 73): ExecutorLostFailure (executor 73 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:23:01] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:23:03] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1985 ! [17:23:05] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1987 ! [17:23:07] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2695 ! [17:23:09] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2075 ! [17:23:11] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_540 ! [17:23:13] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_263 ! [17:23:15] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2062 ! [17:23:17] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1550 ! [17:23:19] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_286 ! [17:23:21] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1472 ! [17:23:23] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1609 ! [17:23:25] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1259 ! [17:23:27] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_749 ! [17:23:29] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2082 ! [17:23:31] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2437 ! [17:23:33] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_792 ! [17:23:35] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1071 ! [17:23:37] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_748 ! [17:23:39] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2287 ! [17:23:41] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1482 ! [17:23:43] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_365 ! [17:23:47] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_814 ! [17:23:49] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_558 ! [17:23:51] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2191 ! [17:23:53] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1661 ! [17:23:55] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_219 ! [17:23:57] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_324 ! [17:24:00] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2205 ! [17:24:02] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 2282.0 in stage 1.0 (TID 2504, an-worker1123.eqiad.wmnet, executor 73): ExecutorLostFailure (executor 73 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:24:04] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:24:06] 22/10/13 16:58:31 WARN TaskSetManager: Lost task 2453.0 in stage 1.0 (TID 2587, an-worker1123.eqiad.wmnet, executor 73): ExecutorLostFailure (executor 73 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:24:08] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:24:10] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_503 ! [17:24:12] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_243 ! [17:24:14] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_281 ! [17:24:16] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1354 ! [17:24:18] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2099 ! [17:24:20] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_48 ! [17:24:22] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1856 ! [17:24:24] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_870 ! [17:24:26] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_177 ! [17:24:28] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_859 ! [17:24:30] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1358 ! [17:24:32] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_694 ! [17:24:34] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_736 ! [17:24:36] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1005 ! [17:24:38] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_33 ! [17:24:40] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1925 ! [17:24:42] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_938 ! [17:24:44] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1067 ! [17:24:46] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_755 ! [17:24:50] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_98 ! [17:24:52] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1341 ! [17:24:55] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_161 ! [17:24:57] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_982 ! [17:24:59] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_84 ! [17:25:01] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_826 ! [17:25:03] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1386 ! [17:25:05] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_90 ! [17:25:07] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1077 ! [17:25:09] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1338 ! [17:25:11] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_257 ! [17:25:13] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_358 ! [17:25:15] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1462 ! [17:25:17] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_309 ! [17:25:19] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_311 ! [17:25:22] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_728 ! [17:25:23] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_366 ! [17:25:26] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_834 ! [17:25:28] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_865 ! [17:25:30] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_135 ! [17:25:32] 22/10/13 16:58:31 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_405 ! [17:25:34] [Stage 1:===============> (1395 + 176) / 4570]22/10/13 16:58:31 WARN TransportChannelHandler: Exception in connection from /10.64.5.7:57842 [17:25:36] java.io.IOException: Connection reset by peer [17:25:38] at sun.nio.ch.FileDispatcherImpl.read0(Native Method) [17:25:40] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) [17:25:42] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) [17:25:44] at sun.nio.ch.IOUtil.read(IOUtil.java:192) [17:25:46] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) [17:25:48] at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) [17:25:50] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) [17:25:54] at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343) [17:25:56] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) [17:25:58] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [17:26:00] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [17:26:02] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [17:26:04] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [17:26:06] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [17:26:08] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) [17:26:10] at java.lang.Thread.run(Thread.java:750) [17:26:12] [Stage 1:===============> (1398 + 175) / 4570]22/10/13 16:58:31 WARN TransportChannelHandler: Exception in connection from /10.64.5.12:60572 [17:26:14] java.io.IOException: Connection reset by peer [17:26:16] at sun.nio.ch.FileDispatcherImpl.read0(Native Method) [17:26:18] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) [17:26:20] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) [17:26:22] at sun.nio.ch.IOUtil.read(IOUtil.java:192) [17:26:24] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) [17:26:26] at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) [17:26:28] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) [17:26:30] at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343) [17:26:33] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) [17:26:35] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [17:26:37] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [17:26:39] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [17:26:41] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [17:26:43] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [17:26:45] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) [17:26:47] at java.lang.Thread.run(Thread.java:750) [17:26:49] [Stage 1:================> (1471 + 167) / 4570]22/10/13 16:58:37 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 105 for reason Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:26:51] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:26:53] 22/10/13 16:58:37 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 138 for reason Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:26:57] 22/10/13 16:58:37 ERROR YarnScheduler: Lost executor 105 on an-worker1118.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:26:59] 22/10/13 16:58:37 WARN TaskSetManager: Lost task 2016.0 in stage 1.0 (TID 2392, an-worker1118.eqiad.wmnet, executor 105): ExecutorLostFailure (executor 105 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:27:01] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:27:03] 22/10/13 16:58:37 WARN TaskSetManager: Lost task 2326.0 in stage 1.0 (TID 2565, an-worker1118.eqiad.wmnet, executor 105): ExecutorLostFailure (executor 105 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:27:05] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:27:07] 22/10/13 16:58:37 WARN TaskSetManager: Lost task 2341.1 in stage 1.0 (TID 2635, an-worker1118.eqiad.wmnet, executor 105): ExecutorLostFailure (executor 105 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:27:09] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:27:11] 22/10/13 16:58:37 WARN TaskSetManager: Lost task 2403.0 in stage 1.0 (TID 2620, an-worker1118.eqiad.wmnet, executor 105): ExecutorLostFailure (executor 105 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.0 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:27:13] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:27:15] 22/10/13 16:58:37 ERROR YarnScheduler: Lost executor 138 on an-worker1143.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:27:17] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2223 ! [17:27:19] 22/10/13 16:58:37 WARN TaskSetManager: Lost task 320.0 in stage 1.0 (TID 2676, an-worker1143.eqiad.wmnet, executor 138): ExecutorLostFailure (executor 138 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:27:21] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:27:23] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1757 ! [17:27:25] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_203 ! [17:27:28] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1713 ! [17:27:30] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2031 ! [17:27:32] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_864 ! [17:27:34] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1873 ! [17:27:36] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1305 ! [17:27:38] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_246 ! [17:27:40] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1439 ! [17:27:42] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1300 ! [17:27:44] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_573 ! [17:27:46] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2016 ! [17:27:48] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_478 ! [17:27:50] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1013 ! [17:27:52] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1030 ! [17:27:54] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_776 ! [17:27:56] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_410 ! [17:28:00] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1568 ! [17:28:02] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_240 ! [17:28:04] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_396 ! [17:28:06] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1285 ! [17:28:08] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_248 ! [17:28:10] 22/10/13 16:58:37 WARN TaskSetManager: Lost task 302.0 in stage 1.0 (TID 2670, an-worker1143.eqiad.wmnet, executor 138): ExecutorLostFailure (executor 138 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:28:13] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:28:15] 22/10/13 16:58:37 WARN TaskSetManager: Lost task 359.0 in stage 1.0 (TID 2689, an-worker1143.eqiad.wmnet, executor 138): ExecutorLostFailure (executor 138 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:28:17] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:28:19] 22/10/13 16:58:37 WARN TaskSetManager: Lost task 29.0 in stage 1.0 (TID 2491, an-worker1143.eqiad.wmnet, executor 138): ExecutorLostFailure (executor 138 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:28:21] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:28:23] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_78 ! [17:28:25] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_149 ! [17:28:27] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_839 ! [17:28:29] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2179 ! [17:28:31] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_137 ! [17:28:33] 22/10/13 16:58:37 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_114 ! [17:28:35] [Stage 1:================> (1481 + 166) / 4570]22/10/13 16:58:38 WARN TransportChannelHandler: Exception in connection from /10.64.5.7:57852 [17:28:37] java.io.IOException: Connection reset by peer [17:28:39] at sun.nio.ch.FileDispatcherImpl.read0(Native Method) [17:28:41] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) [17:28:43] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) [17:28:45] at sun.nio.ch.IOUtil.read(IOUtil.java:192) [17:28:47] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) [17:28:49] at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) [17:28:51] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) [17:28:53] at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343) [17:28:55] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) [17:28:57] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [17:28:59] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [17:29:04] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [17:29:06] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [17:29:08] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [17:29:10] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) [17:29:12] at java.lang.Thread.run(Thread.java:750) [17:29:14] [Stage 1:=================> (1570 + 153) / 4570]22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_249 ! [17:29:16] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_314 ! [17:29:18] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_401 ! [17:29:20] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_939 ! [17:29:22] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_399 ! [17:29:24] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1313 ! [17:29:26] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_285 ! [17:29:28] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1197 ! [17:29:30] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1278 ! [17:29:32] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_901 ! [17:29:34] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1066 ! [17:29:36] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1179 ! [17:29:38] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_584 ! [17:29:40] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_802 ! [17:29:43] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_715 ! [17:29:44] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1073 ! [17:29:46] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_497 ! [17:29:48] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_537 ! [17:29:51] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_627 ! [17:29:53] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1253 ! [17:29:55] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1226 ! [17:29:57] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_446 ! [17:29:59] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_62 ! [17:30:01] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1021 ! [17:30:03] 22/10/13 16:58:45 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_925 ! [17:30:07] 22/10/13 16:58:45 ERROR YarnScheduler: Lost executor 133 on an-worker1088.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 9.2 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:30:09] 22/10/13 16:58:45 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 133 for reason Container killed by YARN for exceeding memory limits. 9.2 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:30:11] 22/10/13 16:58:45 WARN TaskSetManager: Lost task 1246.0 in stage 1.0 (TID 2643, an-worker1088.eqiad.wmnet, executor 133): ExecutorLostFailure (executor 133 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.2 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:30:13] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:30:15] 22/10/13 16:58:45 WARN TaskSetManager: Lost task 1470.0 in stage 1.0 (TID 2771, an-worker1088.eqiad.wmnet, executor 133): ExecutorLostFailure (executor 133 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.2 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:30:17] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:30:19] 22/10/13 16:58:45 WARN TaskSetManager: Lost task 1440.0 in stage 1.0 (TID 2747, an-worker1088.eqiad.wmnet, executor 133): ExecutorLostFailure (executor 133 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.2 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:30:21] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:30:23] 22/10/13 16:58:45 WARN TaskSetManager: Lost task 1408.0 in stage 1.0 (TID 2743, an-worker1088.eqiad.wmnet, executor 133): ExecutorLostFailure (executor 133 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.2 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:30:25] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:30:27] [Stage 1:==================> (1641 + 145) / 4570]22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_55 ! [17:30:29] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_31 ! [17:30:31] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_884 ! [17:30:33] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2050 ! [17:30:35] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_917 ! [17:30:37] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1038 ! [17:30:40] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1378 ! [17:30:42] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_779 ! [17:30:44] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1483 ! [17:30:46] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_887 ! [17:30:48] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1241 ! [17:30:50] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2238 ! [17:30:52] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_425 ! [17:30:54] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1534 ! [17:30:56] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1183 ! [17:30:58] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2008 ! [17:31:00] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2092 ! [17:31:02] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1669 ! [17:31:04] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1220 ! [17:31:06] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_47 ! [17:31:10] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1424 ! [17:31:12] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1823 ! [17:31:14] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2855 ! [17:31:16] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1947 ! [17:31:18] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_419 ! [17:31:20] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1154 ! [17:31:22] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1749 ! [17:31:25] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_56 ! [17:31:27] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1497 ! [17:31:29] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2256 ! [17:31:31] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1379 ! [17:31:32] the 30-minute paste :) [17:31:33] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1968 ! [17:31:35] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2058 ! [17:31:37] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_622 ! [17:31:39] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1501 ! [17:31:41] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_833 ! [17:31:43] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1571 ! [17:31:45] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2000 ! [17:31:47] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2200 ! [17:31:49] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2206 ! [17:31:51] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2234 ! [17:31:53] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1803 ! [17:31:55] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2134 ! [17:31:57] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1536 ! [17:31:59] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_472 ! [17:32:01] 22/10/13 16:58:51 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1356 ! [17:32:03] [Stage 1:==================> (1642 + 145) / 4570]22/10/13 16:58:51 ERROR YarnScheduler: Lost executor 97 on an-worker1129.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:32:05] 22/10/13 16:58:51 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 97 for reason Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:32:07] 22/10/13 16:58:51 WARN TaskSetManager: Lost task 2332.0 in stage 1.0 (TID 2804, an-worker1129.eqiad.wmnet, executor 97): ExecutorLostFailure (executor 97 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:32:09] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:32:13] 22/10/13 16:58:51 WARN TaskSetManager: Lost task 2357.0 in stage 1.0 (TID 2823, an-worker1129.eqiad.wmnet, executor 97): ExecutorLostFailure (executor 97 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:32:15] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:32:17] 22/10/13 16:58:51 WARN TaskSetManager: Lost task 2271.0 in stage 1.0 (TID 2781, an-worker1129.eqiad.wmnet, executor 97): ExecutorLostFailure (executor 97 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:32:19] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:32:21] 22/10/13 16:58:51 WARN TaskSetManager: Lost task 2408.0 in stage 1.0 (TID 2835, an-worker1129.eqiad.wmnet, executor 97): ExecutorLostFailure (executor 97 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:32:23] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:32:26] [Stage 1:==================> (1673 + 135) / 4570]22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_874 ! [17:32:27] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1446 ! [17:32:29] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1828 ! [17:32:32] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2227 ! [17:32:34] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1203 ! [17:32:36] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1189 ! [17:32:38] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2005 ! [17:32:40] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2235 ! [17:32:42] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_465 ! [17:32:44] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1361 ! [17:32:46] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1420 ! [17:32:48] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1767 ! [17:32:50] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2216 ! [17:32:52] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1602 ! [17:32:54] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_877 ! [17:32:56] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1161 ! [17:32:58] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1537 ! [17:33:00] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2027 ! [17:33:02] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_183 ! [17:33:04] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1981 ! [17:33:06] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_758 ! [17:33:08] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3806 ! [17:33:11] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2078 ! [17:33:13] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2289 ! [17:33:17] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_384 ! [17:33:19] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_767 ! [17:33:21] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_729 ! [17:33:21] Boy even *I* feel like I'm out of replicas at this point [17:33:23] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2259 ! [17:33:25] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_184 ! [17:33:27] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_182 ! [17:33:29] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1567 ! [17:33:31] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1079 ! [17:33:33] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1886 ! [17:33:35] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2115 ! [17:33:37] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1692 ! [17:33:39] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2038 ! [17:33:41] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1680 ! [17:33:43] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_583 ! [17:33:45] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_185 ! [17:33:47] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1878 ! [17:33:49] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_976 ! [17:33:52] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_361 ! [17:33:54] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1679 ! [17:33:56] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2712 ! [17:33:58] 22/10/13 16:58:54 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1606 ! [17:34:00] 22/10/13 16:58:54 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 101 for reason Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:34:02] 22/10/13 16:58:54 ERROR YarnScheduler: Lost executor 101 on an-worker1129.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:34:04] 22/10/13 16:58:54 WARN TaskSetManager: Lost task 2450.0 in stage 1.0 (TID 2852, an-worker1129.eqiad.wmnet, executor 101): ExecutorLostFailure (executor 101 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:34:06] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:34:08] 22/10/13 16:58:54 WARN TaskSetManager: Lost task 2271.1 in stage 1.0 (TID 2879, an-worker1129.eqiad.wmnet, executor 101): ExecutorLostFailure (executor 101 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:34:10] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:34:12] 22/10/13 16:58:54 WARN TaskSetManager: Lost task 2302.0 in stage 1.0 (TID 2801, an-worker1129.eqiad.wmnet, executor 101): ExecutorLostFailure (executor 101 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:34:14] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:34:16] ebernhardson: I'm a bit rusty on my IRC-foo, would +oping and kicking him and reinviting him maybe stop the torrent? :D [17:34:16] 22/10/13 16:58:54 WARN TaskSetManager: Lost task 2421.0 in stage 1.0 (TID 2836, an-worker1129.eqiad.wmnet, executor 101): ExecutorLostFailure (executor 101 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:34:20] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:34:22] [Stage 1:===================> (1737 + 138) / 4570]22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1393 ! [17:34:24] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2406 ! [17:34:26] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1156 ! [17:34:28] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_266 ! [17:34:30] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2053 ! [17:34:32] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2628 ! [17:34:34] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1735 ! [17:34:36] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_24 ! [17:34:38] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_37 ! [17:34:40] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2086 ! [17:34:42] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_14 ! [17:34:45] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_300 ! [17:34:47] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1898 ! [17:34:49] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1284 ! [17:34:51] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2088 ! [17:34:53] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2459 ! [17:34:55] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2533 ! [17:34:57] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_44 ! [17:34:59] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2654 ! [17:35:01] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2419 ! [17:35:03] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1776 ! [17:35:05] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1277 ! [17:35:07] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_668 ! [17:35:09] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_871 ! [17:35:11] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1768 ! [17:35:13] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2105 ! [17:35:15] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2562 ! [17:35:17] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_763 ! [17:35:19] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1042 ! [17:35:23] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_271 ! [17:35:25] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1033 ! [17:35:28] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2404 ! [17:35:30] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1404 ! [17:35:32] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_824 ! [17:35:34] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_274 ! [17:35:36] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1134 ! [17:35:38] 22/10/13 16:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_702 ! [17:35:40] 22/10/13 16:59:00 ERROR YarnScheduler: Lost executor 12 on an-worker1118.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:35:42] 22/10/13 16:59:00 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 12 for reason Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:35:44] 22/10/13 16:59:00 WARN TaskSetManager: Lost task 2760.0 in stage 1.0 (TID 2882, an-worker1118.eqiad.wmnet, executor 12): ExecutorLostFailure (executor 12 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:35:46] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:35:48] 22/10/13 16:59:00 WARN TaskSetManager: Lost task 2754.0 in stage 1.0 (TID 2880, an-worker1118.eqiad.wmnet, executor 12): ExecutorLostFailure (executor 12 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:35:50] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:35:52] 22/10/13 16:59:00 WARN TaskSetManager: Lost task 2779.0 in stage 1.0 (TID 2892, an-worker1118.eqiad.wmnet, executor 12): ExecutorLostFailure (executor 12 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:35:54] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:35:56] 22/10/13 16:59:00 WARN TaskSetManager: Lost task 2782.0 in stage 1.0 (TID 2901, an-worker1118.eqiad.wmnet, executor 12): ExecutorLostFailure (executor 12 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:35:58] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:36:00] [Stage 1:===================> (1764 + 133) / 4570]22/10/13 16:59:03 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 71 for reason Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:36:02] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:36:04] 22/10/13 16:59:03 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 106 for reason Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:36:06] 22/10/13 16:59:03 ERROR YarnScheduler: Lost executor 71 on an-worker1119.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:36:08] 22/10/13 16:59:03 WARN TaskSetManager: Lost task 3840.0 in stage 1.0 (TID 2939, an-worker1119.eqiad.wmnet, executor 71): ExecutorLostFailure (executor 71 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:36:10] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:36:12] 22/10/13 16:59:03 WARN TaskSetManager: Lost task 3834.0 in stage 1.0 (TID 2926, an-worker1119.eqiad.wmnet, executor 71): ExecutorLostFailure (executor 71 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:36:14] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:36:16] 22/10/13 16:59:03 WARN TaskSetManager: Lost task 3779.0 in stage 1.0 (TID 2904, an-worker1119.eqiad.wmnet, executor 71): ExecutorLostFailure (executor 71 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:36:19] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:36:21] 22/10/13 16:59:03 WARN TaskSetManager: Lost task 3705.0 in stage 1.0 (TID 2886, an-worker1119.eqiad.wmnet, executor 71): ExecutorLostFailure (executor 71 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:36:23] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:36:27] 22/10/13 16:59:03 ERROR YarnScheduler: Lost executor 106 on an-worker1118.eqiad.wmnet: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:36:29] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2766 ! [17:36:31] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1958 ! [17:36:33] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2617 ! [17:36:35] 22/10/13 16:59:03 WARN TaskSetManager: Lost task 2813.0 in stage 1.0 (TID 2916, an-worker1118.eqiad.wmnet, executor 106): ExecutorLostFailure (executor 106 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:36:37] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:36:39] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2791 ! [17:36:41] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_334 ! [17:36:43] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2263 ! [17:36:45] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3049 ! [17:36:47] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_854 ! [17:36:49] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1099 ! [17:36:51] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1935 ! [17:36:53] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1133 ! [17:36:55] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1820 ! [17:36:57] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_616 ! [17:36:59] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1538 ! [17:37:01] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1242 ! [17:37:03] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1592 ! [17:37:05] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2622 ! [17:37:07] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2270 ! [17:37:10] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_907 ! [17:37:12] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_831 ! [17:37:14] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1857 ! [17:37:16] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2162 ! [17:37:18] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3442 ! [17:37:20] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3101 ! [17:37:22] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_360 ! [17:37:24] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1225 ! [17:37:26] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3406 ! [17:37:30] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3400 ! [17:37:32] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2015 ! [17:37:34] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1849 ! [17:37:36] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3246 ! [17:37:38] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3200 ! [17:37:41] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2391 ! [17:37:43] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_578 ! [17:37:45] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2213 ! [17:37:47] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2222 ! [17:37:49] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2373 ! [17:37:51] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2002 ! [17:37:53] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2815 ! [17:37:55] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1892 ! [17:37:57] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3171 ! [17:37:59] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2993 ! [17:38:01] 22/10/13 16:59:03 WARN TaskSetManager: Lost task 2842.0 in stage 1.0 (TID 2937, an-worker1118.eqiad.wmnet, executor 106): ExecutorLostFailure (executor 106 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:38:03] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:38:05] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_703 ! [17:38:07] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1701 ! [17:38:09] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3410 ! [17:38:11] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_323 ! [17:38:13] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1055 ! [17:38:15] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1565 ! [17:38:17] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3619 ! [17:38:19] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1851 ! [17:38:21] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1503 ! [17:38:23] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1967 ! [17:38:25] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1837 ! [17:38:27] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3139 ! [17:38:29] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1660 ! [17:38:33] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2678 ! [17:38:36] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_3277 ! [17:38:38] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2707 ! [17:38:40] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_666 ! [17:38:42] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2350 ! [17:38:44] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_1230 ! [17:38:46] 22/10/13 16:59:03 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_8_2879 ! [17:38:48] 22/10/13 16:59:03 WARN TaskSetManager: Lost task 2761.0 in stage 1.0 (TID 2883, an-worker1118.eqiad.wmnet, executor 106): ExecutorLostFailure (executor 106 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:38:50] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:38:52] 22/10/13 16:59:03 WARN TaskSetManager: Lost task 2781.0 in stage 1.0 (TID 2895, an-worker1118.eqiad.wmnet, executor 106): ExecutorLostFailure (executor 106 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.1 GB of 8.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling [17:38:54] yarn.nodemanager.vmem-check-enabled because of YARN-4714. [17:38:58] maybe? [17:39:29] ebernhardson: at least dcausse's client was trying to stay under flood limits ;) [17:39:52] bd808: lol, yes :) [17:40:30] he didn't auto-rejoin though [17:43:10] sorry about that [17:43:16] happens :) [17:43:30] I mean for me it was pretty quick :P [17:44:01] as for ideas ...hmm. I would have to play with it :S I would suspect something related to either spark merging multiple partitions into one read partition, but the default is only 128M per partition. [17:44:13] it worked in the end [17:44:24] I mean my settings were not actually used [17:47:12] going offline, but now I know that copying in a jupyter code cell does not work the way you think with the xwin paste buffer... [17:47:21] :) see ya [17:47:56] i usually get a popup from my terminal "are you sure you want to paste 12345kB?" [17:50:00] ryankemper: in the future i think you can op/kick as well, everyone in the team should have op privileges. `/msg chanserv help op` `/msg chanserv op ebernhardson #wikimedia-search`, etc. [17:50:35] ebernhardson: thanks! [18:18:15] ryankemper: diner is running late, I'll be late for our pairing [18:19:14] gehel: ack [18:30:37] gehel ryankemper not feeling well, going to miss pairing [18:31:01] inflatador: get well! [18:31:05] inflatador: ack, feel better [18:59:18] hmm, search_after seems significantly slower for cirrus dumps on the mw side, in 10-03 arwiki took 58 minutes to dump, on 10-11 arwiki took 3.5 hours [18:59:30] i wonder if we did something different with batch sizes perhaps [19:02:24] should emit timestamps...i'm actually guessing based on the timestamps of the log files :P [19:05:53] cebwiki_content took 2 hours last time, looks like just over 4 hours this time [19:07:52] the size per request should be consistent and using $this->inputChunkSize, i suppose the per-request sort is going to be a little slower...also not sure whats an acceptable time to complete. meh [19:15:46] nothing looks to have failed so far, although its only up to commonswiki_file this week. Actually one thing failed, bclwikiquote_content, but that was because the index hadn't been created for a new wiki [20:20:11] * ebernhardson now realizes the whitelist didn't require a new build of the rdf repo....it's a textfile [20:20:22] moments after merging :P [20:21:36] i suppose it can't hurt, the only two patches being shipped are a cookie expire time extension for wcqs and a pom adjustment for moving scala deps into parent pom [22:07:05] certainly seeing spark have trouble reading this dataset :( something simple like spark.sample(False, 0.00001).take(10) OOM's with default settings [22:07:15] s/spark.sample/df.sample/ [22:23:46] meh, loked up my past memory tracking work for spark. It wasn't spark specific, rather it was an extra thread i spun up from inside xgboost4j-spark during training before handing off to the C library, so not generically applyable to get logs from a spark app