Skip to content

Commit 360f070

Browse files
committed
remove infrequent words
1 parent 21a8a14 commit 360f070

File tree

5 files changed

+1645
-0
lines changed

5 files changed

+1645
-0
lines changed

remove_infrequent_words11/log.log

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -616,3 +616,46 @@ java.net.BindException: Address already in use
616616
2014-07-23 06:04:41,818 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.scheduler.TaskSetManager - Serialized task 1.0:18 as 570211 bytes in 3 ms
617617
2014-07-23 06:04:41,818 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.scheduler.TaskSetManager - Starting task 1.0:19 as TID 100 on executor 3: dco-node123-mgt.dco.ethz.ch (PROCESS_LOCAL)
618618
2014-07-23 06:04:41,822 [spark-akka.actor.default-dispatcher-5] INFO org.apache.spark.scheduler.TaskSetManager - Serialized task 1.0:19 as 570211 bytes in 4 ms
619+
2014-07-23 06:06:34,611 [spark-akka.actor.default-dispatcher-17] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 5)
620+
2014-07-23 06:06:34,611 [Result resolver thread-1] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 86 in 112840 ms on dco-node134-mgt.dco.ethz.ch (progress: 1/20)
621+
2014-07-23 06:06:36,723 [spark-akka.actor.default-dispatcher-22] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 6)
622+
2014-07-23 06:06:36,723 [Result resolver thread-2] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 87 in 114950 ms on dco-node133-mgt.dco.ethz.ch (progress: 2/20)
623+
2014-07-23 06:06:50,000 [Result resolver thread-3] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 94 in 128201 ms on dco-node124-mgt.dco.ethz.ch (progress: 3/20)
624+
2014-07-23 06:06:50,000 [spark-akka.actor.default-dispatcher-18] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 13)
625+
2014-07-23 06:06:52,418 [Result resolver thread-0] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 95 in 130615 ms on dco-node123-mgt.dco.ethz.ch (progress: 4/20)
626+
2014-07-23 06:06:52,418 [spark-akka.actor.default-dispatcher-23] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 14)
627+
2014-07-23 06:07:01,009 [spark-akka.actor.default-dispatcher-18] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 19)
628+
2014-07-23 06:07:01,009 [Result resolver thread-1] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 100 in 139187 ms on dco-node123-mgt.dco.ethz.ch (progress: 5/20)
629+
2014-07-23 06:07:02,926 [spark-akka.actor.default-dispatcher-22] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 16)
630+
2014-07-23 06:07:02,926 [Result resolver thread-2] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 97 in 141116 ms on dco-node135-mgt.dco.ethz.ch (progress: 6/20)
631+
2014-07-23 06:07:06,087 [spark-akka.actor.default-dispatcher-19] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 7)
632+
2014-07-23 06:07:06,087 [Result resolver thread-3] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 88 in 144309 ms on dco-node132-mgt.dco.ethz.ch (progress: 7/20)
633+
2014-07-23 06:07:15,995 [Result resolver thread-0] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 96 in 154189 ms on dco-node130-mgt.dco.ethz.ch (progress: 8/20)
634+
2014-07-23 06:07:15,995 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 15)
635+
2014-07-23 06:07:16,553 [Result resolver thread-1] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 98 in 154739 ms on dco-node125-mgt.dco.ethz.ch (progress: 9/20)
636+
2014-07-23 06:07:16,553 [spark-akka.actor.default-dispatcher-15] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 17)
637+
2014-07-23 06:07:21,905 [spark-akka.actor.default-dispatcher-16] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 10)
638+
2014-07-23 06:07:21,905 [Result resolver thread-2] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 91 in 160116 ms on dco-node131-mgt.dco.ethz.ch (progress: 10/20)
639+
2014-07-23 06:07:22,054 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 9)
640+
2014-07-23 06:07:22,054 [Result resolver thread-3] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 90 in 160268 ms on dco-node136-mgt.dco.ethz.ch (progress: 11/20)
641+
2014-07-23 06:07:22,349 [spark-akka.actor.default-dispatcher-24] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 8)
642+
2014-07-23 06:07:22,349 [Result resolver thread-0] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 89 in 160568 ms on dco-node135-mgt.dco.ethz.ch (progress: 12/20)
643+
2014-07-23 06:07:26,182 [spark-akka.actor.default-dispatcher-4] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 12)
644+
2014-07-23 06:07:26,182 [Result resolver thread-1] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 93 in 164387 ms on dco-node125-mgt.dco.ethz.ch (progress: 13/20)
645+
2014-07-23 06:07:28,759 [spark-akka.actor.default-dispatcher-18] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 18)
646+
2014-07-23 06:07:28,759 [Result resolver thread-2] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 99 in 166942 ms on dco-node124-mgt.dco.ethz.ch (progress: 14/20)
647+
2014-07-23 06:07:33,495 [Result resolver thread-3] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 92 in 171703 ms on dco-node129-mgt.dco.ethz.ch (progress: 15/20)
648+
2014-07-23 06:07:33,495 [spark-akka.actor.default-dispatcher-18] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 11)
649+
2014-07-23 06:07:47,766 [Result resolver thread-0] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 85 in 185999 ms on dco-node127-mgt.dco.ethz.ch (progress: 16/20)
650+
2014-07-23 06:07:47,766 [spark-akka.actor.default-dispatcher-15] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 4)
651+
2014-07-23 06:08:26,672 [spark-akka.actor.default-dispatcher-25] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 3)
652+
2014-07-23 06:08:26,672 [Result resolver thread-1] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 84 in 224909 ms on dco-node126-mgt.dco.ethz.ch (progress: 17/20)
653+
2014-07-23 06:08:34,277 [spark-akka.actor.default-dispatcher-22] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 1)
654+
2014-07-23 06:08:34,277 [Result resolver thread-2] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 82 in 232524 ms on dco-node121-mgt.dco.ethz.ch (progress: 18/20)
655+
2014-07-23 06:08:38,519 [spark-akka.actor.default-dispatcher-22] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 2)
656+
2014-07-23 06:08:38,519 [Result resolver thread-3] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 83 in 236760 ms on dco-node128-mgt.dco.ethz.ch (progress: 19/20)
657+
2014-07-23 06:09:09,622 [Result resolver thread-0] INFO org.apache.spark.scheduler.TaskSetManager - Finished TID 81 in 267877 ms on dco-node122-mgt.dco.ethz.ch (progress: 20/20)
658+
2014-07-23 06:09:09,622 [spark-akka.actor.default-dispatcher-23] INFO org.apache.spark.scheduler.DAGScheduler - Completed ResultTask(1, 0)
659+
2014-07-23 06:09:09,622 [Result resolver thread-0] INFO org.apache.spark.scheduler.TaskSchedulerImpl - Removed TaskSet 1.0, whose tasks have all completed, from pool
660+
2014-07-23 06:09:09,622 [spark-akka.actor.default-dispatcher-23] INFO org.apache.spark.scheduler.DAGScheduler - Stage 1 (foreach at RemoveInfrequentWordsApp.scala:27) finished in 267.879 s
661+
2014-07-23 06:09:09,623 [main] INFO org.apache.spark.SparkContext - Job finished: foreach at RemoveInfrequentWordsApp.scala:27, took 268.391831737 s

0 commit comments

Comments
 (0)