cgl.imr.script
Class FileCollector

java.lang.Object
  extended by cgl.imr.script.FileCollector

public class FileCollector
extends Object

A utility class to collect files (data files typically generated by MapReduce computations) distributed among the compute nods to a given directory. This class uses Twister's functionality of creating partition files to get the list of files from all compute nodes.


Field Summary
static String FILE_COPY_PROGRAM
           
static int NUM_FILE_COPY_THREADS
           
 
Constructor Summary
FileCollector()
           
 
Method Summary
 void close()
           
static void main(String[] args)
           
 void pollNodesAndCopyFile(String dir, String filter, String destDir)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

FILE_COPY_PROGRAM

public static String FILE_COPY_PROGRAM

NUM_FILE_COPY_THREADS

public static int NUM_FILE_COPY_THREADS
Constructor Detail

FileCollector

public FileCollector()
Method Detail

main

public static void main(String[] args)

close

public void close()

pollNodesAndCopyFile

public void pollNodesAndCopyFile(String dir,
                                 String filter,
                                 String destDir)


For comments and suggestions please send e-mail to