cgl.imr.client
Class TwisterDriver

java.lang.Object
  extended by cgl.imr.client.TwisterDriver
All Implemented Interfaces:
Subscribable, TwisterConstants

public class TwisterDriver
extends Object
implements Subscribable, TwisterConstants

Client side driver for the MapReduce computations. This is a very important class in Twister framework. Many extensions possible to add features such as fault tolerance etc..


Nested Class Summary
 
Nested classes/interfaces inherited from interface cgl.imr.base.TwisterConstants
TwisterConstants.JobState
 
Field Summary
protected  String combineTopic
           
protected  Combiner currentCombiner
           
protected  boolean isMonitoringMapReduce
           
protected  JobConf jobConf
           
protected  TwisterConstants.JobState jobState
           
protected  boolean mapConfigured
           
protected  ConcurrentHashMap<Integer,TaskAssignment> mapTasksMap
           
protected  TwisterMonitor monitor
           
protected  TwisterConfigurations mrConfig
           
protected  int numMapTasks
           
protected  int numNodes
           
protected  int numReduceTasks
           
protected  PubSubService pubSubService
           
protected  Random randomizer
           
protected  boolean reduceConfigured
           
protected  ConcurrentHashMap<Integer,TaskAssignment> reduceTasksMap
           
protected  String reduceTopic
           
protected  ConcurrentHashMap<String,WorkerResponse> responseMap
           
protected  String responseTopic
           
protected  ShutdownHook shutDownHook
           
protected  ConcurrentHashMap<Integer,TaskAssignment> terminateTasksMap
           
protected  org.safehaus.uuid.UUIDGenerator uuidGen
           
 
Fields inherited from interface cgl.imr.base.TwisterConstants
COMBINE_INPUT, COMBINE_TOPIC_BASE, DIR_LIST_REQ, DIR_LIST_RES, FAILED, FIXED_DATA_FILE, FIXED_DATA_MODEL, HEP_DATA_STRING, MAP_ITERATIONS_OVER, MAP_REDUCE_TOPIC_BASE, MAP_TASK, MAP_TASK_REQUEST, MAP_WORKER_STARTED, MAPPER_REQUEST, MONITOR_REQUEST, MONITOR_RESPONSE, MONITOR_SLEEP_TIME, NEW_JOB_REQUEST, NEW_JOB_RESPONSE, NUM_MAP_TASKS, NUM_REDUCE_TASKS, PARTITION_FILE_RESPONSE_TOPIC_BASE, PARTITION_FILE_SPLIT_PATTERN, PROPERTIES_FILE, REDUCE_INPUT, REDUCE_RESPONSE, REDUCE_TASK, REDUCE_TASK_REQUEST, REDUCE_TOPIC_BASE, REDUCE_WORKER_REQUEST, RESPONSE_TOPIC_BASE, SEND_RECV_MAX_SLEEP_TIME, SEND_RECV_SLEEP_TIME, SUCCESS, TASK_STATUS, WORKER_RESPONSE
 
Constructor Summary
TwisterDriver(JobConf jobConf)
          Constructor for the TwisterDriver.
 
Method Summary
 void close()
          Cleanup the job related states, including the broker connections.
 void configureCombiner(Combiner combiner)
           
 void configureMaps()
          Configure map tasks without any input.
 void configureMaps(String partitionFile)
           
 void configureMaps(Value[] values)
          Configure the map tasks using static values/configurations.
 void configureReduce(Value[] values)
          Allow users to configure reduce tasks with static configuration.
 Combiner getCurrentCombiner()
           
 TwisterMonitor getMonitor()
           
 void onEvent(byte[] message)
          All the incoming messages are received here first, and next this method handles the requests appropriately.
 TwisterMonitor runMapReduce()
          Starts the MapReduce without any variable data (Key, Value).
 TwisterMonitor runMapReduce(List<KeyValuePair> pairs)
           
 TwisterMonitor runMapReduceBCast(Value val)
          Starts the MapReduce with a single Value.
 void setMonitoringCompletes()
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

combineTopic

protected String combineTopic

currentCombiner

protected Combiner currentCombiner

isMonitoringMapReduce

protected boolean isMonitoringMapReduce

jobConf

protected JobConf jobConf

jobState

protected TwisterConstants.JobState jobState

mapConfigured

protected boolean mapConfigured

monitor

protected TwisterMonitor monitor

mrConfig

protected TwisterConfigurations mrConfig

numMapTasks

protected int numMapTasks

numNodes

protected int numNodes

numReduceTasks

protected int numReduceTasks

pubSubService

protected PubSubService pubSubService

randomizer

protected Random randomizer

reduceConfigured

protected boolean reduceConfigured

reduceTopic

protected String reduceTopic

responseTopic

protected String responseTopic

shutDownHook

protected ShutdownHook shutDownHook

reduceTasksMap

protected ConcurrentHashMap<Integer,TaskAssignment> reduceTasksMap

mapTasksMap

protected ConcurrentHashMap<Integer,TaskAssignment> mapTasksMap

responseMap

protected ConcurrentHashMap<String,WorkerResponse> responseMap

terminateTasksMap

protected ConcurrentHashMap<Integer,TaskAssignment> terminateTasksMap

uuidGen

protected org.safehaus.uuid.UUIDGenerator uuidGen
Constructor Detail

TwisterDriver

public TwisterDriver(JobConf jobConf)
              throws TwisterException
Constructor for the TwisterDriver. Takes the JobConf as the input and establishes a connection with the broker network. Then it proceeds to subscribe into the necessary topics and initialize various data structures necessary.

Parameters:
jobConf - - JobConf object relevant to this job.
Throws:
TwisterException
Method Detail

close

public void close()
Cleanup the job related states, including the broker connections.


configureCombiner

public void configureCombiner(Combiner combiner)

configureMaps

public void configureMaps()
                   throws TwisterException
Configure map tasks without any input. Can be used with typical map reduce computations which does not require the configure and use many times behavior as in iterative MapReduce computations.

Throws:
TwisterException

configureMaps

public void configureMaps(String partitionFile)
                   throws TwisterException
Throws:
TwisterException

configureMaps

public void configureMaps(Value[] values)
                   throws TwisterException
Configure the map tasks using static values/configurations. Here the Value[] can carry any list of values that are assigned to the map tasks equally (or nearly equally).

Parameters:
values - - Array of values.
Throws:
TwisterException

configureReduce

public void configureReduce(Value[] values)
                     throws TwisterException
Allow users to configure reduce tasks with static configuration. Here the Value[] can carry any list of values that are assigned to the reduce tasks equally (or nearly equally).

Parameters:
values - - Array of values.
Throws:
TwisterException

getCurrentCombiner

public Combiner getCurrentCombiner()
                            throws TwisterException
Throws:
TwisterException

getMonitor

public TwisterMonitor getMonitor()

onEvent

public void onEvent(byte[] message)
All the incoming messages are received here first, and next this method handles the requests appropriately.

Specified by:
onEvent in interface Subscribable

runMapReduce

public TwisterMonitor runMapReduce()
                            throws TwisterException
Starts the MapReduce without any variable data (Key, Value). Assumes that the map tasks are configured with data using the static data configuration step. Suitable for many map-only style operations.

Returns:
TwisterMonitor - A monitor that allows the user program to asynchronously wait on the progress of the MapReduce computation.
Throws:
TwisterException

runMapReduce

public TwisterMonitor runMapReduce(List<KeyValuePair> pairs)
                            throws TwisterException
Throws:
TwisterException

runMapReduceBCast

public TwisterMonitor runMapReduceBCast(Value val)
                                 throws TwisterException
Starts the MapReduce with a single Value. This a broad cast style operation. One value is sent to all the map tasks. The configureMap method can be used to configure the Map tasks with variable data. We found this pattern is also a common usage scenario in MapReduce.

Returns:
TwisterMonitor - A monitor that allows the user program to asynchronously wait on the progress of the MapReduce computation.
Throws:
TwisterException

setMonitoringCompletes

public void setMonitoringCompletes()


For comments and suggestions please send e-mail to