IK
- input key typeIV
- input value typeMK
- map output key typeMV
- map output value typeOK
- output key typeOV
- output value typepublic class MapReduce<IK,IV,MK,MV,OK,OV> extends Object
Mapper
, Reducer
, and, optionally, the Combiner
interfaces. After implementing
the interfaces, provide them to an instance of this class.
The following sample demonstrates how to create and run a simple MapReduce:
new MapReduce(inputMap, outputMap, MyMapper.class, MyReducer.class)
.withCombiner(MyCombiner.class)
.run(TimeSpan.fromSeconds(10));
The run()
method is the starting point of the MapReduce invocation. There are two run method signatures,
one takes a pre-loaded InvocationGrid
and the other defines and loads an
invocation grid automatically. When an invocation is provided, it should include all of the necessary JARs and classes
that will be needed by the Mapper and Reducer, and if necessary the Combiner. The auto-constructed invocation grid
loads the classes specified as Mapper/Reducer/Combiner and all the classes for the keys and values associated with
the input map, output map, and mapper output. At the end of invocation, the created invocation grid is unloaded.
Manually loading an invocation grid and providing it to the run(com.scaleoutsoftware.soss.client.InvocationGrid, com.scaleoutsoftware.soss.client.TimeSpan)
method should be done in the following cases:
InvocationGridBuilder
's setJVMParameters(param) method can be usedConstructor and Description |
---|
MapReduce(com.scaleoutsoftware.soss.client.map.NamedMap<IK,IV> input,
com.scaleoutsoftware.soss.client.map.NamedMap<OK,OV> output,
Class<? extends Mapper<IK,IV,MK,MV>> mapper,
Class<? extends Reducer<MK,MV,OK,OV>> reducer)
Creates the MapReduce with required parameter.
|
Modifier and Type | Method and Description |
---|---|
void |
run(com.scaleoutsoftware.soss.client.InvocationGrid grid,
com.scaleoutsoftware.soss.client.TimeSpan timeout)
Executes MapReduce with provided invocation grid and timeout.
|
void |
run(com.scaleoutsoftware.soss.client.TimeSpan timeout)
Executes MapReduce.
|
MapReduce |
withCombiner(Class<? extends Combiner<MK,MV>> combiner)
Adds optional combiner to the MapReduce, which is used to compact
mapper output and avoid unnecessary data transfer.
|
MapReduce |
withIntermediateKeySerializer(com.scaleoutsoftware.soss.client.CustomSerializer<MK> keySerializer) |
MapReduce |
withIntermediateValueSerializer(com.scaleoutsoftware.soss.client.CustomSerializer<MV> valueSerializer) |
MapReduce |
withParameterObject(Object parameterObject)
Adds optional parameter object to the MapReduce job.
|
public MapReduce withCombiner(Class<? extends Combiner<MK,MV>> combiner)
combiner
- combiner class, implements Combiner
public MapReduce withParameterObject(Object parameterObject)
parameterObject
- public MapReduce withIntermediateKeySerializer(com.scaleoutsoftware.soss.client.CustomSerializer<MK> keySerializer)
public MapReduce withIntermediateValueSerializer(com.scaleoutsoftware.soss.client.CustomSerializer<MV> valueSerializer)
public void run(com.scaleoutsoftware.soss.client.TimeSpan timeout) throws IOException
timeout
- invocation timeoutIOException
- if MapReduce terminated abnormallypublic void run(com.scaleoutsoftware.soss.client.InvocationGrid grid, com.scaleoutsoftware.soss.client.TimeSpan timeout) throws IOException
gridBuilder.setLibraryPath("SossMapReduce")
method to do it.grid
- invocation grid to be used for MapReduce, it should include all necessary classestimeout
- invocation timeoutIOException
- if MapReduce terminated abnormallyCopyright (C) 2007-2015 ScaleOut Software, Inc.