org.apache.hadoop.hbase.mapreduce

Class IntegrationTestBulkLoad

java.lang.Object

org.apache.hadoop.hbase.util.AbstractHBaseTool

org.apache.hadoop.hbase.IntegrationTestBase

org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad

public class IntegrationTestBulkLoad
extends IntegrationTestBase

Test Bulk Load and MR on a distributed cluster. It starts an MR job that creates linked chains The format of rows is like this: Row Key -> Long L:<< Chain Id >> -> Row Key of the next link in the chain S:<< Chain Id >> -> The step in the chain that his link is. D:<< Chain Id >> -> Random Data. All chains start on row 0. All rk's are > 0. After creating the linked lists they are walked over using a TableMapper based Mapreduce Job. There are a few options exposed: hbase.IntegrationTestBulkLoad.chainLength The number of rows that will be part of each and every chain. hbase.IntegrationTestBulkLoad.numMaps The number of mappers that will be run. Each mapper creates on linked list chain. hbase.IntegrationTestBulkLoad.numImportRounds How many jobs will be run to create linked lists. hbase.IntegrationTestBulkLoad.tableName The name of the table. hbase.IntegrationTestBulkLoad.replicaCount How many region replicas to configure for the table under test.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	IntegrationTestBulkLoad.CompositeKeyComparator Comparator used to order linkKeys so that they are passed to a reducer in order.
static class	IntegrationTestBulkLoad.EmptySplit
static class	IntegrationTestBulkLoad.FixedRecordReader<K,V>
static class	IntegrationTestBulkLoad.ITBulkLoadInputFormat
static class	IntegrationTestBulkLoad.LinkChain Writable used as the value emitted from a pass over the hbase table.
static class	IntegrationTestBulkLoad.LinkedListCheckingMapper Mapper to pass over the table.
static class	IntegrationTestBulkLoad.LinkedListCheckingReducer Class that does the actual checking of the links.
static class	IntegrationTestBulkLoad.LinkedListCreationMapper Mapper that creates a linked list of KeyValues.
static class	IntegrationTestBulkLoad.LinkKey Writable class used as the key to group links in the linked list.
static class	IntegrationTestBulkLoad.NaturalKeyGroupingComparator Comparator used to figure out if a linkKey should be grouped together.
static class	IntegrationTestBulkLoad.NaturalKeyPartitioner Class to figure out what partition to send a link in the chain to.
static class	IntegrationTestBulkLoad.SlowMeCoproScanOperations

Field Summary

Fields inherited from class org.apache.hadoop.hbase.IntegrationTestBase

CHAOS_MONKEY_PROPS, monkey, MONKEY_LONG_OPT, monkeyProps, monkeyToUse, NO_CLUSTER_CLEANUP_LONG_OPT, noClusterCleanUp, util

Fields inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

cmdLineArgs, conf, EXIT_FAILURE, EXIT_SUCCESS

Constructor Summary

Constructors

Constructor and Description
IntegrationTestBulkLoad()

Method Summary

Modifier and Type	Method and Description
protected void	addOptions() Override this to add command-line options using AbstractHBaseTool.addOptWithArg(java.lang.String, java.lang.String) and similar methods.
protected Set<String>	getColumnFamilies()
TableName	getTablename()
static TableName	getTableName(Configuration conf)
static void	main(String[] args)
protected void	processOptions(CommandLine cmd) This method is called to process the options after they have been parsed.
void	runLoad()
int	runTestFromCommandLine()
void	setUpCluster()
void	testBulkLoad()

Methods inherited from class org.apache.hadoop.hbase.IntegrationTestBase

cleanUp, cleanUpCluster, cleanUpMonkey, cleanUpMonkey, doWork, getConf, getDefaultMonkeyFactory, getTestingUtil, processBaseOptions, setUp, setUpMonkey, startMonkey

Methods inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

addOptNoArg, addOptNoArg, addOptWithArg, addOptWithArg, addRequiredOptWithArg, addRequiredOptWithArg, doStaticMain, parseArgs, parseInt, parseLong, printUsage, printUsage, run, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IntegrationTestBulkLoad

public IntegrationTestBulkLoad()

Method Detail

testBulkLoad

public void testBulkLoad()
                  throws Exception

Throws:

Exception

runLoad

public void runLoad()
             throws Exception

Throws:

Exception

setUpCluster

public void setUpCluster()
                  throws Exception

Specified by:

setUpCluster in class IntegrationTestBase

Throws:

Exception

addOptions

protected void addOptions()

Description copied from class: AbstractHBaseTool

Override this to add command-line options using AbstractHBaseTool.addOptWithArg(java.lang.String, java.lang.String) and similar methods.

Overrides:

addOptions in class IntegrationTestBase

processOptions

protected void processOptions(CommandLine cmd)

Description copied from class: AbstractHBaseTool

This method is called to process the options after they have been parsed.

Overrides:

processOptions in class IntegrationTestBase

runTestFromCommandLine

public int runTestFromCommandLine()
                           throws Exception

Specified by:

runTestFromCommandLine in class IntegrationTestBase

Throws:

Exception

getTablename

public TableName getTablename()

Specified by:

getTablename in class IntegrationTestBase

getTableName

public static TableName getTableName(Configuration conf)

getColumnFamilies

protected Set<String> getColumnFamilies()

Specified by:

getColumnFamilies in class IntegrationTestBase

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception