org.apache.hadoop.hbase.test

Class IntegrationTestLoadAndVerify

java.lang.Object

org.apache.hadoop.hbase.util.AbstractHBaseTool

org.apache.hadoop.hbase.IntegrationTestBase

org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify

Direct Known Subclasses:

IntegrationTestWithCellVisibilityLoadAndVerify

public class IntegrationTestLoadAndVerify
extends IntegrationTestBase

A large test which loads a lot of data that has internal references, and verifies the data. In load step, 200 map tasks are launched, which in turn write loadmapper.num_to_write (default 100K) rows to an hbase table. Rows are written in blocks, for a total of 100 blocks. Each row in a block, contains loadmapper.backrefs (default 50) references to random rows in the prev block. Verify step is scans the table, and verifies that for every referenced row, the row is actually there (no data loss). Failed rows are output from reduce to be saved in the job output dir in hdfs and inspected later. This class can be run as a unit test, as an integration test, or from the command line Originally taken from Apache Bigtop.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	IntegrationTestLoadAndVerify.LoadMapper
static class	IntegrationTestLoadAndVerify.VerifyMapper
static class	IntegrationTestLoadAndVerify.VerifyReducer
static class	IntegrationTestLoadAndVerify.WALSearcher Tool to search missing rows in WALs and hfiles.

Field Summary

Fields inherited from class org.apache.hadoop.hbase.IntegrationTestBase

CHAOS_MONKEY_PROPS, monkey, MONKEY_LONG_OPT, monkeyProps, monkeyToUse, NO_CLUSTER_CLEANUP_LONG_OPT, noClusterCleanUp, util

Fields inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

cmdLineArgs, conf, EXIT_FAILURE, EXIT_SUCCESS

Constructor Summary

Constructors

Constructor and Description
IntegrationTestLoadAndVerify()

Method Summary

Modifier and Type	Method and Description
void	cleanUpCluster()
protected Job	doLoad(Configuration conf, HTableDescriptor htd)
protected void	doVerify(Configuration conf, HTableDescriptor htd)
protected java.util.Set<java.lang.String>	getColumnFamilies()
TableName	getTablename()
Path	getTestDir(java.lang.String testName, java.lang.String subdir)
static void	main(java.lang.String[] argv)
protected void	processOptions(CommandLine cmd) This method is called to process the options after they have been parsed.
int	runTestFromCommandLine()
protected void	setMapperClass(Job job)
void	setUpCluster()
static long	swapLong(long value) Converts a "long" value between endian systems.
void	testLoadAndVerify()
void	usage()

Methods inherited from class org.apache.hadoop.hbase.IntegrationTestBase

addOptions, cleanUp, cleanUpMonkey, cleanUpMonkey, doWork, getConf, getDefaultMonkeyFactory, getTestingUtil, processBaseOptions, setUp, setUpMonkey, startMonkey

Methods inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

addOptNoArg, addOptNoArg, addOptWithArg, addOptWithArg, addRequiredOptWithArg, addRequiredOptWithArg, doStaticMain, parseArgs, parseInt, parseLong, printUsage, printUsage, run, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IntegrationTestLoadAndVerify

public IntegrationTestLoadAndVerify()

Method Detail

setUpCluster

public void setUpCluster()
                  throws java.lang.Exception

Specified by:

setUpCluster in class IntegrationTestBase

Throws:

java.lang.Exception

cleanUpCluster

public void cleanUpCluster()
                    throws java.lang.Exception

Overrides:

cleanUpCluster in class IntegrationTestBase

Throws:

java.lang.Exception

swapLong

public static long swapLong(long value)

Converts a "long" value between endian systems. Borrowed from Apache Commons IO

Parameters:

value - value to convert

Returns:

the converted value

doLoad

protected Job doLoad(Configuration conf,
                     HTableDescriptor htd)
              throws java.lang.Exception

Throws:

java.lang.Exception

setMapperClass

protected void setMapperClass(Job job)

doVerify

protected void doVerify(Configuration conf,
                        HTableDescriptor htd)
                 throws java.lang.Exception

Throws:

java.lang.Exception

getTestDir

public Path getTestDir(java.lang.String testName,
                       java.lang.String subdir)
                throws java.io.IOException

Throws:

java.io.IOException

testLoadAndVerify

public void testLoadAndVerify()
                       throws java.lang.Exception

Throws:

java.lang.Exception

usage

public void usage()

processOptions

protected void processOptions(CommandLine cmd)

Description copied from class: AbstractHBaseTool

This method is called to process the options after they have been parsed.

Overrides:

processOptions in class IntegrationTestBase

runTestFromCommandLine

public int runTestFromCommandLine()
                           throws java.lang.Exception

Specified by:

runTestFromCommandLine in class IntegrationTestBase

Throws:

java.lang.Exception

getTablename

public TableName getTablename()

Specified by:

getTablename in class IntegrationTestBase

getColumnFamilies

protected java.util.Set<java.lang.String> getColumnFamilies()

Specified by:

getColumnFamilies in class IntegrationTestBase

main

public static void main(java.lang.String[] argv)
                 throws java.lang.Exception

Throws:

java.lang.Exception