org.apache.hadoop.hbase.mapreduce

Class LoadIncrementalHFiles

java.lang.Object

Configured

org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles

@InterfaceAudience.Public
 @InterfaceStability.Stable
public class LoadIncrementalHFiles
extends Configured

Tool to load the output of HFileOutputFormat into an existing table.

See Also:

usage()

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LoadIncrementalHFiles.LoadQueueItem Represents an HFile waiting to be loaded.

Field Summary

Fields

Modifier and Type	Field and Description
static String	CREATE_TABLE_CONF_KEY
static String	MAX_FILES_PER_REGION_PER_FAMILY
static String	NAME

Constructor Summary

Constructors

Constructor and Description
LoadIncrementalHFiles(Configuration conf)

Method Summary

Modifier and Type	Method and Description
protected void	bulkLoadPhase(Table table, Connection conn, ExecutorService pool, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, <any> regionGroups) This takes the LQI's grouped by likely regions and attempts to bulk load them.
void	doBulkLoad(Path hfofDir, Admin admin, Table table, RegionLocator regionLocator) Perform a bulk load of the given directory into the given pre-existing table.
void	doBulkLoad(Path hfofDir, HTable table) Perform a bulk load of the given directory into the given pre-existing table.
protected List<LoadIncrementalHFiles.LoadQueueItem>	groupOrSplit(<any> regionGroups, LoadIncrementalHFiles.LoadQueueItem item, Table table, Pair<byte[][],byte[][]> startEndKeys) Attempt to assign the given load queue item into its target region group.
static byte[][]	inferBoundaries(TreeMap<byte[],Integer> bdryMap)
void	loadHFileQueue(Table table, Connection conn, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, Pair<byte[][],byte[][]> startEndKeys) Used by the replication sink to load the hfiles from the source cluster.
static void	main(String[] args)
void	prepareHFileQueue(Path hfofDir, Table table, Deque<LoadIncrementalHFiles.LoadQueueItem> queue) Prepare a collection of LoadIncrementalHFiles.LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.
int	run(String[] args)
void	setBulkToken(String stagingDir) Called from replication sink, where it manages bulkToken(staging directory) by itself.
protected List<LoadIncrementalHFiles.LoadQueueItem>	splitStoreFile(LoadIncrementalHFiles.LoadQueueItem item, Table table, byte[] startKey, byte[] splitKey)
protected List<LoadIncrementalHFiles.LoadQueueItem>	tryAtomicRegionLoad(Connection conn, TableName tableName, byte[] first, Collection<LoadIncrementalHFiles.LoadQueueItem> lqis) Attempts to do an atomic load of many hfiles into a region.
protected List<LoadIncrementalHFiles.LoadQueueItem>	tryAtomicRegionLoad(HConnection conn, byte[] tableName, byte[] first, Collection<LoadIncrementalHFiles.LoadQueueItem> lqis) Deprecated. As of release 0.96 (HBASE-9508). This will be removed in HBase 2.0.0. Use tryAtomicRegionLoad(Connection, TableName, byte[], Collection).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NAME

public static final String NAME

See Also:

Constant Field Values

MAX_FILES_PER_REGION_PER_FAMILY

public static final String MAX_FILES_PER_REGION_PER_FAMILY

See Also:

Constant Field Values

CREATE_TABLE_CONF_KEY

public static final String CREATE_TABLE_CONF_KEY

See Also:

Constant Field Values

Constructor Detail

LoadIncrementalHFiles

public LoadIncrementalHFiles(Configuration conf)
                      throws Exception

Throws:

Exception

Method Detail

doBulkLoad

public void doBulkLoad(Path hfofDir,
                       HTable table)
                throws TableNotFoundException,
                       IOException

Perform a bulk load of the given directory into the given pre-existing table. This method is not threadsafe.

Parameters:

hfofDir - the directory that was provided as the output path of a job using HFileOutputFormat

table - the table to load into

Throws:

TableNotFoundException - if table does not yet exist

IOException

doBulkLoad

public void doBulkLoad(Path hfofDir,
                       Admin admin,
                       Table table,
                       RegionLocator regionLocator)
                throws TableNotFoundException,
                       IOException

Perform a bulk load of the given directory into the given pre-existing table. This method is not threadsafe.

Parameters:

hfofDir - the directory that was provided as the output path of a job using HFileOutputFormat

table - the table to load into

Throws:

TableNotFoundException - if table does not yet exist

IOException

prepareHFileQueue

public void prepareHFileQueue(Path hfofDir,
                              Table table,
                              Deque<LoadIncrementalHFiles.LoadQueueItem> queue)
                       throws IOException

Prepare a collection of LoadIncrementalHFiles.LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.

Parameters:

hfilesDir - directory containing list of hfiles to be loaded into the table

table - table to which hfiles should be loaded

queue - queue which needs to be loaded into the table

Throws:

IOException - If any I/O or network error occurred

loadHFileQueue

public void loadHFileQueue(Table table,
                           Connection conn,
                           Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                           Pair<byte[][],byte[][]> startEndKeys)
                    throws IOException

Used by the replication sink to load the hfiles from the source cluster. It does the following, 1. groupOrSplitPhase(Table, ExecutorService, Deque, Pair) 2. LoadIncrementalHFiles#bulkLoadPhase(Table, Connection, ExecutorService, Deque, Multimap)

Parameters:

table - Table to which these hfiles should be loaded to

conn - Connection to use

queue - LoadIncrementalHFiles.LoadQueueItem has hfiles yet to be loaded

startEndKeys - starting and ending row keys of the region

Throws:

IOException

bulkLoadPhase

protected void bulkLoadPhase(Table table,
                             Connection conn,
                             ExecutorService pool,
                             Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                             <any> regionGroups)
                      throws IOException

This takes the LQI's grouped by likely regions and attempts to bulk load them. Any failures are re-queued for another pass with the groupOrSplitPhase.

Throws:

IOException

splitStoreFile

protected List<LoadIncrementalHFiles.LoadQueueItem> splitStoreFile(LoadIncrementalHFiles.LoadQueueItem item,
                                                                   Table table,
                                                                   byte[] startKey,
                                                                   byte[] splitKey)
                                                            throws IOException

Throws:

IOException

groupOrSplit

protected List<LoadIncrementalHFiles.LoadQueueItem> groupOrSplit(<any> regionGroups,
                                                                 LoadIncrementalHFiles.LoadQueueItem item,
                                                                 Table table,
                                                                 Pair<byte[][],byte[][]> startEndKeys)
                                                          throws IOException

Attempt to assign the given load queue item into its target region group. If the hfile boundary no longer fits into a region, physically splits the hfile such that the new bottom half will fit and returns the list of LQI's corresponding to the resultant hfiles. protected for testing

Throws:

IOException

tryAtomicRegionLoad

@Deprecated
protected List<LoadIncrementalHFiles.LoadQueueItem> tryAtomicRegionLoad(HConnection conn,
                                                                                    byte[] tableName,
                                                                                    byte[] first,
                                                                                    Collection<LoadIncrementalHFiles.LoadQueueItem> lqis)
                                                                             throws IOException

Deprecated. As of release 0.96 (HBASE-9508). This will be removed in HBase 2.0.0. Use tryAtomicRegionLoad(Connection, TableName, byte[], Collection).

Throws:

IOException

tryAtomicRegionLoad

protected List<LoadIncrementalHFiles.LoadQueueItem> tryAtomicRegionLoad(Connection conn,
                                                                        TableName tableName,
                                                                        byte[] first,
                                                                        Collection<LoadIncrementalHFiles.LoadQueueItem> lqis)
                                                                 throws IOException

Attempts to do an atomic load of many hfiles into a region. If it fails, it returns a list of hfiles that need to be retried. If it is successful it will return an empty list. NOTE: To maintain row atomicity guarantees, region server callable should succeed atomically and fails atomically. Protected for testing.

Returns:

empty list if success, list of items to retry on recoverable failure

Throws:

IOException

inferBoundaries

public static byte[][] inferBoundaries(TreeMap<byte[],Integer> bdryMap)

run

public int run(String[] args)
        throws Exception

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception

setBulkToken

public void setBulkToken(String stagingDir)

Called from replication sink, where it manages bulkToken(staging directory) by itself. This is used only when SecureBulkLoadEndpoint is configured in hbase.coprocessor.region.classes property. This directory is used as a temporary directory where all files are initially copied/moved from user given directory, set all the required file permissions and then from their it is finally loaded into a table. This should be set only when, one would like to manage the staging directory by itself. Otherwise this tool will handle this by itself.

Parameters:

stagingDir - staging directory path