Package org.apache.hadoop.hbase.io.hfile

Provides implementations of HFile and HFile BlockCache.

See: Description

Interface Summary

Interface	Description
BlockCache	Block cache interface.
Cacheable	Cacheable is an interface that allows for an object to be cached.
CacheableDeserializer<T extends Cacheable>	Interface for a deserializer.
CachedBlock
HFile.CachingBlockReader	An abstraction used by the block index.
HFile.Reader	An interface used by clients to open and iterate an HFile.
HFile.Writer	API required to write an HFile
HFileBlock.BlockIterator	An interface allowing to iterate HFileBlocks.
HFileBlock.BlockWritable	Something that can be written into a block.
HFileBlock.FSReader	A full-fledged reader with iteration ability.
HFileDataBlockEncoder	Controls what kind of data block encoding is used.
HFileScanner	A scanner allows you to position yourself within a HFile and scan through it.
InlineBlockWriter	A way to write "inline" blocks into an HFile.
RandomDistribution.DiscreteRNG	Interface for discrete (integer) random distributions.
ResizableBlockCache	BlockCache which is resizable.

Class Summary

Class	Description
AbstractHFileReader	Common functionality needed by all versions of HFile readers.
AbstractHFileReader.Scanner
AbstractHFileWriter	Common functionality needed by all versions of HFile writers.
AgeSnapshot	Snapshot of block cache age in cache.
BlockCacheKey	Cache Key for use with implementations of BlockCache
BlockCacheUtil	Utilty for aggregating counts in CachedBlocks and toString/toJSON CachedBlocks and BlockCaches.
BlockCacheUtil.CachedBlocksByFile	Use one of these to keep a running account of cached blocks by file.
BlockWithScanInfo	BlockWithScanInfo is wrapper class for HFileBlock with other attributes.
CacheableDeserializerIdManager	This class is used to manage the identifiers for CacheableDeserializer
CacheConfig	Stores all of the cache objects and configuration for a single HFile.
CacheStats	Class that implements cache metrics.
CacheTestUtils
CacheTestUtils.ByteArrayCacheable
ChecksumUtil	Utility methods to compute and validate checksums.
CombinedBlockCache	CombinedBlockCache is an abstraction layer that combines LruBlockCache and BucketCache.
CombinedBlockCache.CombinedCacheStats
FixedFileTrailer	The HFile has a fixed trailer which contains offsets to other variable parts of the file.
HFile	File format for hbase.
HFile.FileInfo	Metadata for this file.
HFile.WriterFactory	This variety of ways to construct writers is used throughout the code, and we want to be able to swap writer implementations.
HFileBlock	Reading HFile version 1 and 2 blocks, and writing version 2 blocks.
HFileBlock.Writer	Unified version 2 HFile block writer.
HFileBlock.Writer.BufferGrabbingByteArrayOutputStream
HFileBlockIndex	Provides functionality to write (HFileBlockIndex.BlockIndexWriter) and read (HFileBlockIndex.BlockIndexReader) single-level and multi-level block indexes.
HFileBlockIndex.BlockIndexReader	The reader will always hold the root level index in the memory.
HFileBlockIndex.BlockIndexWriter	Writes the block index into the output stream.
HFileContext	This carries the information on some of the meta data about the HFile.
HFileContextBuilder	A builder that helps in building up the HFileContext
HFileDataBlockEncoderImpl	Do different kinds of data block encoding according to column family options.
HFilePrettyPrinter	Implements pretty-printing functionality for HFiles.
HFileReaderV2	HFile reader for version 2.
HFileReaderV2.AbstractScannerV2
HFileReaderV2.EncodedScannerV2	ScannerV2 that operates on encoded data blocks.
HFileReaderV2.ScannerV2	Implementation of HFileScanner interface.
HFileReaderV3	HFile reader for version 3.
HFileReaderV3.EncodedScannerV3	ScannerV3 that operates on encoded data blocks.
HFileReaderV3.ScannerV3	Implementation of HFileScanner interface.
HFileUtil
HFileWriterV2	Writes HFile format version 2.
HFileWriterV3	HFile writer for version 3.
InclusiveCombinedBlockCache
LruBlockCache	A block cache implementation that is memory-aware using HeapSize, memory-bound using an LRU eviction algorithm, and concurrent: backed by a ConcurrentHashMap and with a non-blocking eviction thread giving constant-time LruBlockCache.cacheBlock(org.apache.hadoop.hbase.io.hfile.BlockCacheKey, org.apache.hadoop.hbase.io.hfile.Cacheable, boolean, boolean) and LruBlockCache.getBlock(org.apache.hadoop.hbase.io.hfile.BlockCacheKey, boolean, boolean, boolean) operations.
LruCachedBlock	Represents an entry in the LruBlockCache.
LruCachedBlockQueue	A memory-bound queue that will grow until an element brings total size >= maxSize.
MemcachedBlockCache	Class to store blocks into memcached.
NanoTimer	A nano-second timer.
NoOpDataBlockEncoder	Does not perform any kind of encoding/decoding.
PrefetchExecutor
RandomDistribution	A class that generates random numbers that follow some distribution.
RandomDistribution.Binomial	Binomial distribution.
RandomDistribution.Flat	P(i)=1/(max-min)
RandomDistribution.Zipf	Zipf distribution.
TestBlockCacheReporting
TestCacheConfig	Tests that CacheConfig does as expected.
TestCachedBlockQueue
TestCacheOnWrite	Tests HFile cache-on-write functionality for the following block types: data blocks, non-root index blocks, and Bloom filter blocks.
TestChecksum
TestFixedFileTrailer
TestForceCacheImportantBlocks	Make sure we always cache important block types, such as index blocks, as long as we have a block cache, even though block caching might be disabled for the column family.
TestHFile	test hfile features.
TestHFileBlock
TestHFileBlockCompatibility	This class has unit tests to prove that older versions of HFiles (without checksums) are compatible with current readers.
TestHFileBlockCompatibility.Writer	This is the version of the HFileBlock.Writer that is used to create V2 blocks with minor version 0.
TestHFileBlockIndex
TestHFileBlockPositionalRead	Unit test suite covering HFileBlock positional read logic.
TestHFileDataBlockEncoder
TestHFileEncryption
TestHFileInlineToRootChunkConversion	Test a case when an inline index chunk is converted to a root one.
TestHFileSeek	test the performance for seek.
TestHFileWriterV2	Testing writing a version 2 HFile.
TestHFileWriterV3	Testing writing a version 3 HFile.
TestLazyDataBlockDecompression	A kind of integration test at the intersection of HFileBlock, CacheConfig, and LruBlockCache.
TestLruBlockCache	Tests the concurrent LruBlockCache.
TestLruCachedBlock
TestPrefetch
TestReseekTo	Test HFileScanner.reseekTo(byte[])
TestScannerSelectionUsingKeyRange	Test the optimization that does not scan files where all key ranges are excluded.
TestScannerSelectionUsingTTL	Test the optimization that does not scan files where all timestamps are expired.
TestSeekBeforeWithInlineBlocks
TestSeekTo	Test HFileScanner.seekTo(byte[]) and its variants.

Enum Summary

Enum	Description
BlockPriority
BlockType	Various types of HFile blocks.
BlockType.BlockCategory
TagUsage	Used in testcases only.

Exception Summary

Exception	Description
AbstractHFileReader.BlockIndexNotLoadedException
AbstractHFileReader.NotSeekedException	An exception thrown when an operation requiring a scanner to be seeked is invoked on a scanner that is not seeked.
CorruptHFileException	This exception is thrown when attempts to read an HFile fail due to corruption or truncation issues.
InvalidHFileException	Thrown when an invalid HFile format is detected

Package org.apache.hadoop.hbase.io.hfile Description

Provides implementations of HFile and HFile BlockCache. Caches are configured (and instantiated) by CacheConfig. See head of the CacheConfig class for constants that define cache options and configuration keys to use setting cache options. Cache implementations include the default, native on-heap LruBlockCache and a BucketCache that has a bunch of deploy formats including acting as a L2 for LruBlockCache -- when a block is evicted from LruBlockCache, it goes to the BucketCache and when we search a block, we look in both places -- or, the most common deploy type, using CombinedBlockCache, BucketCache is used as a host for data blocks with meta blocks in an instance of LruBlockCache. BucketCache can also be onheap, offheap, and file-backed.

Which BlockCache should I use?

By default LruBlockCache is on. If you would like to cache more, and offheap (offheap usually means less GC headache), try enabling * BucketCache. Fetching will always be slower when fetching from BucketCache but latencies tend to be less erratic over time (roughly because GC is less). See Nick Dimiduk's BlockCache 101 for some numbers.

Enabling BucketCache

See the HBase Reference Guide Enable BucketCache.