org.apache.hadoop.hive.ql.io
Class RCFile.Writer

java.lang.Object
  org.apache.hadoop.hive.ql.io.RCFile.Writer

Enclosing class:

RCFile

public static class RCFile.Writer
extends Object

Write KeyBuffer/ValueBuffer pairs to a RCFile. RCFile's format is compatible with SequenceFile's.

Field Summary

static String

COLUMNS_BUFFER_SIZE_CONF_STR

Constructor Summary

RCFile.Writer(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path name)
Constructs a RCFile Writer.

RCFile.Writer(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path name, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.io.SequenceFile.Metadata metadata, org.apache.hadoop.io.compress.CompressionCodec codec)
Constructs a RCFile Writer.

RCFile.Writer(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path name, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.io.compress.CompressionCodec codec)
Constructs a RCFile Writer.

RCFile.Writer(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path name, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.io.SequenceFile.Metadata metadata, org.apache.hadoop.io.compress.CompressionCodec codec)
Constructs a RCFile Writer.

Method Summary

void	append(org.apache.hadoop.io.Writable val) Append a row of values.
void	close()
void	flushBlock(RCFile.KeyBuffer keyBuffer, RCFile.ValueBuffer valueBuffer, int recordLen, int keyLength, int compressedKeyLen) flush a block out without doing anything except compressing the key part.
org.apache.hadoop.io.compress.CompressionCodec	getCompressionCodec() Returns the compression codec of data in this file.
long	getLength()
void	sync() create a sync point.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

COLUMNS_BUFFER_SIZE_CONF_STR

public static String COLUMNS_BUFFER_SIZE_CONF_STR

Constructor Detail

RCFile.Writer

public RCFile.Writer(org.apache.hadoop.fs.FileSystem fs,
                     org.apache.hadoop.conf.Configuration conf,
                     org.apache.hadoop.fs.Path name)
              throws IOException

Constructs a RCFile Writer.

Throws:

IOException

RCFile.Writer

public RCFile.Writer(org.apache.hadoop.fs.FileSystem fs,
                     org.apache.hadoop.conf.Configuration conf,
                     org.apache.hadoop.fs.Path name,
                     org.apache.hadoop.util.Progressable progress,
                     org.apache.hadoop.io.compress.CompressionCodec codec)
              throws IOException

Constructs a RCFile Writer.

Parameters:

fs - the file system used

conf - the configuration file

name - the file name

Throws:

IOException

RCFile.Writer

public RCFile.Writer(org.apache.hadoop.fs.FileSystem fs,
                     org.apache.hadoop.conf.Configuration conf,
                     org.apache.hadoop.fs.Path name,
                     org.apache.hadoop.util.Progressable progress,
                     org.apache.hadoop.io.SequenceFile.Metadata metadata,
                     org.apache.hadoop.io.compress.CompressionCodec codec)
              throws IOException

Constructs a RCFile Writer.

Parameters:

fs - the file system used

conf - the configuration file

name - the file name

progress -

metadata -

Throws:

IOException

RCFile.Writer

public RCFile.Writer(org.apache.hadoop.fs.FileSystem fs,
                     org.apache.hadoop.conf.Configuration conf,
                     org.apache.hadoop.fs.Path name,
                     int bufferSize,
                     short replication,
                     long blockSize,
                     org.apache.hadoop.util.Progressable progress,
                     org.apache.hadoop.io.SequenceFile.Metadata metadata,
                     org.apache.hadoop.io.compress.CompressionCodec codec)
              throws IOException

Constructs a RCFile Writer.

Parameters:

fs - the file system used

conf - the configuration file

name - the file name

bufferSize -

replication -

blockSize -

progress -

metadata -

Throws:

IOException

Method Detail

getLength

public long getLength()
               throws IOException

Throws:

IOException

getCompressionCodec

public org.apache.hadoop.io.compress.CompressionCodec getCompressionCodec()

Returns the compression codec of data in this file.

sync

public void sync()
          throws IOException

create a sync point.

Throws:

IOException

append

public void append(org.apache.hadoop.io.Writable val)
            throws IOException

Append a row of values. Currently it only can accept < BytesRefArrayWritable. If its size() is less than the column number in the file, zero bytes are appended for the empty columns. If its size() is greater then the column number in the file, the exceeded columns' bytes are ignored.

Parameters:

val -

Throws:

IOException

flushBlock

public void flushBlock(RCFile.KeyBuffer keyBuffer,
                       RCFile.ValueBuffer valueBuffer,
                       int recordLen,
                       int keyLength,
                       int compressedKeyLen)
                throws IOException

flush a block out without doing anything except compressing the key part.

Throws:

IOException

close

public void close()
           throws IOException

Throws:

IOException