Optimizing Performance for HDFS Transparent Encryption

CDP implements the Advanced Encryption Standard New Instructions (AES-NI), which provide substantial performance improvements. To get these improvements, you need a recent version of libcrypto.so on HDFS and MapReduce client hosts -- that is, any host from which you originate HDFS or MapReduce requests. Many OS versions have an older version of the library that does not support AES-NI. The instructions that follow tell you what you need to do for each OS version that CDP supports.

RHEL/CentOS 6.5 or later

The installed version of libcrypto.so supports AES-NI, but you need to install the openssl-devel package on all clients:

sudo yum install openssl-devel

RHEL/CentOS 6.4 or earlier 6.x versions, or SLES 11

Download and extract a newer version of libcrypto.so from a CentOS 6.5 repository and install it on all clients in /var/lib/hadoop/extra/native/:

Download the latest version of the openssl package. For example:
```
wget http://mirror.centos.org/centos/6/os/x86_64/Packages/openssl-1.0.1e-30.el6.x86_64.rpm
```
The libcrypto.so file in this package can be used on SLES 11 as well as RHEL/CentOS.
Decompress the files in the package, but do not install it:
```
rpm2cpio openssl-1.0.1e-30.el6.x86_64.rpm | cpio -idmv
```
If you are using parcels, create the /var/lib/hadoop/extra/native/ directory:
```
sudo mkdir -p /var/lib/hadoop/extra/native
```
Copy the shared library into /var/lib/hadoop/extra/native/. Name the target file libcrypto.so, with no suffix at the end, exactly as in the command that follows.
```
sudo cp ./usr/lib64/libcrypto.so.1.0.1e /var/lib/hadoop/extra/native/libcrypto.so
```

Debian Wheezy

The installed version of libcrypto.so supports AES-NI, but you need to install the libssl-devel package on all clients:

sudo apt-get install libssl-dev

Ubuntu Precise and Ubuntu Trusty

Install the libssl-devel package on all clients:

sudo apt-get install libssl-dev

Testing if encryption optimization works

To verify that a client host is ready to use the AES-NI instruction set optimization for HDFS encryption at rest, use the following command:

hadoop checknative

You should see a response such as the following:

14/12/12 13:48:39 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2
library system-native14/12/12 13:48:39 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /usr/lib64/libsnappy.so.1
lz4:     true revision:99
bzip2:   true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so

If you see true in the openssl row, Hadoop has detected the right version of libcrypto.so and optimization will work. If you see false in this row, you do not have the right version.

Ubuntu Precise, Ubuntu Bionic, and Ubuntu Trusty

Install the libssl-devel package on all clients: sudo apt-get install libssl-dev. This installs OpenSSL 1.1.1 which is supported.