Requirements for Azure Blob FS

  Since    v5.5.5/4  

Contents

Azure Blob FS
Capture Hub Integrate

This section describes the requirements, access privileges, and other features of HVR when using Azure Blob FS for replication. For information about compatibility and support for Azure Blob FS with HVR platforms, see Platform Compatibility Matrix.

Location Connection

This section lists and describes the connection details required for creating Azure Blob FS location in HVR.

Field

Description

Azure Blob FS

Secure connection

The type of security to be used for connecting to Azure Blob Server. Available options:

  • Yes (https) (default) : HVR will connect to Azure Blob Server using HTTPS.
  • No (http) : HVR will connect to Azure Blob Server using HTTP.

Account

The Azure Blob storage account.
Example: mystorageaccount

Container

The name of the container available within storage Account.
Example: myblobcontainer

Directory

The directory path in Container which is to be used for replication.
Example: /folder

Secret key

The access key of the storage Account.

Hive External Tables

Enable/Disable Hive ODBC connection configuration for creating Hive external tables above Azure Blob FS.

Hive ODBC Connection

HVR allows you to create Hive External Tables above Azure Blob FS files which are only used during compare. You can enable/disable the Hive configuration for Azure Blob FS in location creation screen using the field Hive External Tables . For more information about configuring Hive external tables, refer to Hadoop Azure Blob FS Support documentation.

Field

Description

Hive ODBC Connection

Hive Server Type

The type of Hive server.  Available options:

  • Hive Server 1 (default): The driver connects to a Hive Server 1 instance.
  • Hive Server 2: The driver connects to a Hive Server 2 instance.

Service Discovery Mode

The mode for connecting to Hive. This field is enabled only if Hive Server Type is Hive Server 2.  Available options:

  • No Service Discovery (default): The driver connects to Hive server without using the ZooKeeper service.
  • ZooKeeper: The driver discovers Hive Server 2 services using the ZooKeeper service.

Host(s)

The hostname or IP address of the Hive server.
When Service Discovery Mode is ZooKeeper, specify the list of ZooKeeper servers in following format [ZK_Host1]:[ZK_Port1],[ZK_Host2]:[ZK_Port2], where [ZK_Host] is the IP address or hostname of the ZooKeeper server and [ZK_Port] is the TCP port that the ZooKeeper server uses to listen for client connections.
Example: hive-host

Port

The TCP port that the Hive server uses to listen for client connections. This field is enabled only if Service Discovery Mode is No Service Discovery.
Example: 10000

Database

The name of the database schema to use when a schema is not explicitly specified in a query.
Example: mytestdb

ZooKeeper Namespace

The namespace on ZooKeeper under which Hive Server 2 nodes are added. This field is enabled only if Service Discovery Mode is ZooKeeper.

Authentication

Mechanism

The authentication mode for connecting HVR to Hive Server 2.  This field is enabled only if Hive Server Type is Hive Server 2. Available options:

  • No Authentication (default)
  • User Name
  • User Name and Password
  • Kerberos

User

The username to connect HVR to Hive server. This field is enabled only if Mechanism is User Name or User Name and Password.
Example: dbuser

Password

The password of the User to connect HVR to Hive server. This field is enabled only if Mechanism is User Name and Password.

Service Name

The Kerberos service principal name of the Hive server. This field is enabled only if Mechanism is Kerberos.

Host

The Fully Qualified Domain Name (FQDN) of the Hive Server 2 host. The value of Host can be set as _HOST to use the Hive server hostname as the domain name for Kerberos authentication.
If Service Discovery Mode is disabled, then the driver uses the value specified in the Host connection attribute.
If Service Discovery Mode is enabled, then the driver uses the Hive Server 2 host name returned by ZooKeeper.
This field is enabled only if Mechanism is Kerberos.

Realm

The realm of the Hive Server 2 host.
It is not required to specify any value in this field if the realm of the Hive Server 2 host is defined as the default realm in Kerberos configuration. This field is enabled only if Mechanism is Kerberos.

Linux / Unix

Driver Manager Library

The directory path where the Unix ODBC Driver Manager Library is installed.
Example: /opt/unixodbc-2.3.2/lib

ODBCSYSINI

The directory path where odbc.ini and odbcinst.ini files are located.
Example: /opt/unixodbc-2.3.2/etc

ODBC Driver

The user defined (installed) ODBC driver to connect HVR to the Hive server.

SSL Options

Show SSL Options.

SSL Options

Field

Description

Enable SSL

Enable/disable (one way) SSL. If enabled, HVR authenticates the Hive server by validating the SSL certificate shared by the Hive server.

Two-way SSL

Enable/disable two way SSL. If enabled, both HVR and Hive server authenticate each other by validating each others SSL certificate. This field is enabled only if Enable SSL is selected.

SSL Public Certificate

The directory path where the .pem file containing the client's SSL public certificate is located. This field is enabled only if Two-way SSL is selected.

SSL Private Key

The directory path where the .pem file containing the client's SSL private key is located. This field is enabled only if Two-way SSL is selected.

Client Private Key Password

The password of the private key file that is specified in SSL Private Key. This field is enabled only if Two-way SSL is selected.

Hadoop Client

The Hadoop client should be present on the machine from which HVR will access the Azure Blob FS. Internally, HVR uses the WebHDFS REST API to connect to the Azure Blob FS. Azure Blob FS locations can only be accessed through HVR running on Linux or Windows, and it is not required to run HVR installed on the Hadoop NameNode although it is possible to do so. For more information about installing Hadoop client, refer to Apache Hadoop Releases.

Hadoop Client Configuration

The following are required on the machine from which HVR connects to Azure Blob FS:

  • Hadoop 2.6.x client libraries with Java 7 Runtime Environment or Hadoop 3.x client libraries with Java 8 Runtime Environment. For downloading Hadoop, refer to Apache Hadoop Releases.
  • Set the environment variable $JAVA_HOME to the Java installation directory.
  • Set the environment variable $HADOOP_COMMON_HOME or $HADOOP_HOME or $HADOOP_PREFIX to the Hadoop installation directory, or the hadoop command line client should be available in the path.
  • One of the following configuration is recommended,
    • Add $HADOOP_HOME/share/hadoop/tools/lib into Hadoop classpath.
    • Create a symbolic link for $HADOOP_HOME/share/hadoop/tools/lib in $HADOOP_HOME/share/hadoop/common or any other directory present in classpath.

      Since the binary distribution available in Hadoop website lacks Windows-specific executables, a warning about unable to locate winutils.exe is displayed. This warning can be ignored for using Hadoop library for client operations to connect to a HDFS server using HVR. However, the performance on integrate location would be poor due to this warning, so it is recommended to use a Windows-specific Hadoop distribution to avoid this warning. For more information about this warning, refer to Hadoop issue HADOOP-10051.

Verifying Hadoop Client Installation

To verify the Hadoop client installation,

  1. The HADOOP_HOME/bin directory in Hadoop installation location should contain the hadoop executables in it.
  2. Execute the following commands to verify Hadoop client installation:

    $JAVA_HOME/bin/java -version
    $HADOOP_HOME/bin/hadoop version
    $HADOOP_HOME/bin/hadoop classpath
    
  3. If the Hadoop client installation is verified successfully then execute the following command to check the connectivity between HVR and Azure Blob FS:

    To execute this command successfully and avoid the error "ls: Password fs.adl.oauth2.client.id not found", few properties needs to be defined in the file core-site.xml available in the hadoop configuration folder (for e.g., <path>/hadoop-2.8.3/etc/hadoop). The properties to be defined differs based on the Mechanism (authentication mode). For more information, refer to section 'Configuring Credentials' in Hadoop Azure Blob FS Support documentation.

    $HADOOP_HOME/bin/hadoop fs -ls adl://<cluster>/
    

Verifying Hadoop Client Compatibility with Azure Blob FS

To verify the compatibility of Hadoop client with Azure Blob FS, check if the following JAR files are available in the Hadoop client installation location ( $HADOOP_HOME/share/hadoop/tools/lib ):

hadoop-azure-<version>.jar
azure-storage-<version>.jar

Authentication

HVR does not support client side encryption (customer managed keys) for Azure Blob FS. For more information about encryption of data in Azure Blob FS, search for "encryption" in Azure Blob storage documentation.

Client Configuration Files

Client configuration files are not required for HVR to perform replication, however, they can be useful for debugging. Client configuration files contain settings for different services like HDFS or HBASE. If the HVR integrate machine is not part of the cluster, it is recommended to download the configuration files for the cluster so that the Hadoop client knows how to connect to HDFS.

The client configuration files for Cloudera Manager or Ambari for Hortonworks can be downloaded from the respective cluster manager's web interface. For more information about downloading client configuration files, search for "Client Configuration Files" in the respective documentation for Cloudera and Hortonworks.

Hive External Table

HVR allows you to create Hive external tables above Azure Blob FS files which are only used during compare. The Hive ODBC connection can be enabled for Azure Blob FS in the location creation screen by selecting the Hive External Tables field. For more information about configuring Hive external tables for Azure Blob FS, refer to Hadoop Azure Support: Azure Blob Storage documentation.

ODBC Connection

HVR uses ODBC connection to the Hadoop cluster for which it requires the ODBC driver (Amazon ODBC 1.1.1 or HortonWorks ODBC 2.1.2 and above) for Hive installed on the machine (or in the same network).

The Amazon and HortonWorks ODBC drivers are similar and compatible to work with Hive 2.x release. However, it is recommended to use the Amazon ODBC driver for Amazon Hive and the Hortonworks ODBC driver for HortonWorks Hive.

By default, HVR uses Amazon ODBC driver for connecting to Hadoop. To use the Hortonworks ODBC driver the following action definition is required:

For Linux,

Group

Table

Action

Azure Blob FS

*

Environment /Name = HVR_ODBC_CONNECT_STRING_DRIVER /Value = Hortonworks Hive ODBC Driver 64-bit


For Windows,

Group

Table

Action

Azure Blob FS

*

Environment /Name = HVR_ODBC_CONNECT_STRING_DRIVER /Value = Hortonworks Hive ODBC Driver


Channel Configuration

For the file formats (CSV, JSON, and AVRO) the following action definitions are required to handle certain limitations of the Hive deserialization implementation during Bulk or Row-wise Compare:

  • For CSV,

    Group

    Table

    Action

    Azure Blob FS

    *

    FileFormat /NullRepresentation=\\N

    Azure Blob FS

    *

    TableProperties /CharacterMapping="\x00>\\0;\n>\\n;\r>\\r;">\""

    Azure Blob FS

    *

    TableProperties /MapBinary=BASE64

  • For JSON,

    Group

    Table

    Action

    Azure Blob FS

    *

    TableProperties /MapBinary=BASE64

    Azure Blob FS

    *

    FileFormat /JsonMode=ROW_FRAGMENTS

  • For Avro,

    Group

    Table

    Action

    Azure Blob FS

    *

    FileFormat /AvroVersion=v1_8

    v1_8 is the default value for FileFormat /AvroVersion, so it is not mandatory to define this action.