Configure for Hadoop

The Designer Cloud Powered by Trifacta® platform supports integration with a number of Hadoop distributions, using a range of components within each distribution. This page provides information on the set of configuration tasks that you need to complete to integrate the platform with your Hadoop environment.

Before You Begin

Key deployment considerations

1. Hadoop cluster: The Hadoop cluster should already be installed and operational. As part of the install preparation, you should have prepared the Hadoop platform for integration with the Designer Cloud Powered by Trifacta platform. See Prepare Hadoop for Integration with the Platform. 
   1. For more information on the components supported in your Hadoop distribution, See Install Reference.
2. Storage: on-premises, cloud, or hybrid.
   1. The Designer Cloud Powered by Trifacta platform can interact with storage that is in the local environment, in the cloud, or in some combination. How your storage is deployed affects your configuration scenarios. See Storage Deployment Options.

3. Base storage layer: You must configure one storage platform to be the base storage layer. Details are described later.

   NOTE: Some deployments require that you select a specific base storage layer.

   After you have defined the base storage layer, it cannot be changed. Please review your Storage Deployment Options carefully. The required configuration is described later. 

Platform configuration

After the Designer Cloud Powered by Trifacta platform and its databases have been installed, you can perform platform configuration. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

Specify Alteryx user

Set the Hadoop username [hadoop.user (default=trifacta)] for the Designer Cloud Powered by Trifacta platform to use for executing jobs:

"hdfs.username": [hadoop.user],

If the Alteryx software is installed in a Kerberos environment, additional steps are required, which are described later.

Configuration for Hadoop

In the sections below are a series of questions about the Hadoop environment with which the Designer Cloud Powered by Trifacta platform is integrating. Based on your answer, additional configuration may be required.

Data storage

The Designer Cloud Powered by Trifacta platform supports access to the following storage environments:

• HDFS
• S3

Set the base storage layer

At this time, you should define the base storage layer from the platform. See Set Base Storage Layer.

Required configuration for each type of storage is described below.

HDFS

If output files are to be written to an HDFS environment, you must configure the Designer Cloud Powered by Trifacta platform to interact with HDFS. 

• Hadoop Distributed File Service (HDFS) is a distributed file system that provides read-write access to large datasets in a Hadoop cluster. For more information, see http://hadoop.apache.org/docs/r1.2.1/hdfs_design.html. 

Below, replace the value for [hadoop.user (default=trifacta)] with the value appropriate for your environment. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

"hdfs": {
  "username": "[hadoop.user]",
  ...
  "namenode": {
    "host": "hdfs.example.com",
    "port": 8080
  },
}, 

Individual users can configure the HDFS directory where exported results are stored.

See User Profile Page.

Access to HDFS is supported over one of the following protocols:

• See WebHDFS below.
• See HttpFS below.

WebHDFS

If you are using HDFS, it is assumed that WebHDFS has been enabled on the cluster. Apache WebHDFS enables access to an HDFS instance over HTTP REST APIs. For more information, see https://hadoop.apache.org/docs/r1.0.4/webhdfs.html.

The following properties can be modified:

"webhdfs": {
  ...
  "version": "/webhdfs/v1",
  "host": "",
  "port": 50070,
  "httpfs": false
}, 

Steps:

1. Set webhdfs.host to be the hostname of the node that hosts WebHDFS. 
2. Set webhdfs.port to be the port number over which WebHDFS communicates. The default value is 50070. For SSL, the default value is 50470.
3. Set webhdfs.httpfs to false.
4. For hdfs.namenodes, you must set the host and port values to point to the active namenode for WebHDFS.

HttpFS

You can configure the Designer Cloud Powered by Trifacta platform to use the HttpFS service to communicate with HDFS, in addition to WebHDFS. 

In some cases, HttpFS is required:

• High availability requires HttpFS.
• Your secured HDFS user account has access restrictions.

If your environment meets any of the above requirements, you must enable HttpFS. For more information, see Enable HttpFS.

S3

The Designer Cloud Powered by Trifacta platform can integrate with an S3 bucket for reading and writing of data. See Enable S3 Access.

Configure ResourceManager settings

Configure the following:

"yarn.resourcemanager.host": "hadoop",
"yarn.resourcemanager.port": 8032,

For more information, see System Ports.

Default Hadoop job results format

For smaller datasets, the platform recommends using the Alteryx Server.

For larger datasets, if the size information is unavailable, the platform recommends by default that you run the job on the Hadoop cluster. For these jobs, the default publishing action for the job is specified to run on the Hadoop cluster, generating the output format defined by this parameter. Publishing actions, including output format, can always be changed as part of the job specification. 

As needed, you can change this default format. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

"webapp.defaultHadoopFileFormat": "csv",

Accepted values: csv, json, avro, pqt

For more information, see Run Job Page.

Specify distribution client bundle

The Designer Cloud Powered by Trifacta platform ships with client bundles supporting a number of major Hadoop distributions.  You must configure the jarfile for the distribution to use.  These distributions are stored in the following directory:

/opt/trifacta/hadoop-deps

Configure the bundle distribution property (hadoopBundleJar) in platform configuration. Examples:

where:

x.y is the major-minor build number (e.g. 5.4)

Authentication

Kerberos

The Designer Cloud Powered by Trifacta platform supports integration with Kerberos security. The platform can utilize Kerberos' secure impersonation to broker interactions with the Hadoop environment.

See Set up for a Kerberos-enabled Hadoop cluster. 

See Configure for secure impersonation.

Single Sign-On

The Designer Cloud Powered by Trifacta platform can integrate with your SSO platform to manage authentication to the Designer Cloud application. See Configure SSO for AD-LDAP.

Hadoop KMS

If you are using Hadoop KMS to encrypt data transfers to and from the Hadoop cluster, additional configuration is required. See Configure for KMS.

Hadoop distribution version

Cloudera distribution

Some additional configuration is required. See Configure for Cloudera.

Hortonworks distribution

After install, integration with the Hortonworks Data Platform requires additional configuration. See Configure for Hortonworks.

Hive access

Apache Hive is a data warehouse service for querying and managing large datasets in a Hadoop environment using a SQL-like querying language. For more information, see https://hive.apache.org/.

See Configure for Hive.

High availability environment

You can integrate the platform with the Hadoop cluster's high availability configuration, so that the Designer Cloud Powered by Trifacta platform can match the failover configuration for the cluster.

For more information, see Enable Integration with Cluster High Availability.

Acquire Hadoop cluster configuration files

To enable the platform to use YARN installations, you must provide a set of client *-site.xml files.

core-site.xml

hdfs-site.xml

httpfs-site.xml

mapred-site.xml

yarn-site.xml

hive-site.xml

Locate Client Configuration

For CDH 5:

In Cloudera Manager, select Actions > Download Client Configuration.

Configuration files are also available in /etc/hadoop/conf on any cluster edge node.

For HDP 2:

Client configuration files can be retrieved from an existing client node. Acquire  *-site.xml files from  /etc/hadoop/conf.

If you are using Hortonworks, you must complete the following modification to the site configuration file that is hosted on the  Alteryx node. 

In the Alteryx deployment, edit the following file:

/opt/trifacta/conf/hadoop-site/mapred-site.xml

Perform the following global search and replace:

1. Search:

   ${hdp.version}

2. Replace with your hard-coded version and build number:

   A.B.C.D-XXXX

Save the file. 

Restart the Designer Cloud Powered by Trifacta platform.

For YARN:

YARN maintains site site configuration files in a similar location. These XML files should be retrieved, too.

Deploy Client Configuration

After you've collected the Hadoop client configuration, copy all *-site.xml files to the following: 

<installation root>/conf/hadoop-site/  

Restart services. See Start and Stop the Platform.

Debugging

You can review system services and download log files through the Designer Cloud application.

See System Services and Logs.