The 

Before You Begin

Key deployment considerations

1. Hadoop cluster: The Hadoop cluster should already be installed and operational. As part of the install preparation, you should have prepared the Hadoop platform for integration with the 

   D s platform

   . See Prepare Hadoop for Integration with the Platform. 
   1. For more information on the components supported in your Hadoop distribution, See Install Reference.
2. Storage: on-premises, cloud, or hybrid.
   1. The 

      D s platform

       can interact with storage that is in the local environment, in the cloud, or in some combination. How your storage is deployed affects your configuration scenarios. See Storage Deployment Options.

3. Base storage layer: You must configure one storage platform to be the base storage layer. Details are described later.

   Info
   NOTE: Some deployments require that you select a specific base storage layer.

   Warning
   After you have defined the base storage layer, it cannot be changed. Please review your Storage Deployment Options carefully. The required configuration is described later.

Platform configuration

After the

Specify 

D s item
item user

Set the Hadoop username

"hdfs.username": [hadoop.user],

If the 

Configuration for Hadoop

In the sections below are a series of questions about the Hadoop environment with which the 

Data storage

The 

• HDFS

• S3

Set the base storage layer

At this time, you should define the base storage layer from the platform. See Set Base Storage Layer.

Required configuration for each type of storage is described below.

HDFS

If output files are to be written to an HDFS environment, you must configure the

• Hadoop Distributed File Service (HDFS) is a distributed file system that provides read-write access to large datasets in a Hadoop cluster. For more information, see http://hadoop.apache.org/docs/r1.2.1/hdfs_design.html. 

Below, replace the value for 

"hdfs": {
  "username": "[hadoop.user]",
  ...
  "namenode": {
    "host": "hdfs.example.com",
    "port": 8080
  },
}, 

Individual users can configure the HDFS directory where exported results are stored.

See Storage Config Page.

Access to HDFS is supported over one of the following protocols:

• See WebHDFS below.
• See HttpFS below.

WebHDFS

If you are using HDFS, it is assumed that WebHDFS has been enabled on the cluster. Apache WebHDFS enables access to an HDFS instance over HTTP REST APIs. For more information, see https://hadoop.apache.org/docs/r1.0.4/webhdfs.html.

The following properties can be modified:

"webhdfs": {
  ...
  "version": "/webhdfs/v1",
  "host": "",
  "port": 50070,
  "httpfs": false
}, 

Steps:

1. Set webhdfs.host to be the hostname of the node that hosts WebHDFS. 
2. Set webhdfs.port to be the port number over which WebHDFS communicates. The default value is 50070. For SSL, the default value is 50470.
3. Set webhdfs.httpfs to false.
4. For hdfs.namenodes, you must set the host and port values to point to the active namenode for WebHDFS.

HttpFS

You can configure the 

In some cases, HttpFS is required:

• High availability requires HttpFS.
• Your secured HDFS user account has access restrictions.

If your environment meets any of the above requirements, you must enable HttpFS. For more information, see Enable HttpFS.

S3

The

Configure ResourceManager settings

Configure the following:

"yarn.resourcemanager.host": "hadoop",
"yarn.resourcemanager.port": 8032,

For more information, see System Ports.

Default Hadoop job results format

For smaller datasets, the platform recommends using the 

For larger datasets, if the size information is unavailable, the platform recommends by default that you run the job on the Hadoop cluster. For these jobs, the default publishing action for the job is specified to run on the Hadoop cluster, generating the output format defined by this parameter. Publishing actions, including output format, can always be changed as part of the job specification. 

As needed, you can change this default format. 

"webapp.defaultHadoopFileFormat": "csv",

Accepted values: csv, json, avro, pqt

For more information, see Run Job Page.

Specify distribution client bundle

The 

/opt/trifacta/hadoop-deps

Configure the bundle distribution property (hadoopBundleJar) in platform configuration. Examples:

where:

x.y is the major-minor build number (e.g. 5.4)

Authentication

Kerberos

The 

See Configure for Kerberos Integration. 

See Configure for Secure Impersonation.

Single Sign-On

The 

Hadoop KMS

If you are using Hadoop KMS to encrypt data transfers to and from the Hadoop cluster, additional configuration is required. See Configure for KMS.

Hadoop distribution version

Cloudera distribution

Some additional configuration is required. See Configure for Cloudera.

Hortonworks distribution

After install, integration with the Hortonworks Data Platform requires additional configuration. See Configure for Hortonworks.

Hive access

Apache Hive is a data warehouse service for querying and managing large datasets in a Hadoop environment using a SQL-like querying language. For more information, see https://hive.apache.org/.

See Configure for Hive.

High availability environment

You can integrate the platform with the Hadoop cluster's high availability configuration, so that the 

For more information, see Enable Integration with Cluster High Availability.

Acquire Hadoop cluster configuration files

To enable the platform to use YARN installations, you must provide a set of client *-site.xml files.

core-site.xml

hdfs-site.xml

httpfs-site.xml

mapred-site.xml

yarn-site.xml

hive-site.xml

Locate Client Configuration

For CDH 5:

In Cloudera Manager, select Actions > Download Client Configuration.

Configuration files are also available in /etc/hadoop/conf on any cluster edge node.

For HDP 2:

Client configuration files can be retrieved from an existing client node. Acquire  *-site.xml files from  /etc/hadoop/conf.

If you are using Hortonworks, you must complete the following modification to the site configuration file that is hosted on the 

In the 

/opt/trifacta/conf/hadoop-site/mapred-site.xml

Perform the following global search and replace:

1. Search:

   Code Block
   ${hdp.version}

2. Replace with your hard-coded version and build number:

   Code Block
   A.B.C.D-XXXX

Save the file. 

Restart the 

For YARN:

YARN maintains site site configuration files in a similar location. These XML files should be retrieved, too.

Deploy Client Configuration

After you've collected the Hadoop client configuration, copy all *-site.xml files to the following: 

<installation root>/conf/hadoop-site/  

Restart services. See Start and Stop the Platform.

Configure Snappy publication

If you are publishing using Snappy compression, you may need to perform the following additional configuration.

Steps:

1. Verify that the snappy and snappy-devel packages have been installed on the 

   D s node

   . For more information, see https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/NativeLibraries.html.

2. From the 

   D s node

   , execute the following command:
   
   

   Code Block
   hadoop checknative

3. The above command identifies where the native libraries are located on the 

   D s node

   . 
4. Cloudera: 
   1. On the cluster, locate the libsnappy.so file. Verify that this file has been installed on all nodes of the cluster, including the 

      D s node

      . Retain the path to the file on the 

      D s node

      .

   2. D s config

   3. Locate the spark.props configuration block. Insert the following properties and values inside the block:
      
      

      Code Block
      "spark.driver.extraLibraryPath": "/path/to/file", "spark.executor.extraLibraryPath": "/path/to/file",

5. Hortonworks:
   

   1. Verify on the 

      D s node

       that the following locations are available:

      Info
      NOTE: The asterisk below is a wildcard. Please collect the entire path of both values.

      Code Block
      /hadoop-client/lib/snappy*.jar /hadoop-client/lib/native/

   2. D s config

   3. Locate the spark.props configuration block. Insert the following properties and values inside the block:

      Code Block
      "spark.driver.extraLibraryPath": "/hadoop-client/lib/snappy*.jar;/hadoop-client/lib/native/", "spark.executor.extraLibraryPath": "/hadoop-client/lib/snappy*.jar;/hadoop-client/lib/native/",

6. Save your changes and restart the platform.
7. Verify that the /tmp directory has the proper permissions for publication. For more information, see Supported File Formats.

Debugging

You can review system services and download log files through the

See System Services and Logs.