Install Hadoop Configure Data Storage

• HDFS

• S3

Set the base storage layer

At this time, you should define the base storage layer from the platform. 

The platform requires that one backend datastore be configured as the base storage layer. This base storage layer is used for storing uploaded data and writing results and profiles. Please complete the following steps to set the base storage layer for the Designer Cloud Powered by Trifacta platform.

Steps:

1. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

2. Locate the following parameter and set it to the value for your base storage layer:

   "webapp.storageProtocol": "hdfs",

3. Save your changes and restart the platform.
   
   

   NOTE: To complete the integration with the base storage layer, additional configuration is required.

Required configuration for each type of storage is described below.

S3

The Designer Cloud Powered by Trifacta platform can integrate with an S3 bucket:

• If you are using HDFS as the base storage layer, you can integrate with S3 for read-only access.

• Base storage layer requires read-write access.

  NOTE: If you are integrating with S3, additional configuration is required. Instead of completing the HDFS configuration below, please enable read-write access to S3. See S3 Access in the Configuration Guide.

HDFS

If output files are to be written to an HDFS environment, you must configure the Designer Cloud Powered by Trifacta platform to interact with HDFS. 

• Hadoop Distributed File Service (HDFS) is a distributed file system that provides read-write access to large datasets in a Hadoop cluster. For more information, see http://hadoop.apache.org/docs/r1.2.1/hdfs_design.html. 

Below, replace the value for [hadoop.user (default=trifacta)] with the value appropriate for your environment. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

"hdfs": {
  "username": "[hadoop.user]",
  ...
  "namenode": {
    "host": "hdfs.example.com",
    "port": 8080
  },
}, 

Individual users can configure the HDFS directory where exported results are stored.

See Storage Config Page in the User Guide.

Access to HDFS is supported over one of the following protocols:

• See 202959495 below.
• See 202959495 below.

WebHDFS

If you are using HDFS, it is assumed that WebHDFS has been enabled on the cluster. Apache WebHDFS enables access to an HDFS instance over HTTP REST APIs. For more information, see https://hadoop.apache.org/docs/r1.0.4/webhdfs.html.

The following properties can be modified:

"webhdfs": {
  ...
  "version": "/webhdfs/v1",
  "host": "",
  "port": 50070,
  "httpfs": false
}, 

Steps:

1. Set webhdfs.host to be the hostname of the node that hosts WebHDFS. 
2. Set webhdfs.port to be the port number over which WebHDFS communicates. The default value is 50070. For SSL, the default value is 50470.
3. Set webhdfs.httpfs to false.
4. For hdfs.namenodes, you must set the host and port values to point to the active namenode for WebHDFS.

HttpFS

You can configure the Designer Cloud Powered by Trifacta platform to use the HttpFS service to communicate with HDFS, in addition to WebHDFS. 

In some cases, HttpFS is required:

• High availability requires HttpFS.
• Your secured HDFS user account has access restrictions.

If your environment meets any of the above requirements, you must enable HttpFS. For more information, see Enable HttpFS in the Configuration Guide.