Prepare Hadoop for Integration with the Platform

Before you deploy the Alteryx® software, you should complete the following configuration steps within your Hadoop environment. 

• For a technical overview of how the Designer Cloud Powered by Trifacta platform interacts with Hadoop, see Platform Interactions with Hadoop.

Create Alteryx user account on Hadoop cluster

The Designer Cloud Powered by Trifacta platform interacts with Hadoop through a single system user account.  A user for the platform must be added to the cluster.

UserID:

If possible, please create the user ID as: trifacta

This user should belong to the group: trifactausers

User requirements:

• Access to HDFS
• Permission to run YARN jobs on the cluster. 

Verify that the following HDFS paths have been created and that their permissions enable access to the Alteryx user account:

Below, change the values for trifacta to match the [hadoop.user] user for your environment:

hdfs dfs -mkdir /trifacta
hdfs dfs -chown trifacta /trifacta
hdfs dfs -mkdir -p /user/trifacta
hdfs dfs -chown trifacta /user/trifacta

HDFS directories

The following directories must be available to the [hadoop.user] on HDFS. Below, you can review the minimum permissions set for basic and impersonated authentication for each default directory. Secure impersonation is described later.

You can use the following commands to configure permissions on these directories. Following permissions scheme reflects the secure impersonation permissions in the above table:

$ hdfs dfs -mkdir -p /trifacta/uploads
$ hdfs dfs -mkdir -p /trifacta/queryResults
$ hdfs dfs -mkdir -p /trifacta/dictionaries
$ hdfs dfs -mkdir -p /trifacta/tempfiles
$ hdfs dfs -chown -R trifacta:trifacta /trifacta
$ hdfs dfs -chmod -R 770 /trifacta
$ hdfs dfs -chmod -R 730 /trifacta/uploads

If these standard locations cannot be used, you can configure the HDFS paths. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

"hdfs.pathsConfig.fileUpload": "/trifacta/uploads", 
"hdfs.pathsConfig.batchResults": "/trifacta/queryResults",
"hdfs.pathsConfig.dictionaries": "/trifacta/dictionaries",

Kerberos authentication

The Designer Cloud Powered by Trifacta platform supports Kerberos authentication on Hadoop.

Hadoop component configuration

Acquire cluster configuration files

The Hadoop cluster configuration files must be made available to the Designer Cloud Powered by Trifacta platform. You can either copy the files over from the cluster or create a local symlink to them. 

For more information, see Configure for Hadoop.

YARN configuration overview

This section provides an overview of configuration recommendations to be applied to the Hadoop cluster from the Designer Cloud Powered by Trifacta platform.

YARN manages cluster resources (CPU and memory) by running all processes within allocated containers. Containers restrict the resources available to its process(es). Processes are monitored and killed if they overrun the container allocation. 

• Multiple containers can run on a cluster node (if available resources permit).
• A job can request and use multiple containers across the cluster.
• Container requests specify virtual CPU (cores) and memory (in MB).

YARN configuration specifies:

• Per Cluster Node: Available virtual CPUs and memory per cluster node
• Per Container: virtual CPUs and memory for each container

The following parameters are available in yarn-site.xml:

Spark configuration overview

Spark processes run multiple executors per job. Each executor must run within a YARN container. Therefore, resource requests must fit within YARN’s container limits.

Like YARN containers, multiple executors can run on a single node. More executors provide additional computational power and decreased runtime.

Spark’s dynamic allocation adjusts the number of executors to launch based on the following:

• job size

• job complexity

• available resources

You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

The per-executor resource request sizes can be specified by setting the following properties in the spark.props section:

A single special process, the application driver, also runs in a container. Its resources are specified in the spark.props section:

Recommendations

The following configuration settings can be applied through Designer Cloud Powered by Trifacta platform configuration based on the number of nodes in the Hadoop cluster. 

The specified configuration allows, maximally, the following Spark configuration per node: