Configure for EMR

Supported Versions: EMR 6.2.1, 6.3

Supported Versions: EMR 5.13 - EMR 5.30.2

Depending on the version of EMR in use, you must configure the Designer Cloud Powered by Trifacta platform to use the appropriate version of Spark. Please note the appropriate configuration settings below for later use.

"spark.version": "3.0.1",

"spark.scalaVersion": "2.12",

"spark.version": "2.4.6",

"spark.version": "2.3.0",

If you are integrating with a kerberized EMR cluster, the following additional limitations apply:

In the Amazon EMR console, click Create Cluster. Click Go to advanced options. Complete the sections listed below.

For more information on setting up your EMR cluster, see http://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html.

The following cluster roles and their permissions are required. For more information on the specifics of these policies, see "Access Policies" below.

You can use one of two methods for authenticating the EMR cluster's access to S3:

You must set up S3 buckets for read and write access.

For more information, see S3 Access.

On the EMR cluster, all users of the platform must have access to the following locations:

These locations are configured on the Designer Cloud Powered by Trifacta platform later.

Alteryx users require the following policies to run jobs on the EMR cluster:

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "elasticmapreduce:AddJobFlowSteps",
                "elasticmapreduce:DescribeStep",
                "elasticmapreduce:DescribeCluster",
                "elasticmapreduce:ListInstanceGroups",
                "elasticmapreduce:CancelSteps"
            ],
            "Resource": [
                "*"
            ]
        },
       {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::__EMR_LOG_BUCKET__",
                "arn:aws:s3:::__EMR_LOG_BUCKET__/*",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__/*"
            ]
        }

    ]
}

                "elasticmapreduce:listInstances",

The following policies should be assigned to the EMR roles listed below for read/write access:

{
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::__EMR_LOG_BUCKET__",
                "arn:aws:s3:::__EMR_LOG_BUCKET__/*",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__/*"
            ]
        }
}

Please complete the following sections to configure the Designer Cloud Powered by Trifacta platform to communicate with the EMR cluster.

As soon as you have installed the software, you should login to the application and change the admin password. The initial admin password is the instanceId for the EC2 instance. For more information, see Change Password.

EMR integrations requires use of S3 as the base storage layer.

See Set Base Storage Layer.

To integrate with S3, additional configuration is required. See S3 Access.

Authentication to AWS and to EMR supports the following basic modes:

The authentication mode for your access to EMR can be configured independently from the base authentication mode for AWS, with the following exception:

Authentication mode configuration matrix:

"aws.s3.key"
"aws.s3.secret"

Please apply the following configuration to set the EMR authentication mode:

Steps:

"aws.credentialProvider":"default",
"aws.emr.forceInstanceRole":false,

"aws.credentialProvider":"default",
"aws.emr.forceInstanceRole":true,

"aws.credentialProvider":"instance",

After you have configured S3 to be the base storage layer, you must enable EMR integration.

Steps:

You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

The Designer Cloud Powered by Trifacta platform must be aware of the EMR cluster to which to connection.

Steps:

For more information, see Admin Settings Page.

If you have deployed your EMR cluster on a private sub-net that is accessible outside of AWS, you must enable this property, which permits the extraction of the IP address of the master cluster node through DNS.

Steps:

For EMR, you can configure a set of Spark-related properties to manage the integration and its performance.

"spark": { 
  ...
  "props": { 
    "spark.dynamicAllocation.enabled": "true",
    "spark.shuffle.service.enabled": "true", 
    "spark.executor.instances": "0", 
    "spark.executor.memory": "2048M", 
    "spark.executor.cores": "2",
    "spark.driver.maxResultSize": "0"
  }
  ...
}

spark.dynamicAllocation.enabled

true

spark.shuffle.service.enabled

true

spark.executor.instances

spark.executor.memory

spark.executor.cores

spark.driver.maxResultSize

0

When running jobs against a kerberized EMR cluster, you utilize the Spark-submit method of job submission. You must disable the standard EMR integration.

Steps:

You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

To use Spark-submit, the Spark master must be set to use YARN.

Steps:

You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

For integrating with an EMR cluster with Kerberos, the EMR cluster site XML configuration files must be downloaded from the EMR master node to the Alteryx node.

When integrating with a kerberized EMR cluster, the following Alteryx settings are unused:

By default, a job that is started on EMR cannot be canceled through the application. Optionally, you can enable users to cancel their EMR jobs in progress.

Prerequisites:

Steps:

Please complete the following configuration changes to enable job cancellation on EMR.

For smaller datasets, the platform recommends using the Trifacta Photon running environment.

For larger datasets, if the size information is unavailable, the platform recommends by default that you run the job on the Hadoop cluster. For these jobs, the default publishing action for the job is specified to run on the Hadoop cluster, generating the output format defined by this parameter. Publishing actions, including output format, can always be changed as part of the job specification.

As needed, you can change this default format. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

"webapp.defaultHadoopFileFormat": "csv",

Accepted values: csv, json, avro, pqt

For more information, see Run Job Page.

If you are publishing using Snappy compression for jobs run on an EMR cluster, you may need to perform the following additional configuration.

Steps:

You can set the following parameters as needed:

Steps:

You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

hdfs dfs -rm -r -skipTrash /trifacta/tempfiles

For more information on configuring the platform to integrate with Redshift, see Amazon Redshift Connections.

The Designer Cloud Powered by Trifacta platform can be configured to integrate with multiple master EMR nodes, which are deployed in a highly available environment.

If needed, you can switch to a different EMR cluster through the application. For example, if the original cluster suffers a prolonged outage, you can switch clusters by entering the cluster ID of a new cluster. For more information, see Admin Settings Page.

Batch Job Runner manages jobs executed on the EMR cluster. You can modify aspects of how jobs are executed and how logs are collected. For more information, see Configure Batch Job Runner.

In environments where the EMR cluster is shared with other job-executing applications, you can review and specify the job tag prefix, which is prepended to job identifiers to avoid conflicts with other applications.

Steps: