Create EMR Cluster

Use the following section to set up your EMR cluster for use with the 

• Via AWS EMR UI: This method is assumed in this documentation.
• Via AWS command line interface: For this method, it is assumed that you know the required steps to perform the basic configuration. For custom configuration steps, additional documentation is provided below.

Cluster options

In the Amazon EMR console, click Create Cluster. Click Go to advanced options. Complete the sections listed below.

For more information on setting up your EMR cluster, see http://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html.

Advanced Options

In the Advanced Options screen, please select the following:

• Software Configuration:

  • Release: EMR version to select.

  • Select:
    • Hadoop 2.8.3
    • Hue 3.12.0

    • Ganglia 3.7.2

      Tip
      Tip: Although it is optional, Ganglia is recommended for monitoring cluster performance.

      
      

    • Spark version should be set accordingly. See "Supported Spark Versions" above.

  • Deselect everything else.
• Multiple master nodes (optional):
  • To deploy your EMR cluster in a highly available configuration, select the "Use multiple master nodes" checkbox.

  • When selected, this option creates a total of three master nodes. Two of the nodes are configured as failover options if the first one fails.

    Warning
    Deploying EMR with multiple master nodes may incur additional costs.

    Additional configuration is required. For more information, see "Configure for EMR High Availability" below.

• Edit the software settings:

  • Copy and paste the following into Enter Configuration:

    Code Block
    [ { "Classification": "capacity-scheduler", "Properties": { "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator" } } ]

    
    

  • EMR job cancellation: The following configuration entry is required if you wish to enable EMR job cancellation:
    
    

    Code Block
    { "Classification": "core-site", "Properties": { "hadoop.http.filter.initializers": "org.apache.hadoop.security.HttpCrossOriginFilterInitializer,org.apache.hadoop.http.lib.StaticUserWebFilter" } }

    Additional configuration is required later to enable job cancellation. For more information, see "Enable EMR job cancellation" below.

• Auto-terminate cluster after the last step is completed: Leave this option disabled.

Hardware configuration

General Options

• Cluster name: Provide a descriptive name.
• Logging: Enable logging on the cluster. 

  • S3 folder: Please specify the S3 bucket and path to the logging folder.

    Info
    NOTE: Please verify that this location is read accessible to all users of the platform. See below for details.

    
    

• Debugging: Enable.
• Termination protection: Enable.
• Tags:
  
  • No options required.
• Additional Options:
  • EMRFS consistent view: Do not enable.
  • Custom AMI ID: None.
  • Bootstrap Actions:

    • If you are using a custom credential provider JAR, you must create a bootstrap action. 

      Info
      NOTE: This configuration must be completed before you create the EMR cluster. For more information, see Authentication below.

      
      

Security Options

• EC2 key pair: Please select a key/pair to use if you wish to access EMR nodes via SSH. 

• Permissions: Set to Custom to reduce the scope of permissions. For more information, see "Access Policies" below.

  Info
  NOTE: Default permissions give access to everything in the cluster.

  
  

• Encryption Options
  
  • No requirements.

• EC2 Security Groups:

  • The selected security group for the master node on the cluster must allow TCP traffic from the

    D s item
    item instance

    on port 8088. For more information, see System Ports.

Create cluster and acquire cluster ID

If you performed all of the configuration, including the sections below, you can create the cluster.

Specify cluster roles

The following cluster roles and their permissions are required. For more information on the specifics of these policies, see "Access Policies" below. 

• EMR Role: 
  
  • Read/write access to log bucket
  • Read access to resource bucket
• EC2 instance profile:
  • If using instance mode: 
    • EC2 profile should have read/write access for all users. 
    • EC2 profile should have same permissions as EC2 Edge node role. 
  • Read/write access to log bucket
  • Read access to resource bucket
• Auto-scaling role:
  • Read/write access to log bucket
  • Read access to resource bucket
  • Standard auto-scaling permissions

Authentication

You can use one of two methods for authenticating the EMR cluster's access to S3:

• Role-based IAM authentication (recommended): This method leverages your IAM roles on the EC2 instance. 
• Custom credential provider: This method utilizes a custom credential provider JAR file provided with the platform. This custom credential provider is automatically deployed by the

  D s platform

  during job submission.

Role-based IAM authentication

You can leverage your IAM roles to provide role-based authentication to the S3 buckets.

For more information, see Configure for EC2 Role-Based Authentication.

Custom credential provider

If you are not using IAM roles for access, you can manage access using either of the following:

• AWS key and secret values specified in 

  D s triconf

• AWS user mode

In either scenario, the

Set up S3 Buckets

Bucket setup

You must set up S3 buckets for read and write access. 

For more information, see Enable S3 Access.

Set up EMR resources buckets

On the EMR cluster, all users of the platform must have access to the following locations:

These locations are configured on the 

Access Policies

EC2 instance profile

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "elasticmapreduce:AddJobFlowSteps",
                "elasticmapreduce:DescribeStep",
                "elasticmapreduce:DescribeCluster",
                "elasticmapreduce:ListInstanceGroups",
                "elasticmapreduce:CancelSteps"
            ],
            "Resource": [
                "*"
            ]
        },
       {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::__EMR_LOG_BUCKET__",
                "arn:aws:s3:::__EMR_LOG_BUCKET__/*",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__/*"
            ]
        }

    ]
}

Permissions for EMR high availability

If the

                "elasticmapreduce:listInstances",

Additional configuration is required. See "Configure for EMR High Availability" below.

EMR roles

The following policies should be assigned to the EMR roles listed below for read/write access:

{
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::__EMR_LOG_BUCKET__",
                "arn:aws:s3:::__EMR_LOG_BUCKET__/*",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__/*"
            ]
        }
}

General configuration for 

D s platform

Please complete the following sections to configure the 

Change admin password

As soon as you have installed the software, you should login to the application and change the admin password. The initial admin password is the instanceId for the EC2 instance. For more information, see Change Password.

Verify S3 as base storage layer

EMR integrations requires use of S3 as the base storage layer.

See Set Base Storage Layer.

Set up S3 integration

To integrate with S3, additional configuration is required. See Enable S3 Access.

Configure EMR authentication mode

Authentication to AWS and to EMR supports the following basic modes:

• System: A single set of credentials is used to connect to resources. 
• User: Each user has a separate set of credentials. The user can choose to submit key-secret combinations or role-based authentication.

The authentication mode for your access to EMR can be configured independently from the base authentication mode for AWS, with the following exception:

Authentication mode configuration matrix:

"aws.s3.key"
"aws.s3.secret"

Please apply the following configuration to set the EMR authentication mode:

Steps:

1. D s config

2. Locate the following settings and apply the appropriate values. See the table below:
   
   

   Code Block
   "aws.emr.authMode": "user",

   
   

   Setting

   Description

   aws.emr.authMode

   Configure the mode to use to authenticate to the EMR cluster: system - In system mode, the specified AWS key and secret combination are used to authenticate to the EMR cluster. These credentials are used for all users. user - In user mode, user configuration is retrieved from the database. Info NOTE: User mode for EMR authentication requires that aws.mode be set to user. Additional configuration for EMR is below.

   
   

3. Save your changes.

EMR per-user authentication for the

D s platform

If you have enabled per-user authentication for EMR (aws.emr.authMode=user), you must set the following properties based on the credential provider for your AWS per-user credentials.

"aws.credentialProvider":"default",
"aws.emr.forceInstanceRole":false,

"aws.credentialProvider":"default",
"aws.emr.forceInstanceRole":true,

"aws.credentialProvider":"instance",

Configure

D s platform

for EMR

Enable EMR integration

After you have configured S3 to be the base storage layer, you must enable EMR integration.

Steps:

1. Set the following value:

   Code Block
   "webapp.runInEMR": true,

   
   

2. Set the following values:

   Code Block
   "webapp.runWithSparkSubmit": false,

   
   

3. Verify the following property values:

   Code Block
   "webapp.runWithSparkSubmit": false, "webapp.runInDataflow": false,

   
   

4. Save your changes and restart the platform.

5. To enable

   D s photon

   , an in-memory running environment for small- to medium-sized jobs, please do the following:

   1. In the

      D s webapp

      , navigate to User menu > Admin console > Workspace settings.

   2. In the Workspace Settings page, set Photon execution to Enabled.

   3. D s photon

      is available for job execution when you next log in to the

      D s webapp

      .

Apply EMR cluster ID

The 

Steps:

1. D s config
   method admin

2. Under External Service Settings, enter your AWS EMR Cluster ID. Click the Save button below the textbox.

For more information, see Admin Settings Page.

Extract IP address of master node in private sub-net

If you have deployed your EMR cluster on a private sub-net that is accessible outside of AWS, you must enable this property, which permits the extraction of the IP address of the master cluster node through DNS.

Steps:

1. D s config

2. Set the following property to true:

   Code Block
   "emr.extractIPFromDNS": false,

   
   

3. Save your changes and restart the platform.

Configure Spark for EMR

For EMR, you can configure a set of Spark-related properties to manage the integration and its performance.

Configure Spark version

Depending on the version of EMR with which you are integrating, the

Steps:

1. D s config

2. Locate the following:
   
   

   Code Block
   "spark.version": "<SparkVersionForMyEMRVersion>",

   
   

3. This setting is ignored for EMR: spark.useVendorSparkLibraries
4. Save your changes.

Disable Spark job service

The Spark job service is not used for EMR job execution. Please complete the following to disable it:

Steps:

1. D s config

2. Locate the following and set it to false:

   Code Block
   "spark-job-service.enabled": false,

   
   

3. Locate the following and set it to false:

   Code Block
   "spark-job-service.enableHiveSupport": false,

   
   

4. Save your changes.

Specify YARN queue for Spark jobs

Through the Admin Settings page, you can specify the YARN queue to which to submit your Spark jobs. All Spark jobs from the 

Steps:

1. In platform configuration, locate the following:

   Code Block
   "spark.props.spark.yarn.queue"

   
   

2. Specify the name of the queue. 

3. Save your changes.

Allocation properties

The following properties must be passed from the 

"spark": { 
  ...
  "props": { 
    "spark.dynamicAllocation.enabled": "true",
    "spark.shuffle.service.enabled": "true", 
    "spark.executor.instances": "0", 
    "spark.executor.memory": "2048M", 
    "spark.executor.cores": "2",
    "spark.driver.maxResultSize": "0"
  }
  ...
}

spark.dynamicAllocation.enabled

spark.shuffle.service.enabled

spark.executor.instances

spark.executor.memory

spark.executor.cores

Combine transform and profiling for Spark jobs

When profiling is enabled for a Spark job, the transform and profiling tasks are combined by default. As needed, you can separate these two tasks. Publishing behaviors vary depending on the approach. For more information, see Configure for Spark.

Configure

D s platform

for EMR with Kerberos

Disable standard EMR integration

When running jobs against a kerberized EMR cluster, you utilize the Spark-submit method of job submission. You must disable the standard EMR integration.

Steps:

1. Search for the following setting and set it false:

   Code Block
   "webapp.runInEMR": false,

   
   

2. Set the following value:

   Code Block
   "webapp.runWithSparkSubmit": true,

   
   

3. Disable use of Hive, which is not supported with EMR:

   Code Block
   "spark-job-service.enableHiveSupport": false,

   
   

4. Verify the following property value:

   Code Block
   "webapp.runInDataflow": false,

   
   

5. Save your changes.

Enable YARN

To use Spark-submit, the Spark master must be set to use YARN.

Steps:

1. Search for the following setting and set it yarn:

   Code Block
   "spark.master": "yarn",

   
   

2. Save your changes.

Acquire site config files

For integrating with an EMR cluster with Kerberos, the EMR cluster site XML configuration files must be downloaded from the EMR master node to the

1. Download the Hadoop Client Configuration files from the EMR master node. The required files are the following:
   1. core-site.xml
   2. hdfs-site.xml
   3. mapred-site.xml
   4. yarn-site.xml

2. These configuration files must be moved to the 

   D s item
   item deployment

   . By default, these files are in /etc/hadoop/conf:

   Code Block
   sudo cp <location>/*.xml /opt/trifacta/conf/hadoop-site/ sudo chown trifacta:trifacta /opt/trifacta/conf/hadoop-site/*.xml

   
   

3. (Option) If we want to support impersonate, we also need copy *.keytab from the EMR master node under /etc folder to EC2 instance under same folder.

Unused properties for EMR with Kerberos

When integrating with a kerberized EMR cluster, the following

1. External Service Settings: In the Admin Settings page, this section of configuration does not apply to EMR with Kerberos.

2. Unused EMR settings: In the Admin Settings page, the following EMR settings do not apply to EMR with Kerberos:

   Code Block
   aws.emr.tempfilesCleanupAge aws.emr.proxyUser aws.emr.maxLogPollingRetries aws.emr.jobTagPrefix aws.emr.getLogsOnFailure aws.emr.getLogsForAllJobs aws.emr.extractIPFromDNS aws.emr.connectToRmEnabled

   
   

Additional Configuration for EMR

Enable EMR job cancellation

By default, a job that is started on EMR cannot be canceled through the application. Optionally, you can enable users to cancel their EMR jobs in progress.

Pre-requisites:

1. The following permission must be added to the IAM role that is used to interact with EMR:

   Code Block
   elasticmapreduce:CancelSteps

   For more information, See "EC2 instance profile" above.

2. You should add an additional software setting to the cluster definition through the EMR console. For more information, see "Advanced Options" above.

Steps:

Please complete the following configuration changes to enable job cancellation on EMR.

1. D s config

2. Please locate the following parameter and verify that it has been set to true:

   Info
   NOTE: Enabled by default, this parameter is optional for basic EMR connectivity. This parameter must be enabled for EMR job cancellation.

   
   

   Code Block
   "aws.emr.connectToRmEnabled": true,

   
   

3. Please verify that the

   D s node

   can connect to the EMR master node on port 8088. For more information, see "Create EMR Cluster" above.

4. Please locate the following parameter and verify that it is set to true:

   Code Block
   "aws.emr.cancelEnabled": true,

   
   

5. Save your changes and restart the platform.
6. To verify, launch an EMR job. In the Flow View context menu or in the Jobs page, you should see a Cancel job option.

Default Hadoop job results format

For smaller datasets, the platform recommends using the 

For larger datasets, if the size information is unavailable, the platform recommends by default that you run the job on the Hadoop cluster. For these jobs, the default publishing action for the job is specified to run on the Hadoop cluster, generating the output format defined by this parameter. Publishing actions, including output format, can always be changed as part of the job specification. 

As needed, you can change this default format. 

"webapp.defaultHadoopFileFormat": "csv",

Accepted values: csv, json, avro, pqt

For more information, see Run Job Page.

Configure Snappy publication

If you are publishing using Snappy compression for jobs run on an EMR cluster, you may need to perform the following additional configuration.

Steps:

1. SSH into EMR cluster (master) node:

   Code Block
   ssh <EMR master node>

   
   

2. Create tarball of native Hadoop libraries:

   Code Block
   tar -C /usr/lib/hadoop/lib -czvf emr-hadoop-native.tar.gz native

   
   

3. Copy the tarball to the

   D s item
   item EC2

   instance used by the into the /tmp directory:

   Code Block
   scp -p emr-hadoop-native.tar.gz <EC2 instance>:/tmp

   
   

4. SSH to

   D s item
   item EC2

   instance:

   Code Block
   ssh <EC2 instance>

   
   

5. Create path values for libraries:

   Code Block
   sudo -u trifacta mkdir -p /opt/trifacta/services/batch-job-runner/build/libs

   
   

6. Untar the tarball to the

   D s item
   item installation path

   :

   Code Block
   sudo -u trifacta tar -C /opt/trifacta/services/batch-job-runner/build/libs -xzf /tmp/emr-hadoop-native.tar.gz

   
   

7. Verify libhadoop.so* and libsnappy.so* libraries exist and are owned by the

   D s item
   item user

   :

   Code Block
   ls -l /opt/trifacta/services/batch-job-runner/build/libs/native/

   
   

8. Verify that the /tmp directory has the proper permissions for publication. For more information, see Supported File Formats.

9. A platform restart is not required.

Additional parameters

You can set the following parameters as needed:

Steps:

hdfs dfs -rm -r -skipTrash /trifacta/tempfiles

Optional Configuration

Configure for Redshift

For more information on configuring the platform to integrate with Redshift, see Create Redshift Connections.

Configure for EMR high availability

The

Versions

Integration with EMR high availability is supported for EMR 5.23.0 and later.

Permissions

An additional permission must be added to the ARN used to access EMR. For more information, see "Permissions for EMR high availability" above.

For more information, see https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-ha-launch.html.

Create cluster

When you create the cluster, you must select the "Use multiple master nodes to improve cluster availability" checkbox. For more information, see "Create Clusters" above.

Switch EMR Cluster

If needed, you can switch to a different EMR cluster through the application. For example, if the original cluster suffers a prolonged outage, you can switch clusters by entering the cluster ID of a new cluster. For more information, see Admin Settings Page.

Configure Batch Job Runner

Batch Job Runner manages jobs executed on the EMR cluster. You can modify aspects of how jobs are executed and how logs are collected. For more information, see Configure Batch Job Runner.

Modify Job Tag Prefix

In environments where the EMR cluster is shared with other job-executing applications, you can review and specify the job tag prefix, which is prepended to job identifiers to avoid conflicts with other applications.

Steps:

1. D s config

2. Locate the following and modify if needed:

   Code Block
   "aws.emr.jobTagPrefix": "TRIFACTA_JOB_",

   
   

3. Save your changes and restart the platform.