Configure for EMR

Set up EMR Cluster

Use the following section to set up your EMR cluster for use with the Designer Cloud Powered by Trifacta platform.

• Via AWS EMR UI: This method is assumed in this documentation.
• Via AWS command line interface: For this method, it is assumed that you know the required steps to perform the basic configuration. For custom configuration steps, additional documentation is provided below.

Cluster options

In the Amazon EMR console, click Create Cluster. Click Go to advanced options. Complete the sections listed below.

For more information on setting up your EMR cluster, see http://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html.

Advanced Options

In the Advanced Options screen, please select the following:

• Software Configuration:

  • Release: EMR 5.6 - 5.15

  • Select:
    • Hadoop 2.7.3
    • Hue 3.12.0

    • Ganglia 3.7.2

      Tip: Although it is optional, Ganglia is recommended for monitoring cluster performance.

    • Spark:
      • For EMR 5.6 - EMR 5.7: Spark 2.1.1

      • For EMR 5.8 - EMR 5.12.1: Spark 2.2.x

      • For EMR 5.13 - EMR 5.15: Spark 2.3.0

        NOTE: You must apply the Spark version number in the spark.version property in Admin Settings. Additional configuration is required. See Configure for Spark.

  • Deselect everything else.
• Edit the software settings:

  • Copy and paste the following into Enter Configuration:

    [
     {
     "Classification": "capacity-scheduler",
     "Properties": {
     "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator"
     }
     }
    ]

• Auto-terminate cluster after the last step is completed: Disable this option.

Hardware configuration

General Options

• Cluster name: Provide a descriptive name.
• Logging: Enable logging on the cluster. 

  • S3 folder: Please specify the S3 bucket and path to the logging folder.

    NOTE: Please verify that this location is read accessible to all users of the platform. See below for details.

• Debugging: Enable.
• Termination protection: Enable.
• Scale down behavior: Terminate at instance hour.
• Tags:
  • No optiions required.
• Additional Options:
  • EMRFS consistent view: You should enable this setting. Doing so may incur additional costs. For more information, see EMRFS consistent view is recommended below.
  • Custom AMI ID: None.
  • Bootstrap Actions:

    • If you are using the default credential provider, you must create a bootstrap action. 

      NOTE: This configuration must be completed before you create the EMR cluster. For more information, see Authentication below.

Security Options

• EC2 key pair: Please select a key/pair to use if you wish to access EMR nodes via SSH. 

• Permissions: Set to Custom to reduce the scope of permissions. For more information, see EMR cluster policies below.

  NOTE: Default permissions give access to everything in the cluster.

• Encryption Options
  
  • No requirements.

• EC2 Security Groups:

  • The selected security group for the master node on the cluster must allow traffic on port 8088. For more information, see System Ports .

Create cluster and acquire cluster ID

If you performed all of the configuration, including the sections below, you can create the cluster.

Specify cluster roles

The following cluster roles and their permissions are required. For more information on the specifics of these policies, see EMR cluster policies. 

• EMR Role: 
  
  • Read/write access to log bucket
  • Read access to resource bucket
• EC2 instance profile:
  • If using instance mode: 
    • EC2 profile should have read/write access for all users. 
    • EC2 profile should have same permissions as EC2 Edge node role. 
  • Read/write access to log bucket
  • Read access to resource bucket
• Auto-scaling role:
  • Read/write access to log bucket
  • Read access to resource bucket
  • Standard auto-scaling permissions

Authentication

You can use one of two methods for authenticating the EMR cluster:

• Role-based IAM authentication (recommended): This method leverages your IAM roles on the EC2 instance. 
• Custom credential provider JAR file: This method utilizes a JAR file provided with the platform. This JAR file must be deployed to all nodes on the EMR cluster through a bootstrap action script.

Role-based IAM authentication

You can leverage your IAM roles to provide role-based authentication to the S3 buckets.

For more information, see Configure for EC2 Role-Based Authentication.

Specify the custom credential provider JAR file

If you are not using IAM roles for access, you can manage access using either of the following:

• AWS key and secret values specified in trifacta-conf.json
• AWS user mode

In either scenario, you must use the custom credential provider JAR provided in the installation. This JAR file must be available to all nodes of the EMR cluster.

After you have installed the platform and configured the S3 buckets, please complete the following steps to deploy this JAR file.

Steps:

1. From the installation of the Designer Cloud Powered by Trifacta platform, retrieve the following file:

   [TRIFACTA_INSTALL_DIR]/aws/emr/build/libs/trifacta-aws-emr-credential-provider[TIMESTAMP].jar

   NOTE: Do not remove the timestamp value from the filename. This information is useful for support purposes.

2. Upload this JAR file to an S3 bucket location where the EMR cluster can access it:

   1. Via AWS Console S3 UI: See http://docs.aws.amazon.com/cli/latest/reference/s3/index.html.

   2. Via AWS command line:

      aws s3 cp trifacta-aws-emr-credential-provider[TIMESTAMP].jar s3://<YOUR-BUCKET>/

3. Create a bootstrap action script named configure_emrfs_lib.sh. The contents must be the following:

   sudo aws s3 cp s3://<YOUR-BUCKET>/trifacta-aws-emr-credential-provider[TIMESTAMP].jar  /usr/share/aws/emr/emrfs/auxlib/

4. This script must be uploaded into S3 in a location that can be accessed from the EMR cluster. Retain the full path to this location.
5. Add bootstrap action to EMR cluster configuration.

   1. Via AWS Console S3 UI: Create the bootstrap action to point to the script you uploaded on S3.

      
      

   2. Via AWS command line: 
      1. Upload the configure_emrfs_lib.sh file to the accessible S3 bucket.

      2. In the command line cluster creation script, add a custom bootstrap action, such as the following:

         --bootstrap-actions '[
         {"Path":"s3://<YOUR-BUCKET>/configure_emrfs_lib.sh","Name":"Custom action"}
         ]'

When the EMR cluster is launched with the above custom bootstrap action, the cluster does one of the following:

• Interacts with S3 using the credentials specified intrifacta-conf.json
• if aws.mode = user, then the credentials registered by the user are used.

For more information about AWSCredentialsProvider for EMRFS please see:

• http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-credentialsprovider.html
• https://aws.amazon.com/blogs/big-data/securely-analyze-data-from-another-aws-account-with-emrfs/

EMRFS consistent view is recommended

Although it is not required, you should enable the consistent view feature for EMRFS on your cluster.

During job execution, including profiling jobs, on EMR, the Designer Cloud Powered by Trifacta platform writes files in rapid succession, and these files are quickly read back from storage for further processing. However, Amazon S3 does not provide a guarantee of a consistent file listing until a later time.

To ensure that the Designer Cloud Powered by Trifacta platform does not begin reading back an incomplete set of files, you should enable EMRFS consistent view. 

For more information on EMRFS consistent view, see http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-consistent-view.html.

DynamoDB

Amazon's DynamoDB is automatically enabled to store metadata for EMRFS consistent view.