Create EMR Cluster

Use the following section to set up your EMR cluster for use with the 

• Via AWS EMR UI: This method is assumed in this documentation.
• Via AWS command line interface: For this method, it is assumed that you know the required steps to perform the basic configuration. For custom configuration steps, additional documentation is provided below.

Cluster options

In the Amazon EMR console, click Create Cluster. Click Go to advanced options. Complete the sections listed below.

For more information on setting up your EMR cluster, see http://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html.

Advanced Options

In the Advanced Options screen, please select the following:

• Software Configuration:

  • Release: EMR version to select.

  • Select:
    • Hadoop 2.8.3
    • Hue 3.12.0

    • Ganglia 3.7.2

      Tip
      Tip: Although it is optional, Ganglia is recommended for monitoring cluster performance.

      
      

    • Spark version should be set accordingly. See "Supported Spark Versions" above.

  • Deselect everything else.
• Multiple master nodes (optional):
  • To deploy your EMR cluster in a highly available configuration, select the "Use multiple master nodes" checkbox.

  • When selected, this option creates a total of three master nodes. Two of the nodes are configured as failover options if the first one fails.

    Warning
    Deploying EMR with multiple master nodes may incur additional costs.

    Additional configuration is required. For more information, see "Configure for EMR High Availability" below.

• Edit the software settings:

  • Copy and paste the following into Enter Configuration:

    Code Block
    [ { "Classification": "capacity-scheduler", "Properties": { "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator" } } ]

    
    

  • EMR job cancellation: The following configuration entry is required if you wish to enable EMR job cancellation:
    
    

    Code Block
    { "Classification": "core-site", "Properties": { "hadoop.http.filter.initializers": "org.apache.hadoop.security.HttpCrossOriginFilterInitializer,org.apache.hadoop.http.lib.StaticUserWebFilter" } }

    Additional configuration is required later to enable job cancellation. For more information, see "Enable EMR job cancellation" below.

• Auto-terminate cluster after the last step is completed: Leave this option disabled.

Hardware configuration

General Options

• Cluster name: Provide a descriptive name.
• Logging: Enable logging on the cluster. 

  • S3 folder: Please specify the S3 bucket and path to the logging folder.

    Info
    NOTE: Please verify that this location is read accessible to all users of the platform. See below for details.

    
    

• Debugging: Enable.
• Termination protection: Enable.
• Tags:
  
  • No options required.
• Additional Options:
  • EMRFS consistent view: Do not enable.
  • Custom AMI ID: None.
  • Bootstrap Actions:

    • If you are using a custom credential provider JAR, you must create a bootstrap action. 

      Info
      NOTE: This configuration must be completed before you create the EMR cluster. For more information, see Authentication below.

      
      

Security Options

• EC2 key pair: Please select a key/pair to use if you wish to access EMR nodes via SSH. 

• Permissions: Set to Custom to reduce the scope of permissions. For more information, see "Access Policies" below.

  Info
  NOTE: Default permissions give access to everything in the cluster.

  
  

• Encryption Options
  
  • No requirements.

• EC2 Security Groups:

  • The selected security group for the master node on the cluster must allow TCP traffic from the

    D s item
    item instance

    on port 8088. For more information, see System Ports.

Create cluster and acquire cluster ID

If you performed all of the configuration, including the sections below, you can create the cluster.

Specify cluster roles

The following cluster roles and their permissions are required. For more information on the specifics of these policies, see "Access Policies" below. 

• EMR Role: 
  
  • Read/write access to log bucket
  • Read access to resource bucket
• EC2 instance profile:
  • If using instance mode: 
    • EC2 profile should have read/write access for all users. 
    • EC2 profile should have same permissions as EC2 Edge node role. 
  • Read/write access to log bucket
  • Read access to resource bucket
• Auto-scaling role:
  • Read/write access to log bucket
  • Read access to resource bucket
  • Standard auto-scaling permissions

Authentication

You can use one of two methods for authenticating the EMR cluster's access to S3:

• Role-based IAM authentication (recommended): This method leverages your IAM roles on the EC2 instance. 
• Custom credential provider: This method utilizes a custom credential provider JAR file provided with the platform. This custom credential provider is automatically deployed by the

  D s platform

  during job submission.

Role-based IAM authentication

You can leverage your IAM roles to provide role-based authentication to the S3 buckets.

For more information, see Configure for EC2 Role-Based Authentication.

Custom credential provider

If you are not using IAM roles for access, you can manage access using either of the following:

• AWS key and secret values specified in 

  D s triconf

• AWS user mode

In either scenario, the

Set up S3 Buckets

Bucket setup

You must set up S3 buckets for read and write access. 

For more information, see S3 Access.

Set up EMR resources buckets

On the EMR cluster, all users of the platform must have access to the following locations:

These locations are configured on the 

Access Policies

EC2 instance profile

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "elasticmapreduce:AddJobFlowSteps",
                "elasticmapreduce:DescribeStep",
                "elasticmapreduce:DescribeCluster",
                "elasticmapreduce:ListInstanceGroups",
                "elasticmapreduce:CancelSteps"
            ],
            "Resource": [
                "*"
            ]
        },
       {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::__EMR_LOG_BUCKET__",
                "arn:aws:s3:::__EMR_LOG_BUCKET__/*",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__/*"
            ]
        }

    ]
}

Permissions for EMR high availability

If the

                "elasticmapreduce:listInstances",

Additional configuration is required. See "Configure for EMR High Availability" below.

EMR roles

The following policies should be assigned to the EMR roles listed below for read/write access:

{
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::__EMR_LOG_BUCKET__",
                "arn:aws:s3:::__EMR_LOG_BUCKET__/*",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__/*"
            ]
        }
}

General configuration for 

D s platform

Please complete the following sections to configure the 

Change admin password

As soon as you have installed the software, you should login to the application and change the admin password. The initial admin password is the instanceId for the EC2 instance. For more information, see Change Password.

Verify S3 as base storage layer

EMR integrations requires use of S3 as the base storage layer.

See Set Base Storage Layer.

Set up S3 integration

To integrate with S3, additional configuration is required. See S3 Access.

Configure EMR authentication mode

Authentication to AWS and to EMR supports the following basic modes:

• System: A single set of credentials is used to connect to resources. 
• User: Each user has a separate set of credentials. The user can choose to submit key-secret combinations or role-based authentication.

The authentication mode for your access to EMR can be configured independently from the base authentication mode for AWS, with the following exception:

Authentication mode configuration matrix:

"aws.s3.key"
"aws.s3.secret"

Please apply the following configuration to set the EMR authentication mode:

Steps:

1. D s config

2. Locate the following settings and apply the appropriate values. See the table below:
   
   

   Code Block
   "aws.emr.authMode": "user",

   
   

   Setting

   Description

   aws.emr.authMode

   Configure the mode to use to authenticate to the EMR cluster: system - In system mode, the specified AWS key and secret combination are used to authenticate to the EMR cluster. These credentials are used for all users. user - In user mode, user configuration is retrieved from the database. Info NOTE: User mode for EMR authentication requires that aws.mode be set to user. Additional configuration for EMR is below.

   
   

3. Save your changes.

EMR per-user authentication for the

D s platform

If you have enabled per-user authentication for EMR (aws.emr.authMode=user), you must set the following properties based on the credential provider for your AWS per-user credentials.

"aws.credentialProvider":"default",
"aws.emr.forceInstanceRole":false,

"aws.credentialProvider":"default",
"aws.emr.forceInstanceRole":true,

"aws.credentialProvider":"instance",

Configure

D s platform

for EMR

Enable EMR integration

After you have configured S3 to be the base storage layer, you must enable EMR integration.

Steps:

1. Set the following value:

   Code Block
   "webapp.runInEMR": true,

   
   

2. Set the following values:

   Code Block
   "webapp.runWithSparkSubmit": false,

   
   

3. Verify the following property values:

   Code Block
   "webapp.runWithSparkSubmit": false, "webapp.runInDataflow": false,

   
   

4. Save your changes and restart the platform.

5. To enable

   D s photon

   , an in-memory running environment for small- to medium-sized jobs, please do the following:

   1. In the

      D s webapp

      , navigate to User menu > Admin console > Workspace settings.

   2. In the Workspace Settings page, set Photon execution to Enabled.

   3. D s photon

      is available for job execution when you next log in to the

      D s webapp

      .

Apply EMR cluster ID

The 

Steps:

1. D s config
   method admin

2. Under External Service Settings, enter your AWS EMR Cluster ID. Click the Save button below the textbox.

For more information, see Admin Settings Page.

Extract IP address of master node in private sub-net

If you have deployed your EMR cluster on a private sub-net that is accessible outside of AWS, you must enable this property, which permits the extraction of the IP address of the master cluster node through DNS.

Steps:

1. D s config

2. Set the following property to true:

   Code Block
   "emr.extractIPFromDNS": false,

   
   

3. Save your changes and restart the platform.

Configure Spark for EMR

For EMR, you can configure a set of Spark-related properties to manage the integration and its performance.

Configure Spark version

Depending on the version of EMR with which you are integrating, the

Steps:

1. D s config

2. Locate the following:
   
   

   Code Block
   "spark.version": "<SparkVersionForMyEMRVersion>",

   
   

3. This setting is ignored for EMR: spark.useVendorSparkLibraries
4. Save your changes.

Disable Spark job service

The Spark job service is not used for EMR job execution. Please complete the following to disable it:

Steps:

1. D s config

2. Locate the following and set it to false:

   Code Block
   "spark-job-service.enabled": false,

   
   

3. Locate the following and set it to false:

   Code Block
   "spark-job-service.enableHiveSupport": false,

   
   

4. Save your changes.

Specify YARN queue for Spark jobs

Through the Admin Settings page, you can specify the YARN queue to which to submit your Spark jobs. All Spark jobs from the 

Steps:

1. In platform configuration, locate the following:

   Code Block
   "spark.props.spark.yarn.queue"

   
   

2. Specify the name of the queue. 

3. Save your changes.

Allocation properties

The following properties must be passed from the 

"spark": { 
  ...
  "props": { 
    "spark.dynamicAllocation.enabled": "true",
    "spark.shuffle.service.enabled": "true", 
    "spark.executor.instances": "0", 
    "spark.executor.memory": "2048M", 
    "spark.executor.cores": "2",
    "spark.driver.maxResultSize": "0"
  }
  ...
}

spark.dynamicAllocation.enabled

spark.shuffle.service.enabled

spark.executor.instances