...

• This feature is supported for 

  D s product
  product ee

   only.
• The 

  D s platform

   must be deployed on an AWS EC2 Instance that is joined to the same domain as the EMR cluster.
• The EMR cluster must be kerberized using the Cross-Realm Trust method. Additional information is below.

Please apply the following configuration to set the EMR authentication mode:

Steps:

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "elasticmapreduce:AddJobFlowSteps",
                "elasticmapreduce:DescribeStep",
                "elasticmapreduce:DescribeCluster",
                "elasticmapreduce:ListInstanceGroups"
            ],
            "Resource": [
                "*"
            ]
        },
       {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::__EMR_LOG_BUCKET__",
                "arn:aws:s3:::__EMR_LOG_BUCKET__/*",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__/*"
            ]
        }

    ]
}

{
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::__EMR_LOG_BUCKET__",
                "arn:aws:s3:::__EMR_LOG_BUCKET__/*",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__/*"
            ]
        }
}

"aws.s3.key"
"aws.s3.secret"

"aws.emr.authMode":  "user",

EMR per-user authentication for the

D s platform

If you have enabled per-user authentication for EMR (aws.emr.authMode=user), you must set the following properties based on the credential provider for your AWS per-user credentials.

Use default credential provider for all

EMR Authentication for the 

D s platform

Depending on the authentication method you used, you must set the following properties.

"aws.credentialProvider":"default",
"aws.emr.forceInstanceRole":false,

"aws.credentialProvider":"default",
"aws.emr.forceInstanceRole":true,

"aws.credentialProvider":"instance",

Configure

D s platform

for EMR

Enable EMR integration

After you have configured S3 to be the base storage layer, you must enable EMR integration.

Steps:

1. Set the following value:

   Code Block
   "webapp.runInEMR": true,

   
   

2. Set the following values:

   Code Block
   "webapp.runWithSparkSubmit": false,

   
   

3. Verify the following property values:

   Code Block
   "webapp.runInTrifactaServer": true, "webapp.runWithSparkSubmit": false, "webapp.runInDataflow": false,

   
   

Apply EMR cluster ID

The 

Steps:

1. D s config
   method admin

2. Under External Service Settings, enter your AWS EMR Cluster ID. Click the Save button below the textbox.

For more information, see Admin Settings Page.

Extract IP address of master node in private sub-net

If you have deployed your EMR cluster on a private sub-net that is accessible outside of AWS, you must enable this property, which permits the extraction of the IP address of the master cluster node through DNS.

Steps:

1. D s config

2. Set the following property to true:

   Code Block
   "emr.extractIPFromDNS": false,

   
   

3. Save your changes and restart the platform.

Configure authentication mode

You can authenticate to the EMR cluster using either of the following authenticate modes:

• System: A single set of credentials are used to connect to EMR. 
• User: Each user has a separate set of credentials.

Steps:

1. D s config

2. Locate the following settings and apply the appropriate values. See the table below:
   
   

   Code Block
   "aws.emr.authMode": "user",

   
   

   Setting	Description
   aws.emr.authMode	Configure the mode to use to authenticate to the EMR cluster: system - In system mode, the specified AWS key and secret combination are used to authenticate to the EMR cluster. These credentials are used for all users. user - In user mode, user configuration is retrieved from the database. Info NOTE: User mode for EMR authentication requires that aws.mode be set to user.

   
   

3. Save your changes.

Configure Spark for EMR

For EMR, you can configure a set of Spark-related properties to manage the integration and its performance.

Configure Spark version

Depending on the version of EMR with which you are integrating, the

Steps:

1. D s config

2. Locate the following:
   
   

   Code Block
   "spark.version": "<SparkVersionForMyEMRVersion>",

   
   

3. Save your changes.

Use vendor libraries

If you are using EMR 5.20 or later (Spark 2.4 or later), you must configure the vendor libraries provided by the cluster. Please set the following parameter.

Steps:

1. D s config

2. Locate the following:

   Code Block
   "spark.useVendorSparkLibraries": true,

   
   

3. Save your changes.

Disable Spark job service

The Spark job service is not used for EMR job execution. Please complete the following to disable it:

Steps:

1. D s config

2. Locate the following and set it to false:

   Code Block
   "spark-job-service.enabled": false,

   
   

3. Locate the following and set it to false:

   Code Block
   "spark-job-service.enableHiveSupport": false,

   
   

4. Save your changes.

Specify YARN queue for Spark jobs

Through the Admin Settings page, you can specify the YARN queue to which to submit your Spark jobs. All Spark jobs from the 

Steps:

1. In platform configuration, locate the following:

   Code Block
   "spark.props.spark.yarn.queue"

   
   

2. Specify the name of the queue. 

3. Save your changes.

Allocation properties

The following properties must be passed from the 

"spark": { 
  ...
  "props": { 
    "spark.dynamicAllocation.enabled": "true",
    "spark.shuffle.service.enabled": "true", 
    "spark.executor.instances": "0", 
    "spark.executor.memory": "2048M", 
    "spark.executor.cores": "2",
    "spark.driver.maxResultSize": "0"
  }
  ...
}

spark.dynamicAllocation.enabled

spark.shuffle.service.enabled

spark.executor.instances

spark.executor.memory

spark.executor.cores

Configure

D s platform

for EMR with Kerberos

Disable standard EMR integration

When running jobs against a kerberized EMR cluster, you utilize the Spark-submit method of job submission. You must disable the standard EMR integration.

Steps:

1. Search for the following setting and set it false:

   Code Block
   "webapp.runInEMR": false,

   
   

2. Set the following value:

   Code Block
   "webapp.runWithSparkSubmit": true,

   
   

3. Disable use of Hive, which is not supported with EMR:

   Code Block
   "spark-job-service.enableHiveSupport": false,

   
   

4. Verify the following property values:

   Code Block
   "webapp.runInTrifactaServer": true, "webapp.runInDataflow": false,

   
   

5. Save your changes.

Enable YARN

To use Spark-submit, the Spark master must be set to use YARN.

Steps:

1. Search for the following setting and set it yarn:

   Code Block
   "spark.master": "yarn",

   
   

2. Save your changes.

Acquire site config files

For integrating with an EMR cluster with Kerberos, the EMR cluster site XML configuration files must be downloaded from the EMR master node to the

1. Download the Hadoop Client Configuration files from the EMR master node. The required files are the following:
   1. core-site.xml
   2. hdfs-site.xml
   3. mapred-site.xml
   4. yarn-site.xml

2. These configuration files must be moved to the 

   D s item
   item deployment

   . By default, these files are in /etc/hadoop/conf:

   Code Block
   sudo cp <location>/*.xml /opt/trifacta/conf/hadoop-site/ sudo chown trifacta:trifacta /opt/trifacta/conf/hadoop-site/*.xml

   
   

3. (Option) If we want to support impersonate, we also need copy *.keytab from the EMR master node under /etc folder to EC2 instance under same folder.

Unused properties for EMR with Kerberos

When integrating with a kerberized EMR cluster, the following

1. External Service Settings: In the Admin Settings page, this section of configuration does not apply to EMR with Kerberos.

2. Unused EMR settings: In the Admin Settings page, the following EMR settings do not apply to EMR with Kerberos:

   Code Block
   aws.emr.tempfilesCleanupAge aws.emr.proxyUser aws.emr.maxLogPollingRetries aws.emr.jobTagPrefix aws.emr.getLogsOnFailure aws.emr.getLogsForAllJobs aws.emr.extractIPFromDNS aws.emr.connectToRmEnabled

   
   

Additional Configuration for EMR

Default Hadoop job results format

For smaller datasets, the platform recommends using the 

For larger datasets, if the size information is unavailable, the platform recommends by default that you run the job on the Hadoop cluster. For these jobs, the default publishing action for the job is specified to run on the Hadoop cluster, generating the output format defined by this parameter. Publishing actions, including output format, can always be changed as part of the job specification. 

As needed, you can change this default format. 

"webapp.defaultHadoopFileFormat": "csv",

Accepted values: csv, json, avro, pqt

For more information, see Run Job Page.

Configure Snappy publication

If you are publishing using Snappy compression for jobs run on an EMR cluster, you may need to perform the following additional configuration.

Steps:

1. SSH into EMR cluster (master) node:

   Code Block
   ssh <EMR master node>

   
   

2. Create tarball of native Hadoop libraries:

   Code Block
   tar -C /usr/lib/hadoop/lib -czvf emr-hadoop-native.tar.gz native

   
   

3. Copy the tarball to the

   D s item
   item EC2

   instance used by the into the /tmp directory:

   Code Block
   scp -p emr-hadoop-native.tar.gz <EC2 instance>:/tmp

   
   

4. SSH to

   D s item
   item EC2

   instance:

   Code Block
   ssh <EC2 instance>

   
   

5. Create path values for libraries:

   Code Block
   sudo -u trifacta mkdir -p /opt/trifacta/services/batch-job-runner/build/libs

   
   

6. Untar the tarball to the

   D s item
   item installation path

   :

   Code Block
   sudo -u trifacta tar -C /opt/trifacta/services/batch-job-runner/build/libs -xzf /tmp/emr-hadoop-native.tar.gz

   
   

7. Verify libhadoop.so* and libsnappy.so* libraries exist and are owned by the

   D s item
   item user

   :

   Code Block
   ls -l /opt/trifacta/services/batch-job-runner/build/libs/native/

   
   

8. Verify that the /tmp directory has the proper permissions for publication. For more information, see Supported File Formats.

9. A platform restart is not required.

Additional parameters

You can set the following parameters as needed:

Steps:

hdfs dfs -rm -r -skipTrash /trifacta/tempfiles

Optional Configuration

Configure for Redshift

For more information on configuring the platform to integrate with Redshift, see Create Redshift Connections.

Switch EMR Cluster

If needed, you can switch to a different EMR cluster through the application. For example, if the original cluster suffers a prolonged outage, you can switch clusters by entering the cluster ID of a new cluster. For more information, see Admin Settings Page.

Configure Batch Job Runner

Batch Job Runner manages jobs executed on the EMR cluster. You can modify aspects of how jobs are executed and how logs are collected. For more information, see Configure Batch Job Runner.

Modify Job Tag Prefix

In environments where the EMR cluster is shared with other job-executing applications, you can review and specify the job tag prefix, which is prepended to job identifiers to avoid conflicts with other applications.

Steps:

1. D s config

2. Locate the following and modify if needed:

   Code Block
   "aws.emr.jobTagPrefix": "TRIFACTA_JOB_",

   
   

3. Save your changes and restart the platform.

...