Page tree

 

Contents:


You can configure your instance of the Trifacta platform to integrate with Amazon Elastic MapReduce (EMR), a highly scalable Hadoop-based execution environment. 

NOTE: This section applies only to installations of Trifacta Wrangler Enterprise where a license key file has been acquired from Trifacta and applied to the platform.

  • Amazon EMR (Elastic MapReduce) provides a managed Hadoop framework that makes it easy, fast, and cost-effective to process vast amounts of data across dynamically scalable Amazon EC2 instances. For more information on EMR, see http://docs.aws.amazon.com/cli/latest/reference/emr/.

This section can be used to integrate with the following cluster deployments:

  • EMR: default
  • EMR with Kerberos: a separate configuration path is documented in this section

Supported Versions

This section outlines how to create a new EMR cluster and integrate the Trifacta platform with it. The platform can be integrated with existing EMR clusters.

Supported Versions: EMR 5.13 - EMR 5.29.0

NOTE: EMR 5.28.0 is not supported, due to Spark compatibility issues. Please use 5.28.1 or later.

NOTE: EMR 5.20 - EMR 5.29 requires Spark 2.4. For more information, see Configure for Spark.

Supported Spark Versions

Depending on the version of EMR in use, you must configure the Trifacta platform to use the appropriate version of Spark. Please note the appropriate configuration settings below for later use.

NOTE: The version of Spark to use for the platform is defined in the spark.version property. This configuration step is covered later.

EMR versionsSpark versionAdditional configuration and notes
EMR 5.13 - EMR 5.19
"spark.version": "2.3.0",

EMR 5.20 - EMR 5.29
"spark.version": "2.4.4",

For EMR 5.20 and later, please set the following property value:

"spark.useVendorSparkLibraries": true,


Limitations

NOTE: Job cancellation is not supported on an EMR cluster.

  •  The Trifacta platform must be installed on AWS.

Limitations for Kerberos

If you are integrating with a kerberized EMR cluster, the following additional limitations apply:

  • This feature is supported for  Trifacta Wrangler Enterprise only.
  • The Trifacta platform must be deployed on an AWS EC2 Instance that is joined to the same domain as the EMR cluster.
  • The EMR cluster must be kerberized using the Cross-Realm Trust method. Additional information is below.

 

Create EMR Cluster

Use the following section to set up your EMR cluster for use with the Trifacta platform.

  • Via AWS EMR UI: This method is assumed in this documentation.
  • Via AWS command line interface: For this method, it is assumed that you know the required steps to perform the basic configuration. For custom configuration steps, additional documentation is provided below.

NOTE: If you are integrating with a kerberized EMR cluster, the cluster must be kerberized using the Cross-Realm Trust method. The KDC on the EMR cluster must establish a cross-realm trust with the external KDC. No other Kerberos method is supported.

For more information, see https://docs.amazonaws.cn/en_us/emr/latest/ManagementGuide/emr-kerberos-options.html.


NOTE: It is recommended that you set up your cluster for exclusive use by the Trifacta platform.

Cluster options

In the Amazon EMR console, click Create Cluster. Click Go to advanced options. Complete the sections listed below.

NOTE: Please be sure to read all of the cluster options before setting up your EMR cluster.

NOTE: Please perform your configuration through the Advanced Options workflow.

For more information on setting up your EMR cluster, see http://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html.

Advanced Options

In the Advanced Options screen, please select the following:

  • Software Configuration:

    • Release: EMR version to select.

    • Select:
      • Hadoop 2.8.3
      • Hue 3.12.0
      • Ganglia 3.7.2

        Tip: Although it is optional, Ganglia is recommended for monitoring cluster performance.

      • Spark version should be set accordingly. See "Supported Spark Versions" above.

    • Deselect everything else.
  • Edit the software settings:
    • Copy and paste the following into Enter Configuration:

      [
       {
       "Classification": "capacity-scheduler",
       "Properties": {
       "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator"
       }
       }
      ]
  • Auto-terminate cluster after the last step is completed: Leave this option disabled.

Hardware configuration

NOTE: Please apply the sizing information for your EMR cluster that was recommended for you. If you have not done so, please contact your Trifacta representative.

General Options

  • Cluster name: Provide a descriptive name.
  • Logging: Enable logging on the cluster. 
    • S3 folder: Please specify the S3 bucket and path to the logging folder.

      NOTE: Please verify that this location is read accessible to all users of the platform. See below for details.

  • Debugging: Enable.
  • Termination protection: Enable.
  • Tags:
    • No options required.
  • Additional Options:
    • EMRFS consistent view: Do not enable. The platform can generate its own job output manifests. See Enable S3 Access.
    • Custom AMI ID: None.
    • Bootstrap Actions:
      • If you are using a custom credential provider JAR, you must create a bootstrap action. 

        NOTE: This configuration must be completed before you create the EMR cluster. For more information, see Authentication below.

Security Options

  • EC2 key pair: Please select a key/pair to use if you wish to access EMR nodes via SSH. 
  • Permissions: Set to Custom to reduce the scope of permissions. For more information, see EMR cluster policies below.

    NOTE: Default permissions give access to everything in the cluster.

  • Encryption Options
    • No requirements.
  • EC2 Security Groups:

    • The selected security group for the master node on the cluster must allow TCP traffic from the Trifacta instance on port 8088. For more information, see System Ports.

Create cluster and acquire cluster ID

If you performed all of the configuration, including the sections below, you can create the cluster.

NOTE: You must acquire your EMR cluster ID for use in configuration of the Trifacta platform.

Specify cluster roles

The following cluster roles and their permissions are required. For more information on the specifics of these policies, see EMR cluster policies

  • EMR Role: 
    • Read/write access to log bucket
    • Read access to resource bucket
  • EC2 instance profile:
    • If using instance mode: 
      • EC2 profile should have read/write access for all users. 
      • EC2 profile should have same permissions as EC2 Edge node role. 
    • Read/write access to log bucket
    • Read access to resource bucket
  • Auto-scaling role:
    • Read/write access to log bucket
    • Read access to resource bucket
    • Standard auto-scaling permissions

Authentication

You can use one of two methods for authenticating the EMR cluster:

  • Role-based IAM authentication (recommended): This method leverages your IAM roles on the EC2 instance. 
  • Custom credential provider JAR file: This method utilizes a JAR file provided with the platform. This JAR file must be deployed to all nodes on the EMR cluster through a bootstrap action script.

Role-based IAM authentication

You can leverage your IAM roles to provide role-based authentication to the S3 buckets.

NOTE: The IAM role that is assigned to the EMR cluster and to the EC2 instances on the cluster must have access to the data of all users on S3.


For more information, see Configure for EC2 Role-Based Authentication.

Specify the custom credential provider JAR file

If you are not using IAM roles for access, you can manage access using either of the following:

  • AWS key and secret values specified in  trifacta-conf.json
  • AWS user mode

In either scenario, you must use the custom credential provider JAR provided in the installation. This JAR file must be available to all nodes of the EMR cluster.

After you have installed the platform and configured the S3 buckets, please complete the following steps to deploy this JAR file.

NOTE: These steps must be completed before you create the EMR cluster.

NOTE: This section applies if you are using the default credential provider mechanism for AWS and are not using the IAM instance-based role authentication mechanism.

 

Steps:

  1. From the installation of the Trifacta platform, retrieve the following file:

    [TRIFACTA_INSTALL_DIR]/aws/credential-provider/build/libs/trifacta-aws-emr-credential-provider.jar
  2. Upload this JAR file to an S3 bucket location where the EMR cluster can access it:

    1. Via AWS Console S3 UI: See http://docs.aws.amazon.com/cli/latest/reference/s3/index.html.
    2. Via AWS command line:

      aws s3 cp trifacta-aws-emr-credential-provider.jar s3://<YOUR-BUCKET>/
  3. Create a bootstrap action script named configure_emrfs_lib.sh. The contents must be the following:

    sudo aws s3 cp s3://<YOUR-BUCKET>/trifacta-aws-emr-credential-provider.jar  /usr/share/aws/emr/emrfs/auxlib/
  4. This script must be uploaded into S3 in a location that can be accessed from the EMR cluster. Retain the full path to this location.
  5. Add bootstrap action to EMR cluster configuration.
    1. Via AWS Console S3 UI: Create the bootstrap action to point to the script you uploaded on S3.

       

    2. Via AWS command line: 
      1. Upload the configure_emrfs_lib.sh file to the accessible S3 bucket.
      2. In the command line cluster creation script, add a custom bootstrap action, such as the following:

        --bootstrap-actions '[
        {"Path":"s3://<YOUR-BUCKET>/configure_emrfs_lib.sh","Name":"Custom action"}
        ]'

When the EMR cluster is launched with the above custom bootstrap action, the cluster does one of the following:

  • Interacts with S3 using the credentials specified in trifacta-conf.json
  • if aws.mode = user, then the credentials registered by the user are used.

For more information about AWSCredentialsProvider for EMRFS please see:

Set up S3 Buckets

Bucket setup

You must set up S3 buckets for read and write access. 

NOTE: Within the Trifacta platform, you must enable use of S3 as the default storage layer. This configuration is described later.

For more information, see Enable S3 Access.

Set up EMR resources buckets

NOTE: If you are connecting to a kerberized EMR cluster, please skip to the next section. This section is not required.

On the EMR cluster, all users of the platform must have access to the following locations:

LocationDescriptionRequired Access
EMR Resources bucket and path

The S3 bucket and path where resources can be stored by the Trifacta platform for execution of Spark jobs on the cluster.

The locations are configured separately in the Trifacta platform.

Read/Write
EMR Logs bucket and path

The S3 bucket and path where logs are written for cluster job execution.  

Read

These locations are configured on the Trifacta platform later.

Access Policies

EC2 instance profile

Trifacta users require the following policies to run jobs on the EMR cluster:

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "elasticmapreduce:AddJobFlowSteps",
                "elasticmapreduce:DescribeStep",
                "elasticmapreduce:DescribeCluster",
                "elasticmapreduce:ListInstanceGroups"
            ],
            "Resource": [
                "*"
            ]
        },
       {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::__EMR_LOG_BUCKET__",
                "arn:aws:s3:::__EMR_LOG_BUCKET__/*",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__/*"
            ]
        }

    ]
}

EMR roles

The following policies should be assigned to the EMR roles listed below for read/write access:

{
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::__EMR_LOG_BUCKET__",
                "arn:aws:s3:::__EMR_LOG_BUCKET__/*",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__",
                "arn:aws:s3:::__EMR_RESOURCE_BUCKET__/*"
            ]
        }
}

General configuration for Trifacta platform

Please complete the following sections to configure the Trifacta platform to communicate with the EMR cluster.

Change admin password

As soon as you have installed the software, you should login to the application and change the admin password. The initial admin password is the instanceId for the EC2 instance. For more information, see Change Password.

Verify S3 as base storage layer

EMR integrations requires use of S3 as the base storage layer.

NOTE: The base storage layer must be set during initial installation and set up of the Trifacta node.

See Set Base Storage Layer.

Set up S3 integration

To integrate with S3, additional configuration is required. See Enable S3 Access.

Configure EMR authentication mode

Authentication to AWS and to EMR supports the following basic modes:

  • System: A single set of credentials is used to connect to resources. 
  • User: Each user has a separate set of credentials. The user can choose to submit key-secret combinations or role-based authentication.

NOTE: Your method of authentication to AWS should already be configured. For more information, see Configure for AWS.

The authentication mode for your access to EMR can be configured independently from the base authentication mode for AWS, with the following exception:

NOTE: If aws.emr.authMode is set to user, then aws.mode must also be set to user.

Authentication mode configuration matrix:

AWS mode
(aws.mode)
systemuser
EMR mode
(aws.emr.authMode)



system

AWS and EMR use a single key-secret combination. Parameters to set:

"aws.s3.key"
"aws.s3.secret"

See Configure for AWS.

AWS access uses a single key-secret combination.

EMR access is governed by per-user credentials. Per-user credentials can be provided from one of several different providers.

NOTE: Per-user access requires additional configuration for EMR. See the following section.

For more information on configuring per-user access, see Configure for AWS.

userNot supported

AWS and EMR use the same per-user credentials for access. Per-user credentials can be provided from one of several different providers.

NOTE: Per-user access requires additional configuration for EMR. See the following section.

For more information on configuring per-user access, see Configure AWS Per-User Authentication.

Please apply the following configuration to set the EMR authentication mode:

Steps:

  1. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.
  2. Locate the following settings and apply the appropriate values. See the table below:

    "aws.emr.authMode":  "user",
    SettingDescription
    aws.emr.authMode

    Configure the mode to use to authenticate to the EMR cluster:

    system - In system mode, the specified AWS key and secret combination are used to authenticate to the EMR cluster. These credentials are used for all users.

    user - In user mode, user configuration is retrieved from the database.

    NOTE: User mode for EMR authentication requires that aws.mode be set to user. Additional configuration for EMR is below.

  3. Save your changes.

EMR per-user authentication for the Trifacta platform

If you have enabled per-user authentication for EMR (aws.emr.authMode=user), you must set the following properties based on the credential provider for your AWS per-user credentials.

You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

Authentication methodProperties and values

Use default credential provider for all Trifacta access including EMR.

NOTE: This method requires the deployment of a custom credential provider JAR.

"aws.credentialProvider":"default",
"aws.emr.forceInstanceRole":false,

Use default credential provider for all Trifacta access. However, EC2 role-based IAM authentication is used for EMR.

"aws.credentialProvider":"default",
"aws.emr.forceInstanceRole":true,

EC2 role-based IAM authentication for all Trifacta access

"aws.credentialProvider":"instance",

Configure Trifacta platform for EMR

NOTE: This section assumes that you are integrating with an EMR cluster that has not been kerberized. If you are integrating with a Kerberized cluster, please skip to "Configure for EMR with Kerberos".

Enable EMR integration

After you have configured S3 to be the base storage layer, you must enable EMR integration.

Steps:

You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

  1. Set the following value:

    "webapp.runInEMR": true,
  2. Set the following values:

    "webapp.runWithSparkSubmit": false,
  3. Verify the following property values:

    "webapp.runInTrifactaServer": true,
    "webapp.runWithSparkSubmit": false,
    "webapp.runInDataflow": false,

Apply EMR cluster ID

The Trifacta platform must be aware of the EMR cluster to which to connection. 

Steps:

  1. Administrators can apply this configuration change through the Admin Settings Page in the application. If the application is not available, the settings are available in trifacta-conf.json. For more information, see Platform Configuration Methods.

  2. Under External Service Settings, enter your AWS EMR Cluster ID. Click the Save button below the textbox.

For more information, see Admin Settings Page.

Extract IP address of master node in private sub-net

If you have deployed your EMR cluster on a private sub-net that is accessible outside of AWS, you must enable this property, which permits the extraction of the IP address of the master cluster node through DNS.

NOTE: This feature must be enabled if your EMR is accessible outside of AWS on a private network.

Steps:

  1. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.
  2. Set the following property to true:

    "emr.extractIPFromDNS": false,
  3. Save your changes and restart the platform.

Configure authentication mode

You can authenticate to the EMR cluster using either of the following authenticate modes:

  • System: A single set of credentials are used to connect to EMR. 
  • User: Each user has a separate set of credentials.

Steps:

  1. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.
  2. Locate the following settings and apply the appropriate values. See the table below:

    "aws.emr.authMode":  "user",
    SettingDescription
    aws.emr.authMode

    Configure the mode to use to authenticate to the EMR cluster:

    system - In system mode, the specified AWS key and secret combination are used to authenticate to the EMR cluster. These credentials are used for all users.

    user - In user mode, user configuration is retrieved from the database.

    NOTE: User mode for EMR authentication requires that aws.mode be set to user.

  3. Save your changes.

Configure Spark for EMR

For EMR, you can configure a set of Spark-related properties to manage the integration and its performance.

Configure Spark version

Depending on the version of EMR with which you are integrating, the Trifacta platform must be modified to use the appropriate version of Spark to connect to EMR.

NOTE: You should have already acquired the value to apply. See "Supported Spark Versions" above.

Steps:

  1. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.
  2. Locate the following:

    "spark.version": "<SparkVersionForMyEMRVersion>",
  3. Save your changes.

Use vendor libraries

If you are using EMR 5.20 or later (Spark 2.4 or later), you must configure the vendor libraries provided by the cluster. Please set the following parameter.

Steps:

  1. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.
  2. Locate the following:

    "spark.useVendorSparkLibraries": true,
  3. Save your changes.

Disable Spark job service

The Spark job service is not used for EMR job execution. Please complete the following to disable it:

Steps:

  1. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.
  2. Locate the following and set it to false:

    "spark-job-service.enabled": false,
  3. Locate the following and set it to false:

    "spark-job-service.enableHiveSupport": false,
  4. Save your changes.

Specify YARN queue for Spark jobs

Through the Admin Settings page, you can specify the YARN queue to which to submit your Spark jobs. All Spark jobs from the Trifacta platform are submitted to this queue.

Steps:

  1. In platform configuration, locate the following:

    "spark.props.spark.yarn.queue"
  2. Specify the name of the queue. 
  3. Save your changes.

Allocation properties

The following properties must be passed from the Trifacta platform to Spark for proper execution on the EMR cluster. 

To apply this configuration change, login as an administrator to the Trifacta node. Then, edit trifacta-conf.json. Some of these settings may not be available through the Admin Settings Page. For more information, see Platform Configuration Methods.

NOTE: Do not modify these properties through the Admin Settings page. These properties must be added as extra properties through the Spark configuration block. Ignore any references in trifacta-conf.json to these properties and their settings.

"spark": { 
  ...
  "props": { 
    "spark.dynamicAllocation.enabled": "true",
    "spark.shuffle.service.enabled": "true", 
    "spark.executor.instances": "0", 
    "spark.executor.memory": "2048M", 
    "spark.executor.cores": "2",
    "spark.driver.maxResultSize": "0"
  }
  ...
}
PropertyDescriptionValue
spark.dynamicAllocation.enabled
Enable dynamic allocation on the Spark cluster, which allows Spark to dynamically adjust the number of executors.true
spark.shuffle.service.enabled
Enable Spark shuffle service, which manages the shuffle data for jobs, instead of the executors.true
spark.executor.instances
Default count of executor instances.See Sizing Guide.
spark.executor.memory
Default memory allocation of executor instances.See Sizing Guide.
spark.executor.cores
Default count of executor cores.See Sizing Guide.
spark.driver.maxResultSizeEnable serialized results of unlimited size by setting this parameter to zero (0).0

Configure Trifacta platform for EMR with Kerberos

NOTE: This section applies only if you are integrating with a kerberized EMR cluster. If you are not, please skip to "Additional Configuration for EMR".

Disable standard EMR integration

When running jobs against a kerberized EMR cluster, you utilize the Spark-submit method of job submission. You must disable the standard EMR integration.

Steps:

You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

  1. Search for the following setting and set it false:

    "webapp.runInEMR": false,
  2. Set the following value:

    "webapp.runWithSparkSubmit": true,
  3. Disable use of Hive, which is not supported with EMR:

    "spark-job-service.enableHiveSupport": false,
  4. Verify the following property values:

    "webapp.runInTrifactaServer": true,
    "webapp.runInDataflow": false,
  5. Save your changes.

Enable YARN

To use Spark-submit, the Spark master must be set to use YARN.

Steps:

You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

  1. Search for the following setting and set it yarn:

    "spark.master": "yarn",
  2. Save your changes.

Acquire site config files

For integrating with an EMR cluster with Kerberos, the EMR cluster site XML configuration files must be downloaded from the EMR master node to the Trifacta node.

NOTE: This step is not required for non-Kerberized EMR clusters.

NOTE: When these files change, you must update the local copies.

  1. Download the Hadoop Client Configuration files from the EMR master node. The required files are the following:
    1. core-site.xml
    2. hdfs-site.xml
    3. mapred-site.xml
    4. yarn-site.xml
  2. These configuration files must be moved to the Trifacta deployment. By default, these files are in /etc/hadoop/conf:

    sudo cp <location>/*.xml /opt/trifacta/conf/hadoop-site/
    sudo chown trifacta:trifacta /opt/trifacta/conf/hadoop-site/*.xml
  3. (Option) If we want to support impersonate, we also need copy *.keytab from the EMR master node under /etc folder to EC2 instance under same folder.

Unused properties for EMR with Kerberos

When integrating with a kerberized EMR cluster, the following Trifacta settings are unused:

  1. External Service Settings: In the Admin Settings page, this section of configuration does not apply to EMR with Kerberos.
  2. Unused EMR settings: In the Admin Settings page, the following EMR settings do not apply to EMR with Kerberos:

    aws.emr.tempfilesCleanupAge
    aws.emr.proxyUser
    aws.emr.maxLogPollingRetries
    aws.emr.jobTagPrefix
    aws.emr.getLogsOnFailure
    aws.emr.getLogsForAllJobs
    aws.emr.extractIPFromDNS
    aws.emr.connectToRmEnabled


Additional Configuration for EMR

Default Hadoop job results format

For smaller datasets, the platform recommends using the Trifacta Photon running environment.

For larger datasets, if the size information is unavailable, the platform recommends by default that you run the job on the Hadoop cluster. For these jobs, the default publishing action for the job is specified to run on the Hadoop cluster, generating the output format defined by this parameter. Publishing actions, including output format, can always be changed as part of the job specification. 

As needed, you can change this default format. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

"webapp.defaultHadoopFileFormat": "csv",

Accepted values: csvjsonavropqt

For more information, see Run Job Page.

Configure Snappy publication

If you are publishing using Snappy compression for jobs run on an EMR cluster, you may need to perform the following additional configuration.

Steps:



  1. SSH into EMR cluster (master) node:

    ssh <EMR master node>
  2. Create tarball of native Hadoop libraries:

    tar -C /usr/lib/hadoop/lib -czvf emr-hadoop-native.tar.gz native
  3. Copy the tarball to the Trifacta EC2 instance used by the into the /tmp directory:

    scp -p emr-hadoop-native.tar.gz <EC2 instance>:/tmp
  4. SSH to Trifacta EC2 instance:

    ssh <EC2 instance>
  5. Create path values for libraries:

    sudo -u trifacta mkdir -p /opt/trifacta/services/batch-job-runner/build/libs
  6. Untar the tarball to the Trifacta installation path:

    sudo -u trifacta tar -C /opt/trifacta/services/batch-job-runner/build/libs -xzf /tmp/emr-hadoop-native.tar.gz
  7. Verify libhadoop.so* and libsnappy.so* libraries exist and are owned by the Trifacta user:

    ls -l /opt/trifacta/services/batch-job-runner/build/libs/native/
  8. Verify that the /tmp directory has the proper permissions for publication. For more information, see Supported File Formats.

  9. A platform restart is not required.

Additional parameters

You can set the following parameters as needed:

Steps:

You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

PropertyRequiredDescription
aws.emr.resource.bucketY

S3 bucket name where Trifacta executables, libraries, and other resources can be stored that are required for Spark execution.

aws.emr.resource.pathY

S3 path within the bucket where resources can be stored for job execution on the EMR cluster.

NOTE: Do not include leading or trailing slashes for the path value.

aws.emr.proxyUserY

This value defines the user for the Trifacta users to use for connecting to the cluster.

NOTE: Do not modify this value.

aws.emr.maxLogPollingRetriesNConfigure maximum number of retries when polling for log files from EMR after job success or failure. Minimum value is 5.
aws.emr.tempfilesCleanupAgeN

Defines the number of days that temporary files in the /trifacta/tempfiles directory on EMR HDFS are permitted to age.

By default, this value is set to 0, which means that cleanup is disabled.

If needed, you can set this to a positive integer value. During each job run, the platform scans this directory for temp files older than the specified number of days and removes any that are found. This cleanup provides an additional level of system hygiene.

Before enabling this secondary cleanup process, please execute the following command to clear the tempfiles directory:

hdfs dfs -rm -r -skipTrash /trifacta/tempfiles

Optional Configuration

Configure for Redshift

For more information on configuring the platform to integrate with Redshift, see Create Redshift Connections.

Switch EMR Cluster

If needed, you can switch to a different EMR cluster through the application. For example, if the original cluster suffers a prolonged outage, you can switch clusters by entering the cluster ID of a new cluster. For more information, see Admin Settings Page.

Configure Batch Job Runner

Batch Job Runner manages jobs executed on the EMR cluster. You can modify aspects of how jobs are executed and how logs are collected. For more information, see Configure Batch Job Runner.

Modify Job Tag Prefix

In environments where the EMR cluster is shared with other job-executing applications, you can review and specify the job tag prefix, which is prepended to job identifiers to avoid conflicts with other applications.

Steps:

  1. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.
  2. Locate the following and modify if needed:

    "aws.emr.jobTagPrefix": "TRIFACTA_JOB_",
  3. Save your changes and restart the platform.

Testing

  1. Load a dataset from the EMR cluster.
  2. Perform a few simple steps on the dataset.
  3. Click Run Job in the Transformer page. 
  4. When specifying the job: 
    1. Click the Profile Results checkbox.
    2. Select Spark.
  5. When the job completes, verify that the results have been written to the appropriate location.

This page has no comments.