1. To create a new service principal, see https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal#create-an-azure-active-directory-application.
2. After you have registered, please acquire the following property values prior to install.
   1. For an existing service principal, see https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal#get-application-id-and-authentication-key to obtain the property values.

These values are applied during the install process.

Install Process

Methods

You can install from the Azure Marketplace using one of the following methods:

1. Create cluster: Create a brand-new HDI cluster and add

   D s product
   product ee

   as an application.

   Tip
   Tip: This method is easiest and fastest to deploy.

2. Add application: Use an existing HDI cluster and add

   D s product
   product ee

   as an application.

   Tip
   Tip: This method does not support choosing a non-default size for the D s item item node . If you need more flexibility, please choose the following option.

3. Create a custom ARM template: Use an existing HDI cluster and configure configure a custom application via Azure Resource Manager (ARM) template.

   Tip
   Tip: Use the third method only if your environment requires additional configuration flexibility or automated deployment via ARM template.

Depending on your selection, please follow the steps listed in one of the following sections.

Install Method - New cluster

Please use the following steps if you are creating a new HDI cluster and adding the

Steps:

1. From the Azure Marketplace listing, click Get it Now.
2. In Microsoft Azure Portal, click the New blade.
3. Select

   D s product
   product ee

   .
4. Click Create. Then, click the Quick Create tab.
   1. Please configure any settings that are not listed below according to your enterprise requirements.
   2. For more information on the Quick Create settings, see https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-provision-linux-clusters.
5. Basics tab:
   1. Cluster type:
      1. Hadoop: One of the following:
         1. Hadoop
         2. HBase
         3. Spark
      2. Version:
         1. HDI 3.6

      3. Click Select.

   2. Click Next.
6. Storage tab:

   1. Select your primary storage:

      Info
      NOTE: D s product product ee does not support Additional storage options. Additional configuration is required after install is complete for the supported storage options.

       

      1. Azure Storage
      2. Azure Data Lake Store
   2. Select a SQL database for Hive: If you plan to use Hive, you can choose to specify a database. If not, the platform creates one for you.
   3. Click Next.

7. D s product
   product ee

   tab:
   1. Please review and accept the terms for using the product.
   2. Click Create. Click Ok.
   3. Click Next.

8. Custom tab:

   1. By default, the

      D s item
      item node

      is defined as: Standard_D13_V2.

   2. If you need to change the edge node size, please click the Custom settings tab and make your selections. The following virtual machine types are supported:

      Code Block
      Standard_A4 Standard_A6 Standard_A7 Standard_A10 Standard_A11 Standard_D4_v2 Standard_D5_v2 Standard_D12_v2 Standard_D13_v2 Standard_D14_v2 Standard_D15_v2

   3. Click Next.

9. Advanced Settings tab:

   1. No changes are needed here.

10. Summary tab:

    1. Review the specification of the cluster you are creating.
    2. Make modifications as needed.
    3. To create the cluster, click Create.
11. After the cluster has been started and deployed:
    1. Login to the application.
    2. Change the admin password.
    3. Perform required additional configuration.
    4. Instructions are provided below.

Install Method - Add application to a cluster

Please use the following steps if you are adding the

Steps:

1. In the Microsoft Azure Portal, select the HDInsight cluster to which you are adding the application.
2. In the Portal, select the Applications blade. Click + Add.
3. From the list of available applications, select

   D s product
   product ee

   .
4. Please accept the legal terms.
5. Click Next.
6. The application is created for you.
7. After the cluster has been started and deployed:
   
   1. Login to the application.
   2. Change the admin password.
   3. Perform required additional configuration.
   4. Instructions are provided below.

Install Method - Build custom ARM template

Please use the following steps if you are creating a custom application template for later deployment. This method provides more flexible configuration options and can be used for deployments in the future.

Steps:

1. Start here: https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Ftrifacta%2Fazure-deploy%2Fmaster%2FmainTemplatedeploy%2Frelease%2F5.0%2FmainTemplate.json
2. From the Microsoft Azure Portal, select the custom deployment link.
3. Resource Group: Create or select one.
4. Cluster Name: Select an existing cluster name.
5. Edge Node Size: Select the instance type. For more information, see the Sizing Guide above.

6. D s item
   item version

   : For the version, select the latest listed version of

   D s product
   product ee

   .
7. Application Name: If desired, modify the application name as needed. This name must be unique per cluster.
8. Subdomain Application URI Suffix: If desired, modify the three-character alphanumeric string used in the DNS name of the application. This suffix must be unique per cluster.
9. Please specify values for the following:
   1. Application ID
   2. Directory ID
   3. Secret
10. Gallery Package Identifier: please leave the default value.
11. Please accept the Microsoft terms of use.
12. To create the template, click Purchase.
13. The custom template can be used to create the

    D s product
    product ee

    application. For more information, please see the Azure documentation.
14. After the application has been started and deployed:
    1. Login to the application.
    2. Change the admin password.
    3. Perform required additional configuration.
    4. Instructions are provided below.

Login to

D s product
product ee

Steps:

1. In the Azure Portal, select the HDI cluster.
2. Select the Applications blade.
3. Select the

   D s product
   product ee

   application.
4. Click the Portal link.

5. You may be required to apply the cluster username and password.

   Info

   NOTE: You can create a local user of the cluster to avoid enabling application users to use the administrative user's cluster credentials. To create such a user: 1. Navigate to the cluster's Ambari console. 2. In the user menu, select the Manage Ambari page. 3. Select Users. 4. Select Create Local User. 5. Enter a unique (lowercase) user name. 6. Enter a password and confirm that password. 7. Click Save.

6. You are connected to the

   D s item
   item application

   .
7. In the login screen, enter the default username and password:
   
   1. Username: admin@trifacta.local
   2. Password: admin
   3. Click Login.

Change admin password

Steps:

1. If you haven't done so already, login to the

   D s item
   item application

   as an administrator.
2. In the menu bar, select Settings menu > Administrator.
3. In the User Profile, enter and re-enter a new password.
4. Click Save.
5. Logout and login again using the new password.

Post-install configuration

Base parameter settings

If you are integrating with HDI and did not install via a custom ARM template, the following settings must be specified with the

Steps:

1. Login to the

   D s item
   item application

   as an administrator.
2. In the menu bar, select Settings menu > Admin Settings.

3. In the Admin Settings page, specify the values for the following parameters:

   Code Block
   "azure.secret" "azure.applicationId" "azure.directoryId"

4. Save your changes and restart the platform.

5. When the platform is restarted, continue the following configuration.

Apply license file

When the application is first created, the license is valid for 24 hours. Before the license expires, you must apply the license key file to the

Steps:

1. Locate the license key file that was provided to you by

   D s company

   . Please store this file in a safe location that is not on the

   D s item
   item node

   .
2. In the Azure Portal, select the HDI cluster.
3. Select the Applications blade.
4. Select the

   D s product
   product ee

   application.
5. From the application properties, acquire the SSH endpoint.
6. Connect via SSH to the

   D s item
   item node

   . For more information, see https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-linux-use-ssh-unix.

7. Drop the license key file in the following directory on the node:

   Code Block
   /opt/trifacta/license

8. Restart the platform.

Review refresh token encryption key

By default, the

If preferred, you can generate your own key value, which is unique to your instance of the platform.

Steps:

1. Login at the command line to the

   D s item
   item node

   .

2. To generate a new refresh token encryption key, please execute the following command:

   Code Block
   cd /opt/trifacta/services/secure-token-service/ && java -cp server/build/install/secure-token-service/secure-token-service.jar:server/build/install/secure-token-service/lib/* com.trifacta.services.secure_token_service.tools.RefreshTokenEncryptionKeyGeneratorTool

3. The refresh token encryption key is printed to the screen. Please copy this value to the clipboard.

4. D s config

5. Paste the value in the following property:

   Code Block
   "com.trifacta.services.secure_token_service.refresh_token_encryption_key": "<generated_refresh_token_encryption_key>",

6. Save your changes.

Configure Spark Execution (Advanced)

The cluster instance of Spark is used to execute larger jobs across the nodes of the cluster. Spark processes run multiple executors per job. Each executor must run within a YARN container. Therefore, resource requests must fit within YARN’s container limits.

Like YARN containers, multiple executors can run on a single node. More executors provide additional computational power and decreased runtime.

Spark’s dynamic allocation adjusts the number of executors to launch based on the following:

• job size

• job complexity

• available resources

The per-executor resource request sizes can be specified by setting the following properties in the spark.props section.

A single special process (the application driver) also runs in a container. Its resources are specified in the spark.props section:

Optimizing "Small" Joins (Advanced)

Broadcast, or map-side, joins materialize one side of the join and send it to all executors to be stored in memory. This technique can significantly accelerate joins by skipping the sort and shuffle phases during a "reduce" operation. However, there is also a cost in communicating the table to all executors. Therefore, only "small" tables should be considered for broadcast join. The definition of "small" is set by the spark.sql.autoBroadcastJoinThreshold parameter which can be added to the spark.props section of

Example Spark configuration parameters for given YARN configurations:

Property settings in bold are provided by the cluster.