Enable ADLS Gen2 Access

Microsoft Azure deployments can integrate with with the next generation of Azure Data Lake Store (ADLS Gen2).

• Microsof Azure Data Lake Store Gen2 (ADLS Gen2) combines the power of a high-performance file system with massive scale and economy. Azure Data Lake Storage Gen2 extends Azure Blob Storage capabilities and is optimized for analytics workloads. 
• For more information, see https://azure.microsoft.com/en-us/services/storage/data-lake-storage/.

Limitations of ADLS Gen2 Integration

Read-only access

If the base storage layer has been set to WASB, you can follow these instructions to set up read-only access to ADLS Gen2. 

Pre-requisites

General

• The Designer Cloud Powered by Trifacta platform has already been installed and integrated with an Azure Databricks cluster. See Configure for Azure Databricks.

• For each combination of blob host and container, a separate Azure Key Vault Store entry must be created. For more information, please contact your Azure admin. 

• When running against ADLS Gen2, the product requires that you create a filesystem object in your ADLS Gen2 storage account.
• ABFSS must be set as the base storage layer for the Designer Cloud Powered by Trifacta platform instance. See Set Base Storage Layer.

Create a registered application

Before you integrate with Azure ADLS Gen2, you must create the Designer Cloud Powered by Trifacta platform as a registered application. See Configure for Azure.

Azure properties

The following properties should already be specified in the Admin Settings page. Please verify that the following have been set:

• azure.applicationId
• azure.secret
• azure.directoryId

The above properties are needed for this configuration.

Registered application role

For more information, see Configure for Azure.

Key Vault Setup

An Azure Key Vault has already been set up and configured for use by the Designer Cloud Powered by Trifacta platform. Properties must be specified in the platform, if they have not been configured already.

For more information on configuration for Azure key vault, see Configure for Azure.

Configure the Designer Cloud Powered by Trifacta platform

Define base storage layer

Per earlier configuration:

• webapp.storageProtocol must be set to abfss.

  NOTE: Base storage layer must be configured when the platform is first installed and cannot be modified later.

  NOTE: To enable read-only access to ADLS Gen2, do not set the base storage layer to abfss.

• hdfs.protocolOverride is ignored.

See Set Base Storage Layer.

Review Java VFS Service

Use of ADLS Gen2 requires the Java VFS service in the Designer Cloud Powered by Trifacta platform.

For more information on configuring this service, see Configure Java VFS Service.

Configure file storage protocols and locations

The Designer Cloud Powered by Trifacta platform must be provided the list of protocols and locations for accessing ADLS Gen2 blob storage. 

Steps:

1. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

2. Locate the following parameters and set their values according to the table below:
   
   

   "fileStorage.whitelist": ["abfss"],
   "fileStorage.defaultBaseUris": ["abfss://filesystem@storageaccount.dfs.core.windows.net/"],

   Parameter	Description
   filestorage.whitelist	A comma-separated list of protocols that are permitted to read and write with ADLS Gen2 storage. NOTE: The protocol identifier "abfss" must be included in this list.
   filestorage.defaultBaseUris	For each supported protocol, this param must contain a top-level path to the location where Designer Cloud Powered by Trifacta platform files can be stored. These files include uploads, samples, and temporary storage used during job execution. NOTE: A separate base URI is required for each supported protocol. You may only have one base URI for each protocol.

3. Save your changes and restart the platform.

Configure access mode

System mode access

When access to ADLS Gen2 is requested, the platform uses the combination of Azure directory ID, Azure application ID, and Azure secret to complete access.

Steps:

Please verify the following steps to specify the ADLS access mode.

1. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

2. Verify that the following parameter to system:

   "azure.adlsgen2.mode": "system",

3. Save your changes.

User mode access

In user mode, a user ID hash is generated from the Key Vault key/secret and the user's AD login. This hash is used to generate the access token, which is stored in the Key Vault.

Pre-requisites:

• Designer Cloud Powered by Trifacta platform must be integrated with a Databricks 8.3 cluster. For more information, see Configure for Azure Databricks.
• User mode access to ADLS requires Single Sign On (SSO) to be enabled for integration with Azure Active Directory. For more information, see Configure SSO for Azure AD.

Steps:

Please verify the following steps to specify the ADLS access mode.

1. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

2. Set the following parameter to user:

   "azure.adlsgen2.mode": "user",

3. Save your changes.

Testing

Restart services. See Start and Stop the Platform.

After the configuration has been specified, an ADLS Gen2 connection appears in the Import Data page. Select it to begin navigating for data sources.

Try running a simple job from the Designer Cloud application. For more information, see Verify Operations.

• Except as noted above, the ADLS Gen2 browser is identical to the ADLS one. See \ADLS Gen2 Browser.
• Except as noted above, the basic usage of ADLS Gen2 is identical to the ADLS (Gen1) version. See Using ADLS.

Troubleshooting

Problem: SSLHandshakeException : Unsupported curveId: 29 error when retrieving Databricks token

This issue is caused by the Designer Cloud Powered by Trifacta platform sending a known set of elliptic curve algorithms to Microsoft during SSL handshake, but an unsupported curve algorithm is being negotiated and used by the Microsoft server. 

• This issue applies only when SSL is enabled when accessing the base storage layer.
• This issue applies to all Azure-based base storage layers supported by the Designer Cloud Powered by Trifacta platform.

A similar issue is described here: https://bugs.openjdk.java.net/browse/JDK-8171279

Solution:

Microsoft should fix the problem.

Within the Designer Cloud Powered by Trifacta platform, you can apply the following workaround:

1. Login to the Alteryx node as an administrator.

2. Edit the following file:

   $JAVA_HOME/jre/lib/security/java.security
   

3. Locate the following parameter: jdk.tls.disabledAlgorithms.

4. To the above parameter, add the following algorithm references to disable them:

   TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
   

5. Save your changes and restart the platform.