Azure Install Base Storage Layer ADLS Gen2

Contents:

Expand all Collapse all

use the CDH 6.1 bundle JARs

Please complete the following configuration steps in the Designer Cloud Powered by Trifacta® platform.

NOTE: Integration with ADLS Gen2 is supported only on Azure Databricks.

Steps:

You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

Enable ADLS Gen2 as the base storage layer:

"webapp.storageProtocol": "abfss",
"hdfs.enabled": false,
"hdfs.protocolOverride": "",

Parameter	Description
webapp.storageProtocol	Sets the base storage layer for the platform. Set this value to `abfss`. NOTE: After this parameter has been saved, you cannot modify it. You must re-install the platform to change it.
hdfs.enabled	For ADLS Gen2 access, set this value to `false`.
hdfs.protocolOverride	For ADLS Gen2 access, this special parameter should be empty. It is ignored when the storage protocol is set to `abfss`.

Configure ADLS Gen2 access mode. The following parameter must be set to system.
```
"azure.adlsgen2.mode": "system",
```
The platform must be configured to use the CDH 6.1 bundle JARs:

NOTE: If you have integrated with Databricks Tables, do not overwrite the value for data-service.hiveJdbcJar with the following value, even if it's set to a different distribution JAR file.
```
"hadoopBundleJar": "hadoop-deps/cdh-6.2/build/libs/cdh-6.2-bundle.jar",
"spark-job-service.hiveDependenciesLocation": %(topOfTree)s/hadoop-deps/cdh-6.2/build/libs",
"data-service.hiveJdbcJar": "hadoop-deps/cdh-6.2/build/libs/cdh-6.2-hive-jdbc.jar",
```

Set the protocol whitelist and base URIs for ADLS Gen2:

"fileStorage.whitelist": ["abfss"],
"fileStorage.defaultBaseUris": ["abfss://filesystem@storageaccount.dfs.core.windows.net/"],

Parameter	Description
fileStorage.whitelist	A comma-separated list of protocols that are permitted to read and write with ADLS Gen2 storage. NOTE: The protocol identifier `"abfss"` must be included in this list.
fileStorage.defaultBaseUris	For each supported protocol, this param must contain a top-level path to the location where Designer Cloud Powered by Trifacta platform files can be stored. These files include uploads, samples, and temporary storage used during job execution. NOTE: A separate base URI is required for each supported protocol. You may only have one base URI for each protocol.

Save your changes.
The Java VFS service must be enabled for ADLS Gen2 access. For more information, see Configure Java VFS Service in the Configuration Guide.