In your platform configuration, you must specify the storage platform that is your base storage layer. The base storage layer defines the primary storage integration for the
|D s platform|
After you define the base storage layer and restart the platform, you cannot change the base storage layer to another option. Please consider your options carefully before you define the base storage layer.
If S3 is the base storage layer, you must also define the default storage bucket to use during initial installation, which cannot be changed at a later time. For additional requirements, see Enable S3 Access.
NOTE: If HDFS is specified as your base storage layer, you cannot publish to Redshift.
Base Storage Layer Options
If you are integrating with a Hadoop cluster, you can use HDFS for base storage.
This option is required for ADLS (Gen1) from deployments on Azure:
- ADLS (Gen1): If you have deployed the platform into Microsoft Azure and are integrating with Microsoft ADLS (Gen1), you must set the base storage layer to
hdfs. Additionally, you must set
adl. For more information, see Enable ADLS Access.
- ADLS Gen2: This protocol is not used for ADLS Gen2 storage.Please see ABFSS below.
- Access to ADLS Gen1 (Azure deployments only)
If you have installed the product on-premises or on an EC2 instance in AWS, you can set the base storage layer to S3.
Read access to S3 is supported if HDFS is the base storage layer.
For more information, see Enable S3 Access.
- Enable write access to S3
- Publish to Redshift
If you have installed the product from the Azure Marketplace and are integrating with WASB, you must set to the base storage layer to WASBS.
For more information, see Enable WASB Access.
- Access to WASB (Azure deployments only)
Set the base storage layer to
abfss if you are integrating with ADLS Gen2.
NOTE: ADLS Gen2 storage requires an Azure Databricks cluster for execution.
For more information, see Enable ADLS Gen2 Access.
- Access to ADLS Gen2 (Azure deployments only)
Base storage layer port options
When you configure your base storage layer, you must also define the port number to use for access.
NOTE: If you change the port number of the base storage layer in the future, all results from previous jobs are lost. Please choose the port number with care.
Set Storage Layer
When you have decided on the final base storage layer, set the following property to one of the above values in platform configuration.
|D s config|
This value cannot be changed after saving.
Disable Hadoop Access
If you are not using Hadoop at all, please complete the following configuration change.
- Login to the
D s node
Edit the following files:
In these files, set the following property value to
Save the files and restart the platform.