In your platform configuration, you must specify the storage platform that is your base storage layer. The base storage layer defines the primary storage integration for the . In some cases, integration with other storage layers is supported.

After you define the base storage layer and restart the platform, you cannot change the base storage layer to another option. Please consider your options carefully before you define the base storage layer.

If S3 is the base storage layer, you must also define the default storage bucket to use during initial installation, which cannot be changed at a later time. For additional requirements, see Enable S3 Access.

NOTE: If S3 is specified as your base storage layer, you cannot publish to Hive. If HDFS is specified as your base storage layer, you cannot publish to Redshift.

Base Storage Layer Options

Base Storage LayerDescriptionNotes
hdfs

If you are integrating with a Hadoop cluster for job execution, you can use HDFS for base storage.

Tip: For , HDFS is the default base storage layer.

This option is required for ADLS integration from deployments on Azure.

NOTE: If you have deployed the platform into Microsoft Azure and are integrating with Microsoft ADLS, you must set the base storage layer to hdfs. Additionally, you must set webapp.protocolOverride to adl.

For more information, see Enable ADLS Access.

Required for:

  • Publish to Hive
  • Access to ADLS (Azure deployments only)
s3

If you have installed the product on-premises or on an EC2 instance in AWS, you can set the base storage layer to S3.

Read access to S3 is supported if HDFS is the base storage layer.

For more information, see Enable S3 Access.

Required for:

  • Enable write access to S3
  • Publish to Redshift
wasbs or wasb

If you have installed the product from the Azure Marketplace and are integrating with WASB, you can optionally set to the base storage layer to WASBS (secure) or WASB (non-secure).

Tip: WASBS is preferred.

NOTE: If you are using WASB storage, you must also configure the appropriate setting for the storage protocol, which may depend on how you have created the security token on Azure.

For more information, see Enable WASB Access.

Required for:

  • Access to WASB (Azure deployments only)

For more information on options, see Storage Deployment Options.

Set Storage Layer

When you have decided on the final base storage layer, set the following property to one of the above values in platform configuration.

"webapp.storageProtocol": "hdfs",

This value cannot be changed after saving.

Disable Hadoop Access

If you are not using Hadoop at all, please complete the following configuration change.

Steps:

  1. Login to the . 

  2. Edit the following files:

    site-config-*-s3.json
site-config.installer-*-edge.json


  3. In these files, set the following property value to hostname:

    "hdfs.namenode.host":"hostname",

  4. Save the files and restart the platform.