Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r0681
Excerpt

ADLS access leverages HDFS protocol and storage, so additional configuration is required.

Steps:

  1. D s config
  2. Enable ADLS as the base storage layer:

    Code Block
    "webapp.storageProtocol": "hdfs",
    "hdfs.enabled": true,
    "hdfs.protocolOverride": "adl",
    ParameterDescription
    webapp.storageProtocol

    Sets the base storage layer for the platform. Set this value to hdfs.

    Info

    NOTE: After this parameter has been saved, you cannot modify it. You must re-install the platform to change it.

    hdfs.enabledFor ADLS blob storage, set this value to true.
    hdfs.protocolOverrideFor ADLS blob storage, this special parameter must be set to adl.
  3. These parameters specify the Azure Data Lake for the platform:

    Code Block
    "azure.adl.enabled": =  "true"
    "azure.adl.store": =  "adl://xxx.azuredatalakestore.net"
    ParameterDescription
    azure.adl.enabledTo enable access to the Azure Data Lake, set this value to true.
    azure.adl.store

    Specify the value of the Azure Data Lake Store here.

    Info

    NOTE: Protocol should be set to adl://.

  4. Configure the appropriate Hadoop bundle JAR to use:

    Code Block
    "hadoopBundleJar": "hadoop-deps/hdp-2.6/build/libs/hdp-2.6-bundle.jar",
  5. Configure access to HDFS resources:

    Code Block
    "hdfs.namenode.host": "xxx.azuredatalakestore.net",
    "hdfs.namenode.port": "443",
    "hdfs.webhdfs.host": "xxx.azuredatalakestore.net",
    "hdfs.webhdfs.ssl.enabled": "true",
    "hdfs.webhdfs.port": "443",
    "hdfs.highavailability.serviceName": "xxx.azuredatalakestore.net",
    ParameterDescription
    hdfs.namenode.hostHostname of the HDFS namenode
    hdfs.namenode.portPort number for the HDFS namenode
    hdfs.webhdfs.hostHostname of WebHDFS service
    hdfs.webhdfs.ssl.enabledIf SSL has been enabled on the WebHDFS host, please set this value to true. You are likely to need to set the port value to a non-default value.
    hdfs.webhdfs.portPort number of WebHDFS service
    hdfs.highavailability.serviceNameSet this value the high availability service name for HDFS, if you have enabled integration with the cluster high availability. For more information, see Enable Integration with Cluster High Availability in the Configuration Guide.
  6. Save your changes.