D toc |
---|
D s install marketplace |
---|
Please complete the following steps in the listed order to configure your installed instance of the
D s platform |
---|
Pre-requisites
Deploy HDI cluster and
.D s item item node Info NOTE: The HDI cluster can be deployed as part of installation from the Marketplace. You can also integrate the platform with a pre-existing cluster. Details are below.
- Install
on the node.D s platform
For more information, see Install from Azure Marketplace.
Excerpt | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Configure AzureCreate registered applicationYou must create a Azure Active Directory (AAD) application and grant it the desired access permissions, such as read/write access to the ADLS resource and read/write access to the Azure Key Vault secrets . This service principal is used by the
After you have registered, acquire the following information:
These properties are applied later in the configuration process. Configure the PlatformConfigure for HDIIf you are integrating the
Configure base storage layerFor Azure installations, you can set your base storage layer to be HDFS or WASB.
Configure for Key VaultFor authentication purposes, the
Please complete the following sections to create and configure your Azure Key Vault. Create a Key Vault resource in Azure
Enable Key Vault access for the
In the Azure portal, you must assign access policies for application principal of the
Steps:
Create WASB access tokenIf you are enabling access to WASB, you must create this token within the Azure Portal.
For more information, see https://docs.microsoft.com/en-us/rest/api/storageservices/delegating-access-with-a-shared-access-signature. Configure Key Vault key and secret for WASBIn the Key Vault, you can create key and secret pairs for use.
WASB: To enable access to the Key Vault, you must specify your key and secret values as follows:
Acquire shared access signature value: In the Azure portal, please do the following:
Create a custom key: To create a custom key and secret pair for WASB use by the
Configure Key Vault locationFor ADLS or WASB, the location of the Azure Key Vault must be specified for the
Steps:
This value is the location for the Key Vault. It must be applied in the
Steps:
Apply SAS token identifier for WASBIf you are using WASB as your base storage layer, you must apply the SAS token value into the configuration of the
Steps:
Configure Secure Token ServiceAccess to the Key Vault requires use of the secure token service (STS) from the
Configure for SSOIf needed, you can integrate the
Configure for ADLSEnable read-only or read-write access to ADLS. For more information, see Enable ADLS Access. Configure for WASBEnable read-only or read-write access to WASB. For more information on integrating with WASB, see Enable WASB Access. Configure relational connectionsIf you are integrating
Create encryption key fileAn encryption key file must be created on the
Create Hive connectionYou can create a connection to the Hive instance on the HDI cluster with some modifications. Natively, Azure supports high availability for HiveServer2 via Zookeeper. As a result, host and port information in the JDBC URL must be replaced with a Zookeeper quorum. In addition to the other Hive connection properties, please specify the following values for the properties listed below:
Connections are created through the Connections page. See Connections Page. For additional details on creating a conection to Hive, see Create Hive Connections. A Hive connection can also be created using the above property substitutions via CLI or API.
Create Azure SQL DB connectionFor more information, see Create SQL DB Connections. Create Azure SQL DW connectionFor more information, see Create SQL DW Connections. Workaround for missing Python packagesAfter installation, the supervisord process may complain about some Python packages that are "missing."
These packages are present but lack the appropriate permissions. To enable the packages for use, please run the following on the
|
Testing
- Load a dataset from the HDI cluster through either ADLS or WASB.
- Perform a few simple steps on the dataset.
- Click Run Job in the Transformer page.
- When specifying the job:
- Click the Profile Results checkbox.
- Select Hadoop.
- When the job completes, verify that the results have been written to the appropriate location.