This section provides additional configuration requirements for integrating the
|D s platform|
- This section applies only to the versions of HDP that are supported by the
. For more information, see Supported Deployment Scenarios for Hortonworks.
D s platform
NOTE: Except as noted, the following configuration items apply to the latest supported version of Hortonworks Data Platform.
Before you begin, it is assumed that you have completed the following tasks:
- Successfully installed a supported version of Hortonworks Data Platform into your enterprise infrastructure.
- Installed the
in your environment. For more information, see Install Software.
D s item item software
- Reviewed the mechanics of platform configuration. See Required Platform Configuration.
- Configured access to the
. See Configure the Databases.
D s item item database
- Performed the basic Hadoop integration configuration. See Configure for Hadoop.
You have access to platform configuration either via the
or through the Admin Settings page.
D s node
Hortonworks Cluster Configuration
The following changes need to be applied to Hortonworks cluster configuration files or to configuration areas inside Ambari.
Tip: Ambari is the recommended method for configuring your Hortonworks cluster.
Configure for Ranger
Configure Ranger to use Kerberos
If you have deployed Ranger in a Kerberized environment, you must verify and complete the following changes in Ambari.
- If you have enabled Ranger, navigate to Hive > Configs > Settings.
- Choose Authorization: Ranger.
- Hiveserver2 Authentication: Kerberos.
- If you have enabled Ranger and Hive, navigate to Hive > Configs > Advanced > General.
- hive.security.authorization.manager: org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory
- Navigate to Hive > Configs > Advanced > Advanced hive-site.
- hive.security.authentication.manager: org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator
- hive.conf.restricted.list: hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role,hive.security.authorization.enabled
- Navigate to Hive > Configs > Advanced > Custom hive-site. Changes in this area update
D s defaultuser Type hadoop.group Full true
- hadoop.proxyuser.trifacta.hosts: *
- hive.metastore.sasl.enabled: true
- Save your configuration changes.
Configure for Spark Profiling
Disable intermediate caching for Hive profiling jobs
For Hortonworks 3.0 and later, the intermediate dataset files that are generated as part of Spark profiling of your job can cause the job to hang when the source is a Hive table. As a precaution, if you are profiling jobs from Hive sources, you should disable the following property on Hortonworks 3.0 and later.
D s config
- Locate the
Insert the following setting:
- Save your changes and restart the platform.
Additional configuration for Spark profiling on S3
If you are using S3 as your datastore and have enabled Spark profiling, you must apply the following configuration, which adds the
hadoop-aws JAR and the
aws-java-sdk JAR to the extra class path for Spark.
- In Ambari, navigate to Spark2 > Configs.
- Add a new parameter to Custom Spark2-defaults.
Set the parameter as follows, which is specified for HDP 18.104.22.168, build 37:
- Restart Spark from Ambari.
- Restart the
D s platform item node
Additional configuration for Spark profiling
If you are using Spark for profiling, you must add environment properties to your cluster configuration. See Configure for Spark.
Set up directory permissions
On all Hortonworks cluster nodes, verify that the YARN user has access to the YARN working directories:
If you are upgrading from a previous version of Hortonworks, you may need to clear the YARN user cache for the
The following changes need to be applied to the
Except as noted, these changes are applied to the following file in the
Configure WebHDFS port
Configure Resource Manager port
Hortonworks uses a custom port number for Resource Manager. You must update the setting for the port number used by Resource Manager.
Save your changes.
Configure location of Hadoop bundle JAR
Set the value for the Hadoop bundle JAR to the appropriate distribution. The following is for Hortonworks 3.0:
Save your changes.
Configure Hive Locations
If you are enabling an integration with Hive on the Hadoop cluster, there are some distribution-specific parameters that must be set. For more information, see Configure for Hive.
To apply your changes, restart the platform. See Start and Stop the Platform.
After restart, you should verify operations. For more information, see Verify Operations.