This section contains hardware and software requirements for successful installation of .

Edge Node Requirements

If the is installed in a Hadoop cluster, the platform is typically installed on an edge node of the cluster.

NOTE: The does not require edge node installation.

Hardware Requirements

Minimum hardware:

ItemRequired
Number of cores

8 cores

RAM

64 GB

NOTE: The platform requires 12GB of dedicated RAM to start and perform basic operations.


Disk space to install software4 GB
Total free disk space

16 GB

Space requirements by volume:

  • /opt - 10 GB
  • /var - Remainder

Recommended hardware:

ItemRecommended
Number of cores

16 cores

RAM

128 GB

NOTE: The platform requires 12GB of dedicated RAM to start and perform basic operations.


Disk space to install software16 GB
Total free disk space

100 GB

Space requirements by volume:

  • /opt - 10 GB
  • /var - Remainder

Operating System Requirements

The following operating systems are supported for the .

NOTE: The requires 64-bit versions of any supported operating system.

NOTE: During normal operations, the platform may maintain a high number of open files, which may exceed the default limit defined by the operating system. Before you begin using the system, you should raise this limit to 64000. For more information on raising the ulimit, see Miscellaneous Configuration.


NOTE: If you are enabling SSO and want to use an Apache Server as a reverse proxy for the , you may need to upgrade to Apache Server. See Configure SSO for AD-LDAP.

For more information on RPM dependencies, see System Dependencies.

Database Requirements

The following database versions are supported by the for storing metadata and the user's recipes.

NOTE: One of these supported versions must be installed on the .

Supported versions:

NOTE: H2 database type is used for internal testing. It is not a supported database.

 

For more information on installing and configuring the database, see Install Databases.

Other Software Requirements

The following software components must be present.

Java

Tip: Where possible, you should install the same version of Java on the and on the cluster with which you are integrating.

 

Other Software

NOTE: For Ubuntu installations, the following packages must be manually installed using Ubuntu-specific versions. Instructions and version numbers are provided later in the process.

 

Root User Access

Installation must be executed as the root user on the .

SSL Access

(Optional) If users are connecting to the , an SSL certificate must be created and deployed. See Install SSL Certificate.

Internet Access

(Optional) Internet access is not required for installation or operation of the platform. However, if the server does not have Internet access, you must acquire additional software as part of the disconnected install. For more information, see Install Dependencies without Internet Access.

Hadoop Cluster Requirements

The following requirements apply if you are integrating the with an enterprise Hadoop cluster.

NOTE: For general guidelines on sizing the cluster, see Sizing Guidelines.


NOTE: If you have upgrades to the Hadoop cluster planned for the next year, you should review those plans with Support prior to installation. For more information, please contact .

Supported Hadoop Distributions

NOTE: The only supports the latest major release and its minor releases of each distribution.

The only supports the versions of any required components included in a supported distribution. Even if they are upgraded components, use of non-default versions of required components is not supported.

 

The supports the following minimum Hadoop distributions:

VendorSupported VersionsLink

Cloudera

  • CDH 6.2 (Release 6.0.1 and later)

  • CDH 6.1 (Release 6.0.1 and later)

  • CDH 6.0 Recommended

    NOTE: CDH 6.x requires that you use the native Spark libraries provided by the cluster. Additional configuration is required. For more information, see Configure for Spark.


  • CDH 5.16 (Release 6.0.1 and later)
  • CDH 5.15
  • CDH 5.14 Recommended
Supported Deployment Scenarios for Cloudera

Hortonworks

 

  • HDP 3.1 (Release 6.0.1 and later)

  • HDP 3.0   Recommended

    NOTE: HDP 3.x requires that you use the native Spark libraries provided by the cluster. Additional configuration is required. For more information, see Configure for Spark.


  • HDP 2.6
  • HDP 2.5

 

Supported Deployment Scenarios for Hortonworks
EMRSee Configure for EMR. 
HDInsightSee Configure for HDInsight. 
Azure DatabricksSee Configure for Azure Databricks. 

Node Requirements

Each cluster node must have the following software:

Hadoop Component Access

The must have access to the following.

Java and Spark version requirements

The following matrix identifies the supported versions of Java and Spark on the Hadoop cluster.

NOTE: In the next release of after Release 6.0, the use of Java 1.7 will no longer be supported. The platform will not work with this version of Java. Please upgrade the and the cluster to use Java 1.8.


Tip: Where possible, you should install the same version of Java on the and on the cluster with which you are integrating.

Notes:

 Spark 2.1Spark 2.2Spark 2.3Spark 2.4
Java 1.7Supported.Not supported.Not supported.Not supported.
Java 1.8Supported.Required.Required.Required.

Other components

Hadoop System Ports

For more information, see System Ports.

Site Configuration Files

Hadoop cluster configuration files must be copied into the . It is especially important in a YARN deployment. See Configure for Hadoop.

Security Requirements

Cluster Configuration

For more information on integration with Hadoop, see Prepare Hadoop for Integration with the Platform.

User Requirements

Users must access the through the Google Chrome browser. For more information on user system requirements, see Desktop Requirements.

I/O Requirements

See Supported File Formats.