Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r092

...

Info

NOTE: Where possible, you should define or select a user with a userID value greater than 1000. In some environments, lower userID values can result in failures when running jobs on Hadoop.

 


Set the Hadoop username 

D s defaultuser
Typehadoop
Fulltrue
 for the 
D s platform
 to use for executing jobs:

...

  1. Verify that the snappy and snappy-devel packages have been installed on the 

    D s node
    . For more information, see https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/NativeLibraries.html.

  2. From the 

    D s node
    , execute the following command:

    Code Block
    hadoop checknative
  3. The above command identifies where the native libraries are located on the 
    D s node
  4. Cloudera: 
    1. On the cluster, locate the libsnappy.so file. Verify that this file has been installed on all nodes of the cluster, including the 
      D s node
      . Retain the path to the file on the 
      D s node
      .
    2. D s config
    3. Locate the spark.props configuration block. Insert the following properties and values inside the block:

      Code Block
      "spark.driver.extraLibraryPath": "/path/to/file",
      "spark.executor.extraLibraryPath": "/path/to/file",
    Hortonworks:
    1. Verify on the 

      D s node
       that the following locations are available:

      Info

      NOTE: The asterisk below is a wildcard. Please collect the entire path of both values.

      Code Block
      /hadoop-client/lib/snappy*.jar
      /hadoop-client/lib/native/
    2. D s config
    3. Locate the spark.props configuration block. Insert the following properties and values inside the block:

      Code Block
      "spark.driver.extraLibraryPath": "/hadoop-client/lib/snappy*.jar;/hadoop-client/lib/native/",
      "spark.executor.extraLibraryPath": "/hadoop-client/lib/snappy*.jar;/hadoop-client/lib/native/",
  5. Save your changes and restart the platform.
  6. Verify that the /tmp directory has the proper permissions for publication. For more information, see Supported File Formats.

...