Release 5.0.1
Page tree


Support | Blog | Contact Us 

Versions Compared

Old Version 2

changes.mady.by.user docuser1

Saved on

compared with

New Version Current

changes.mady.by.user docuser1

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

D toc

For individual users of the 

D s platform
, an administrator may like to submit a custom set of properties to the Hadoop cluster for each job execution. For example, you may wish to change the available Java heap size for users who submit large jobs. This section describes how to define and deploy user-specific Java properties files for cluster jobs.

Info

NOTE: User-specific custom properties override or append any other custom properties that are passed to Hadoop. Suppose the Java properties file contains a single property.

  • If the property is not specified elsewhere in the job definition, it is appended to any other properties that are passed.
  • If the property is specified elsewhere in the job definition, the Java properties file overrides the other custom property value.


Info

NOTE: You cannot specify user-specific properties for S3 jobs.

 

Enable Java properties directory

This feature is enabled by defining the values for the Java properties directory for Spark in the platform configuration.

D s config

 

Steps:

  1. Set the following properties to the directories where the user-specific property files can be stored on the 

    D s item
    itemnode
    . Example:

    Code Block
    "spark.userPropsDirectory": "/opt/trifacta/conf/usr",


  2. Save your changes and restart the platform.

Required Permissions

For the above locations, the 

D s item
itemservice
 requires the following permissions:

  • Execute permission on the directory defined in spark.userPropsDirectory.
  • Read permission on the files in them.

Define user-specific properties files

For each user that is passing in custom properties, a separate file must be created in the appropriate directory with the following filename pattern:

Code Block
userEmail-user.properties

where:

userEmail is the email address for the user that is registered with the 

D s platform
. For example, for userId joe@example.com, the filename is joe@example.com-user.properties. 

File Format:

Each file must follow Java Properties file format, which is the following:

Code Block
property.a=value.a
property.b=value.b


Info

NOTE: Property names must use the full property nameDo not include spark.props in your property names. For example, if you are modifying the Spark YARN queue (spark.props.spark.yarn.queue) for the user, the property name must be spark.props.spark.yarn.queue. Setting it to spark.yarn.queue does not work.

For more information on this format, see https://docs.oracle.com/cd/E23095_01/Platform.93/ATGProgGuide/html/s0204propertiesfileformat01.html.

© 2013-2023 Alteryx® Inc. Privacy Policy  |  Terms of Use