D toc |
---|
A visual profile can be optionally generated as part of execution of a job. While profiling can extend the time it takes to complete the job, a visual profile can provide key statistical information on the columns of your data and the overall dataset itself. This section provide information on each of the available profilers and how to configure them.
Info |
---|
NOTE: To enable use of a profiler, the running environment on which it is hosted must also be enabled. See Running Environment Options. |
The
D s platform | ||
---|---|---|
|
Overview
Type of Profiler | Requires Hadoop cluster? | Supported Running Environment | Description | |||
---|---|---|---|---|---|---|
Scala Spark Profiler | Yes | all | Generates a profile on the
See Configure for Spark. | |||
Photon Profiler | No | Photon | Default profiler when Photon running environment is enabled, and the Scala Spark Profiler has not been enabled. |
Configure
Making changes to your profiling type:
- Apply the configuration changes listed below.
- Save your changes and restart the platform.
- Restart your browser and login again.
Run Profiling as a Second Job in Spark
By default, profiling jobs execute on the running environment where the transformation job was executed. You can configure the
D s server |
---|
Steps:
D s config Locate the following property and set its value to
true
:Code Block "photon.runProfileWithSpark": false,
- Save your changes and restart the platform.
When profiling is selected for jobs executing on the
D s server |
---|
Disable Profiling Option
Profiling is invoked at job execution time by the user. See See Run Job Page.
To disable user choice through the UI, set the following parameter:
Code Block |
---|
"profiler.userOption": false, |
When the above setting is disabled:
- Any available checkbox no longer works. User cannot choose whether to profile or not.
- Profiles are always executed.
Info |
---|
NOTE: This setting does not affect profiling through the APIs. Profiling can always be enabled or disabled for jobs that are executed via API. |
To disable user choice through the UI, set the following parameter:
Code Block |
---|
"profiler.userOption": false, |
Info |
---|
NOTE: Profiling can still be enabled for jobs that are executed via the CLI or API. |