D toc |
---|
The
D s webapp | ||
---|---|---|
|
D s item | ||
---|---|---|
|
By default, the Photon running environment is enabled for new installations. Additional configuration is required.
Features:
- Faster execution times for transform and profiling jobs
Larger sample sizes (up to 10MB by default).
Info NOTE: Since Photon supports larger sample sizes, some interactions may be slightly impacted. Loading states have been introduced to enable faster responsiveness from the application.
Tip Tip: For datasets that are smaller than the sample size limit, the Transformer Page displays the entire dataset in its transformed state. So, you can download the dataset from the Recipe Panel in Transformer Page without having to execute it on a remote server. This expanded capability allows for faster and more immediate local use of the product. See Recipe Panel.
- Better consistency with typecasting done in Hadoop jobs
This section provides information on how to enable and configure the Photon running environment.
Known Limitations and Issues
Info |
---|
NOTE: For profiles executed in the Photon running environment, percentages for valid, missing, or mismatched column values may not add up to 100% due to rounding. |
Desktop Requirements
Info |
---|
NOTE: To interact with the Photon running environment, all desktop instances of Google Chrome must have the PNaCl component enabled and updated to the minimum supported version. See Desktop Requirements. |
Example Configuration
The following are the default values for Photon enablement:
Code Block |
---|
"photon": { "cacheEnabled": true, "numThreads": 4, "enabled": true, "distroPath": "/photon/dist/centos6/photon", "loadScalingFactor": 20, "traceExecution": false, "websocket": { "host": "localhost", "port": 8082 }, "mode": "pnacl" }, |
Parameter | Description | ||
---|---|---|---|
cacheEnabled | Debugging setting. Leave the default value. | ||
numThreads | Maximum number of threads permitted to the Photon process. See below for recommended values. | ||
enabled | Verifiy that this value to | ||
distroPath | Please verify that this property is set to the following value, which works for all operating system distributions:
| ||
loadScalingFactor | Used in conjunction with other parameters to define the maximum size of samples for Photon. For more information, see Sample Size below. | ||
traceExecution | Debugging setting. Leave the default value. | ||
websocket.host | Hostname of the web socket service. Leave this value as localhost . | ||
websocket.port | Port number of the web socket service. Default value is 8082 . | ||
mode | Set this value is This parameter must be properly set to enable Photon. See Enable Photon below. |
Recommended Photon Configuration by Core Count
On the
D s item | ||
---|---|---|
|
Parameter | 8 cores | 16 cores (default) | 32 cores |
---|---|---|---|
webapp.numProcesses | 2 | 2 | 5 |
vfs-service.numProcesses | 2 | 2 | 3 |
photon.numThreads | 2 | 4 | 4 |
batchserver.workers.photon.max | 2 | 2 | 4 |
The number of simultaneous users is a competing factor.
- For a high number, more resources should be reserved for the webapp and the VFS services.
- For a low number, more resources for Photon should improve performance for sampling and job execution on the Photon running environment.
The following table illustrates some adjustments for a 16-core system:
Parameter | 16 cores (default) | Low number of simultaneous users | High number of simultaneous users |
---|---|---|---|
webapp.numProcesses | 2 | 1 | 4 |
vfs-service.numProcesses | 2 | 1 | 4 |
photon.numThreads | 4 | 4 | 4 |
batchserver.workers.photon.max | 2 | 2 | 2 |
Enable Photon
Photon isDisable
This running environment is enabled by default. Please verify complete the following configuration to enable the Photon running environmentdisable the running environment.
Info |
---|
NOTE: A cluster-based running environment, such as Spark, must be available for processing jobs when this one is disabled. |
Steps:
D s config To enable Photondisable the
, apply the following configuration settings:D s server Code Block "photonwebapp.enabledrunInTrifactaServer": truefalse, "photonfeature.modeenableSamplingScanOptions": "pnacl"false, "feature.enableFirstRowsSample": false,
Do not change the following, which applies to the Photon web-client:
Code Block "photon.enabled": true,
Save your changes and restart the platform.
Change Limits
Info |
---|
NOTE: Increasing these values can have a significant impact on load times and performance. Change these values only if you are experiencing difficulties. Make incremental changes. |
Sample Size Limits
Increasing the sample size may degrade the user experience in the Transformer page in the following ways:
- Generation of column details and data grid histograms
- Preview card loading time
Time required to complete brushing and linking in histograms
Info NOTE: If you increase the sample size above the default setting and encounter unacceptable performance in the above areas, you should reduce the sample size settings.
When samples are created using the Photon running environment, their maximum size is determined by multiplying the values for the following settings. Default value is 10 MB.
Setting | webapp.client.loadLimit | photon.loadScalingFactor | Total |
---|---|---|---|
Default Value | 512000 | 20 | 10240000 |
Maximum Data in the Client
The following settings determine the maximum amount of data that is permitted to be passed to the client from Photon:
Setting | webapp.client.maxResultsBytes | photon.loadScalingFactor | Total |
---|---|---|---|
Default Value | 2097152 | 20 | 41943040 |
Timeouts
Photon runtime job timeout
By default, the
D s platform |
---|
Steps:
D s config Code Block "photon.timeoutEnabled": false, "photon.timeoutMinutes": 180,
Setting Description timeoutEnabled
Set to false
to disable job limiting. Set totrue
to enable the timeout specified below.timeoutMinutes
Defines the number of minutes that a Photon job is permitted to run. Default value is 180
(three hours).- Save your changes and restart the platform.
When a job has failed due to exceeding a timeout, additional information is available in the job logs. The following is a good search term for this type of error:
Code Block |
---|
java.lang.Exception: Photon job '<jobId>' timeout |
where <jobId>
is the internal job identifier.
Job logs can be downloaded from the Job page. See Jobs Page.
Photon memory timeout
To prevent crashes of the server, Photon imposes a memory consumption limit for each job. If this memory timeout is exceeded, the job is automatically killed. As needed, you can disable this memory protection (not recommended) or change the memory threshold when jobs are killed.
Steps:
D s config Locate the following settings:
Code Block "photon.memoryMonitorEnabled": false, "photon.memoryPercentageThresold": 60,
Setting Description memoryMonitorEnabled
Set to false
to disable memory monitoring. Set totrue
to enable the threshold specified below.memoryPercentageThreshold
Defines the percentage of total available system memory that a Photon job process is permitted to consume. Default value is
60
(60%).Tip Tip: This threshold applies to individual Photon jobs. If this threshold value is over 50%, it is possible for two concurrent Photon jobs to use more than the available memory, crash the server, and force a restart. You may wish to start by setting threshold values at a lower level.
- Save your changes and restart the platform.
When a job has failed due to exceeding this memory threshold, additional information is available in the job logs. The following is a good search term for this type of error:
Code Block |
---|
java.lang.Exception: Photon job '<jobId>' failed with memory consumption over threshold |
where <jobId>
is the internal job identifier.
Below this line item, you may see the following entries, which can provide additional information to adjust the memory settings:
Code Block |
---|
2017-05-04T02:26:40.549Z [job-id 740] com.trifacta.joblaunch.util.ProcessMonitorUtil [Thread-20] INFO com.trifacta.joblaunch.util.ProcessMonitorUtil - Global memory size: 8373186560 bytes 2017-05-04T02:26:40.555Z [job-id 740] com.trifacta.joblaunch.util.ProcessMonitorUtil [Thread-20] INFO com.trifacta.joblaunch.util.ProcessMonitorUtil - Available global memory size at process start: 2672959488 bytes ... 2017-05-04T02:29:15.690Z [job-id 740] com.trifacta.joblaunch.util.ProcessMonitorUtil [Thread-20] INFO com.trifacta.joblaunch.util.ProcessMonitorUtil - Current memory consumption: 5.614080429077148% 2017-05-04T02:29:15.691Z [job-id 740] com.trifacta.joblaunch.util.ProcessMonitorUtil [Thread-20] ERROR com.trifacta.joblaunch.util.ProcessMonitorUtil - Average memory consumption for the past 15 seconds over 5% threshold: 5.174326801300049 %. Current available global memory: 2244628480 bytes |
Item | Description |
---|---|
Global memory size | Total available global memory in bytes |
Available global memory size at process start | Total available memory in bytes when the job is launched |
Current memory consumption | Current memory usage for the job process as a percentage of the total. This metric is posted to the log every 30 seconds and can be used to debug memory leaks. |
Average memory consumption for the past 15 seconds over x% threshold | When the job fails due to the memory threshold, this metric identifies the average memory consumption percentage over the past 15 seconds. The defined threshold percentage is included. |
Current available global memory | When the job fails, this metric identifies the total available memory at the time of failure. |
Job logs can be downloaded from the Job page. See Jobs Page.
Batch FileSystem Access Timeout Settings
The default timeout settings for reading and writing of data from the client browser through Photon should work in most cases.
Particularly when reading from large tables, you might discover errors similar to the following:
Code Block |
---|
06:21:21.365 [Job 23] INFO com.trifacta.hadoopdata.photon.BatchPhotonRunner - terminating with uncaught exception of type Poco::TimeoutException: Timeout 06:21:21.375 [Job 23] INFO com.trifacta.hadoopdata.photon.BatchPhotonRunner - /vagrant/photon/dist/centos6/photon/bin/photon-cli: line 22: 15639 Aborted $ Unknown macro: {command[@]} |
Steps:
D s config Locate the
photon.extraCliArgs
node.Add the following values to the
extraCliArgs
entry:Code Block "photon.extraCliArgs" : "-batch_vfs_read_timeout <300> -batch_vfs_write_timeout <300>"
Argument Description -batch_vfs_read_timeout
Timeout limit in seconds of read operations from the datastore. Default value is
300
seconds (5 minutes).Tip Tip: Raising the value to
3600
seconds should be fine in most environments. Avoid setting this value above7200
seconds (2 hours).-batch_vfs_write_timeout
Timeout limit in seconds of write operations to the datastore. Default value is
300
seconds (5 minutes).Info NOTE: Do not modify unless specifically instructed by
.D s support - To reduce timeouts, raise the above settings.
- Save your changes and restart the platform.
Configure VFS Service
The VFS Service serves the front-end interface and brokers connections with the backend datastores when the Photon running environment is enabled.
Info |
---|
NOTE: The VFS service must be enabled when Photon is enabled. |
Steps:
D s config Locate the following configuration:
Code Block "vfs-service.port":41913, "vfs-service.loggerOptions.silent":false, "vfs-service.loggerOptions.level":"info", "vfs-service.loggerOptions.json":false, "vfs-service.loggerOptions.format":":method :url :status :res[content-length] :response-time :referrer :remote-addr :trifacta-user :user-agent", "vfs-service.host":"localhost", "vfs-service.enabled":true, "vfs-service.bindHost":"0.0.0.0", "vfs-service.autoRestart":true,
- Verify that the
enabled
parameter is set totrue
. - Additional configuration settings are described below.
- Save your changes and restart the platform.
Parameter | Description | ||||
---|---|---|---|---|---|
port | Port number that VFS service uses to communicate. Default value is
| ||||
loggerOptions.silent | When set to true , messages are suppressed in the user interface. | ||||
loggerOptions.level | Supported logging levels:
| ||||
loggerOptions.json | When set to | ||||
loggerOptions.format | If needed, you can re-order the fields that are included in each log message. | ||||
host | Host of the VFS Service. Leave this value as localhost . | ||||
enabled | Set this value to true to enable the VFS Service. | ||||
bindHost | Do not modify this value. | ||||
autoRestart | When set to This value should be set to |
Use Photon
When Photon is enabled, it is available like any other running environment in the application. When executing a job, select the Run on
option from the drop-down in the Run Job dialog. D s server
Info |
---|
NOTE: Before you test, please be sure to complete all steps of Required Platform Configuration. |