...
For an expected number of peak concurrent (active) users, P, set the following parameters.
| D s config |
|---|
| Parameter | Default Value | Recommended Value | Example |
|---|---|---|---|
webapp.numProcesses | 2 | P / 15, rounded up to the nearest integer, minimum of 2 | for P = 40, set to 3 |
webapp. |
database.pool.maxConnections | 10 | 5 * webapp.numProcesses | for webapp.numProcesses = 3, set to 15 |
vfs-service.numProcesses | 2 | P / 50, rounded up to the nearest integer, minimum of 2 | for P = 225, set to 5 |
Other applicable parameters
...
| Info | |
|---|---|
NOTE: Avoid modifying these parameters unless instructed by
|
| Parameter | Default Value |
|---|
batch-job-runner.executor.maxPoolSize 50batch-job-runner.db.maxPoolSize50batch-job-runner.systemProperties.httpMaxConnectionsPerDestination | 50 |
Limit Application Memory Utilization
Several
| D s node |
|---|
-Xmx value. These can be limited by modifying the following parameters:| Parameter | Default Configuration |
|---|---|
| -Xmx1024m |
diagnostic-server.jvmOptions | -Xmx512m |
data-service.jvmOptions | -Xmx128m |
spark-job-service.jvmOptions | -Xmx128m |
Other services have low memory requirements.
...
Jobs run on mthe
| D s node |
|---|
| D s node |
|---|
| Parameter | Description | Default |
|---|---|---|
batchserver.workers.photon.max | Maximum number of simultaneous Photon processes; once exceeded, jobs are queued | 2 |
photon.numThreads | Number of threads used by each Photon process | 4 |
Increasing the number of concurrent processes allows more users' jobs to execute concurrently. However, it also leads to resource contention among the jobs and the application services.
Photon's execution is purely in memory. It does not spill to disk when the total data size exceeds available memory. As a result, you should configure limits on Photon's memory utilization. If a job exceeds the configured memory threshold, it is killed by a parent process tasked with monitoring the job.
| Parameter | Description | Default |
|---|---|---|
photon.memoryMonitorEnabled | Set true to enable the monitor, and set to false (limited only by operating system) otherwise | true |
photon.memoryPercentageThreshold | Percentage of available system memory that each Photon process can use (if photon.memoryMonitorEnabled is true) | 50 |
| Tip |
|---|
A reasonable rule of thumb: the input data size should not exceed one tenth of the job’s memory limit. This rule of thumb accounts for joins and pivots and other operations that can increase memory usage over the data size. However, this parameter is intended as a safeguard; it is unlikely that all running jobs would approach the memory limit simultaneously. So you should "oversubscribe" and use slightly more than |
In addition to a memory threshold, execution time of any Photon job can also be limited via the following parameters:
| Parameter | Description | Default |
|---|---|---|
photon.timeoutEnabled | Set true to enable the timeout, and set to false (unlimited) otherwise | true |
photon.timeoutMinutes | Time in minutes after which to kill the Photon process (if photon.timeoutEnabled is true) |
|
For more information, see Configure Photon Running Environment.
