This section describes some of the configuration options for the JDBC (relational) ingestion, which support faster execution of JDBC-based jobs.

Data ingestion works by streaming a JDBC source into a temporary storage space in the base storage layer to stage the data for job execution. The job can then be run on Photon or Spark. When the job is complete, the temporary data is removed from base storage or retained in the cache (if it is enabled).

Data caching refers to the process of ingesting and storing data sources on the  for a period of time for faster access if they are needed for additional platform operations. 

Tip: Data ingestion and data caching can work together. For more information on data caching, see Configure Data Source Caching.


Job TypeJDBC Ingestion Enabled onlyJDBC Ingestion and Caching Enabled
transformation jobData is retrieved from the source and stored in a temporary backend location for use in sampling.Data is retrieved from the source for the job and refreshes the cache where applicable.
sampling jobSee previous.

Cache is first checked for valid data objects. Outdated objects are retrieved from the data source.

Retrieved data refreshes the cache.

NOTE: Caching applies only to full scan sampling jobs. Quick scan sampling is performed in the running environment.

As needed you can force an override of the cache when executing the sample. Data is collected from the source. See Samples Panel.

Recommended Table Size

Although there is no absolute limit, you should avoid executing jobs on tables over several 100 GBs. Larger data sources can significantly impact end-to-end performance.

NOTE: This recommendation applies to all JDBC-based jobs.

Performance

Rule of thumb:

Scalability:

Enable

To enable JDBC ingestion and performance caching, the first two of the following parameters must be enabled.

NOTE: For new installations, this feature is enabled by default. For customers upgrading to Release 5.1 and later, this feature is disabled by default.

Parameter NameDescription
webapp.connectivity.ingest.enabledEnables JDBC ingestion. Default is true.
feature.jdbcIngestionCaching.enabled

Enables caching of ingested JDBC data.

NOTE: webapp.connectivity.ingest.enabled must be set to true to enable JDBC caching.

When disabled, no caching of JDBC data sources is performed. For more information on caching, see Configure Data Source Caching.
feature.enableLongLoading

When enabled, you can monitor the ingestion of long-loading JDBC datasets through the Import Data page. Default is true.

Tip: After a long-loading dataset has been ingested, importing the data and loading it in the Transformer page should perform faster.

longloading.addToFlowWhen long-loading is enabled, set this value to true to enable monitoring of the ingest process when large relational sources are added to a flow. Default is true. See Flow View Page.
longloading.addToLibrary

When long-loading is enabled, this feature enables monitoring of the ingest process when large relational sources are added to the library. Default is true. See Library Page.

Configure

In the following sections, you can review the available configuration parameters for JDBC ingest.

Configure Ingestion

Parameter NameDescription
batchserver.workers.ingest.max

Maximum number of ingester threads that can run on the at the same time.

batchserver.workers.ingest.bufferSizeBytes

Memory buffer size while copying to backend storage.

A larger size for the buffer yields fewer network calls, which in rare cases may speed up ingest.

batch-job-runner.cleanup.enabled

Clean up after job, which deletes the ingested data from backend storage. Default is true.

NOTE: If JDBC ingestion is disabled, relational source data is not removed from platform backend storage. This feature can be disabled for debugging and should be re-enabled afterward.

NOTE: This setting rarely applies if JDBC ingest caching has been enabled.

Logging

Parameter NameDescription
data-service.systemProperties.logging.level

When the logging level is set to debug, log messages on JDBC caching are recorded in the data service log.

NOTE: Use this setting for debug purposes only, as the log files can grow quite large. Lower the setting after the issue has been debugged.

See Logging below.

Monitoring Progress

You can use the following methods to track progress of ingestion jobs.

Logging

During and after an ingest job, you can download the job logs through the Jobs page. Logs include:

See Jobs Page.