Page tree

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.


  • Data ingestion happens only for Spark jobs. Data ingestion does not apply to

    D s photon


  • Data ingestion applies only to JDBC sources that are not native to the running environment. For example, JDBC ingestion is not supported for Hive.
  • Supported for HDFS and other large-scale backend datastores.


  • For a single job with 16 ingest jobs occurring in parallel, maximum expected transfer rate is 1 GB/minute.


  • Good for 100's of MBs. Not good for tables of GB size.
  • 1 ingest job per source, meaning a dataset with 3 sources = 3 ingest jobs.
  • Rule of thumb for max concurrent jobs for a similar edge node:

    Code Block
    max concurrent sources = max cores - cores used for services


    • Above is valid until the network becomes a bottleneck. Internally, the above maxed out at about 15 concurrent sources.
    • Default concurrent jobs = 16, pool size of 10, 2 minute timeout on pool.   This is to prevent overloading of your database.
    • Adding more concurrent jobs once network has bottleneck will start slow down all the transfer jobs simultaneously.
  • If processing is fully saturated (# of workers is maxed): 
    • max transfer can drop to 1/3 GB/minute.
    • Ingest waits for two minutes to acquire a connection. If after two minutes a connection cannot be acquired, the job fails.
  • When job is queued for processing:
    • Job is silently queued and appears to be in progress.
    • Service waits until other jobs complete. 
    • Currently, there is no timeout for queueing based on the maximum number of concurrent ingest jobs.


  • JDBC ingest caching is not supported for Hive.


To enable JDBC ingestion and performance caching, both of the following parameters must be enabled.


NOTE: For new installations, this feature is enabled by default. For customers upgrading to Release 5.1 and later, this feature is disabled by default.

D s config

Parameter NameDescription
webapp.connectivity.ingest.enabledEnables JDBC ingestion. Default is true.

Enables caching of ingested JDBC data.


NOTE: webapp.connectivity.ingest.enabled must be set to true to enable JDBC caching.

When disabled, no caching of JDBC data sources is performed.

When enabled, you can monitor the ingestion of long-loading JDBC datasets through the Import Data page. Default is true.


Tip: After a long-loading dataset has been ingested, importing the data and loading it in the Transformer page should perform faster.



The number of hours that an object can be cached. If the object has not been refreshed in that period of time, the next request for the datasource collects fresh data from the source.

By default, this value is set to 168 (one week).


Maximum size in bytes of the datasource cache. This value applies to individual user caches when either global or user-specific caching is enabled.



Parameter NameDescription

When the logging level is set to debug, log messages on JDBC caching are recorded in the data service log.


NOTE: Use this setting for debug purposes only, as the log files can grow quite large. Lower the setting after the issue has been debugged.

See Logging below.