Page tree

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r0682


  • Good for 100's of MBs. Not good for tables of GB size.
  • 1 ingest job per source, meaning a dataset with 3 sources = 3 ingest jobs.
  • Rule of thumb for max concurrent jobs for a similar edge node:

    Code Block
    max concurrent sources = max cores - cores used for services


    • Above is valid until the network becomes a bottleneck. Internally, the above maxed out at about 15 concurrent sources.
    • Default concurrent jobs = 16, pool size of 10, 2 minute timeout on pool.   This is to prevent overloading of your database.
    • Adding more concurrent jobs once network has bottleneck will start slow down all the transfer jobs simultaneously.
  • If processing is fully saturated (# of workers is maxed): 
    • max transfer can drop to 1/3 GB/minute.
    • Ingest waits for two minutes to acquire a connection. If after two minutes a connection cannot be acquired, the job fails.
  • When job is queued for processing:
    • Job is silently queued and appears to be in progress.
    • Service waits until other jobs complete. 
    • Currently, there is no timeout for queueing based on the maximum number of concurrent ingest jobs.


  • JDBC ingest caching is not supported for Hive.


To enable JDBC ingestion and performance caching, both of the following parameters must be enabled.


NOTE: For new installations, this feature is enabled by default. For customers upgrading to Release 5.1 and later, this feature is disabled by default.

D s config

Parameter NameDescription
webapp.connectivity.ingest.enabledEnables JDBC ingestion. Default is true.

Enables caching of ingested JDBC data.


NOTE: webapp.connectivity.ingest.enabled must be set to true to enable JDBC caching.

When disabled, no caching of JDBC data sources is performed.

When enabled, you can monitor the ingestion of long-loading JDBC datasets through the Import Data page. Default is true.


Tip: After a long-loading dataset has been ingested, importing the data and loading it in the Transformer page should perform faster.