- Write access requires using S3 as the base storage layer. See Set Base Storage Layer.
NOTE: Spark 2.3.0 jobs may fail on S3-based datasets due to a known incompatibility. For details, see https://github.com/apache/incubator-druid/issues/4456.
If you encounter this issue, please set
- On the
, you must install the Oracle Java Runtime Environment for Java 1.8. Other versions of the JRE are not supported. For more information on the JRE, see http://www.oracle.com/technetwork/java/javase/downloads/index.html.
D s node
S3 does not guarantee that at any time the files that have been written to a directory will be consistent with the files available for reading. S3 does guarantee that eventually the files are in sync.
This guarantee is important for some platform jobs that write data to S3 and then immediately attempt to read from the written data.
This timeout defines how long the platform waits for this guarantee. If the timeout is exceeded, the job is failed. The default value is
Depending on your environment, you may need to modify this value.
This value should be the S3 endpoint DNS name value.
If your S3 deployment is either of the following:
Then, you can specify this setting to point to the S3 endpoint for Java/Spark services. This value should be the S3 endpoint DNS name value.
For more information on this location, see https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region.
Restart services. See Start and Stop the Platform.