...

• Some running environments may perform additional inference on the typing.

  Info
  NOTE: During job execution on Spark, inputs of Datetime data type may result in row values being inferred for data type individually. For example, the String value 01/10/2020 may be inferred by date transformations as 1st Oct, 2020 or 10th Jan, 2020. Resulting outputs of Datetime values may not be deterministic in this scenario.

  
  

• Some formatting options may not be supported. 

Differences between 

D s photon

 and Spark running environments

If your Datetime data does not contain time zone information, by default:

• Spark uses the time zone of the 

  D s item
  item node

   for Datetime values.

• D s photon

   uses the UTC time zone for Datetime values.

This difference in how the values are treated can result in differences in Datetime-based calculations, such as the DATEDIF function.

Workarounds:

You can do one of the following:

• Set the time zone for the 

  D s item
  item node

   to be UTC. You must also set the time zone for your Spark running environment to UTC.

• Apply the following Spark property overrides:
  
  

  Code Block
  "spark": "props": { ... "spark.driver.extraJavaOptions" : "-Duser.timezone=\"UTC\"", "spark.executor.extraJavaOptions" : "-Duser.timezone=\"UTC\"" } ... }