Page tree

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.


  • Some running environments may perform additional inference on the typing.


    NOTE: During job execution on Spark, inputs of Datetime data type may result in row values being inferred for data type individually. For example, the String value 01/10/2020 may be inferred by date transformations as 1st Oct, 2020 or 10th Jan, 2020. Resulting outputs of Datetime values may not be deterministic in this scenario.

  • Some formatting options may not be supported. 

Differences between 
D s photon
 and Spark running environments

If your Datetime data does not contain time zone information, by default:

  • Spark uses the time zone of the 
    D s item
     for Datetime values.
  • D s photon
     uses the UTC time zone for Datetime values.

This difference in how the values are treated can result in differences in Datetime-based calculations, such as the DATEDIF function.


You can do one of the following:

  • Set the time zone for the 
    D s item
     to be UTC. You must also set the time zone for your Spark running environment to UTC.
  • Apply the following Spark property overrides:

    Code Block
      "props": {
        "spark.driver.extraJavaOptions" : "-Duser.timezone=\"UTC\"",
        "spark.executor.extraJavaOptions" : "-Duser.timezone=\"UTC\""