Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version next

...

  • Some running environments may perform additional inference on the typing.

    Info

    NOTE: During job execution on Spark, inputs of Datetime data type may result in row values being inferred for data type individually. For example, the String value 01/10/2020 may be inferred by date transformations as 1st Oct, 2020 or 10th Jan, 2020. Resulting outputs of Datetime values may not be deterministic in this scenario.

  • Some formatting options may not be supported. 

Differences between 
D s photon
 and Spark running environments

If your Datetime data does not contain time zone information, by default:

  • Spark uses the time zone of the 

    D s item
    itemnode
     for Datetime values.

    Tip

    Tip: This use of time zone applies to any Spark-based running environment, such as EMR.

  • D s photon
     uses the UTC time zone for Datetime values.

This difference in how the values are treated can result in differences in Datetime-based calculations, such as the DATEDIF function.

Workarounds:

You can do one of the following:

  • Set the time zone for the 
    D s item
    itemnode
     to be UTC. You must also set the time zone for your Spark running environment to UTC.
  • Apply the following Spark property overrides:

    Code Block
    "spark": 
      "props": {
        ...
        "spark.driver.extraJavaOptions" : "-Duser.timezone=\"UTC\"",
        "spark.executor.extraJavaOptions" : "-Duser.timezone=\"UTC\""
      }
      ...
    }

    For more information, see Spark Execution Properties Settings.

...