Improvements to the Type System

As needed, you can modify the limits that the Trifacta Application uses during the data type and split row inference processes. For more information, see Configure Type Inference.

Where there are mismatches between inputs and the expected input data type, the following values are generated for the mismatches:

State values and custom data types are converted to string values, if they are mismatched.

The Trifacta Photon running environment has been augmented to use three-value logic for null values.

When values are compared, the result can be true or false in most cases.

If a null value was compared to a null value in the Trifacta Photon running environment:

This change aligns the behavior of the running environment with that of SQL and Hadoop Pig.

Assume that the column nuller contains null values and that you have the following transform:

derive value:(nuller >= 0)

Prior to Release 3.1, the above transform generated a column of true values.

In Release 3.1 and later, the transform generates a column of null values.

In the following example, a_null_expression always evaluates to a null value.

derive value: (a_null_expression ? 'a' : 'b')

In Release 3.0, this expression generated b for all inputs on the Trifacta Photon running environment and a null value on Hadoop Pig.

In Release 3.1 and later, this expression generates a null value for all inputs on both running environments.

For example, you have the following dataset:

You test each row for the presence of the string can't:

derive value: if(find(MyStringCol, 'can\'t',true,0) > -1, true, false) as:'MyFindResults'

The above transform results in the following:

In this case, the value of false is not written to the other columns, since the find function returns a null value. This null value, in turn, nullifies the entire expression, resulting in a null value written in the new column.

You can use the following to locate the null values:

derive value:isnull(MyFindResults) as:'nullInMyFindResults'

Before this release, you could create a transform like the following:

derive value:date(2016,2,15)

This transform generated a column of map values, like the following:

{"year":"2016","month":"2","date":"15"}

Beginning this release, the above command is invalid, as the date values must be properly formatted prior to display. The following works:

derive value:dateformat(date(2016,2,15),'yyyy-MM-dd')

This transform generates a column of Datetime values in the following format:

2016-02-15

Time:

Before this release:

derive value:time(11,34,58)

Prior release output:

{"hours":"11","minutes":"34","seconds":"58"}

This release:

derive value:dateformat(time(11,34,58), 'HH-mm-ss')

This release's output:

11-34-58

Beginning in this release, the unixtimeformat and dateformat functions requires an AM/PM indicator (a) if the date formatting string uses a 12-hour time indicator (h or hh).

Valid for earlier releases:

derive value: unixtimeformat(myDate, 'yyyy-MM-dd hh:mm:ss') as:'myUnixDate'

Valid for this release and later:

derive value: unixtimeformat(myDate, 'yyyy-MM-dd hh:mm:ss a') as:'myUnixDate'

These references in recipes fail to validate in this release or later and must be fixed.

If a formatting string is not a datetime format recognized by Designer Cloud Powered by Trifacta Enterprise Edition, the output is generated as a string value.

This change was made to provide clarity to some ambiguous conditions.

Beginning in this release, the colon (:) is no longer supported as a delimiter for date values. It is still supported for time values.

When data such as the above is imported, it may not be initially recognized by the Trifacta Application as Datetime type.

To fix, you might apply the following transform:

replace col:myDateValue with:'-' on:`-` global:true

The new column values are more likely to be inferred as Datetime values. If not, you can choose the appropriate Datetime format from the data type drop-down for the column. See Data Grid Panel.