Improvements to the Type System

...

This section provides information on improvements to the

D s item

item	type system
r	true

.

Release 8.2.2

Snowflake date publishing improvements

In prior releases, the

D s webapp

did not fully support Datetime values on publication. It published Date type values to Snowflake as follows:

Publishing Action→	Create/Drop Table	Append/Truncate
TargetTable→		Date	Datetime
Date	DateTime Appends 00:00:00	Append 00:00:00	Append 00:00:00

Beginning in this release, the

D s webapp

publishes Date values as follows:

Publishing Action	Create/Drop	Append/Truncate
TargetTables→		Date	Datetime
Date	Date	Date	Datetime Appends 00:00:00

For more information, see Snowflake Data Type Conversions.

Release 8.2

None.

Release 8.1

Running environments now support output of Datetime values as Datetime/Timestamp values in Parquet files

...

Also, the constraints around valid 16-digit numbers have been improved with better recognition around values for different types of credit cards. In the following table, you can see lists of valid test numbers for different credit card types and can see how detection of these values has changed between releases.

Test Number	Credit Card Type	Is Detected 7.4?	Is detected 7.5?
2222 4053 4324 8877	Mastercard	No	Yes
2222 9909 0525 7051	Mastercard	No	Yes
2223 0076 4872 6984	Mastercard	No	Yes
2223 5771 2001 7656	Mastercard	No	Yes
5105 1051 0510 5100	Mastercard	Yes	Yes
5111 0100 3017 5156	Mastercard	Yes	Yes
5204 7400 0990 0014	Mastercard	Yes	Yes
5420 9238 7872 4339	Mastercard	Yes	Yes
5455 3307 6000 0018	Mastercard	Yes	Yes
5506 9004 9000 0444	Mastercard	Yes	Yes
5553 0422 4198 4105	Mastercard	Yes	Yes
5555 5555 5555 4444	Mastercard	Yes	Yes
4012 8888 8888 1881	Visa	Yes	Yes
4111 1111 1111 1111	Visa	Yes	Yes
6011 0009 9013 9424	Discover	Yes	Yes
6011 1111 1111 1117	Discover	Yes	Yes
3714 496353 98431	American Express	Yes	Yes
3782 822463 10005	American Express	Yes	Yes
3056 9309 0259 04	Diners	No	Yes
3852 0000 0232 37	Diners	No	Yes
3530 1113 3330 0000	JCB	Yes	Yes
3566 0020 2036 0505	JCB	Yes	Yes

For more information, see Credit Card Data Type.

...

Where there are mismatches between inputs and the expected input data type, the following values are generated for the mismatches:

Source data type	Output if mismatched
Primitive data types: Integer Decimal Boolean Arrays Maps	null value, if mismatched
Datetime	null value, if mismatched
Other non-primitive data types, including: SSN Phone Number Email Address Credit Card Gender IP Address URL HTTP Code Zip Code	Converted to string values, if mismatched
String	Anything can be a String value.

Three-value logic for null values

...

For example, you have the following dataset:

MyStringCol
This works.
You can't break this.
Not broken yet.

You test each row for the presence of the string can't:

...

The above transform results in the following:

MyStringCol	MyFindResults
This works.
You can't break this.	true
Not broken yet.

In this case, the value of false is not written to the other columns, since the find function returns a null value. This null value, in turn, nullifies the entire expression, resulting in a null value written in the new column.

...

Beginning in this release, the colon (:) is no longer supported as a delimiter for date values. It is still supported for time values.

myDateValue	Recognized?
02:03:2016	No
02:03:16	Recognized as a time value

When data such as the above is imported, it may not be initially recognized by the

D s webapp

as Datetime type.

...

Page tree

Versions Compared

Old Version 2

New Version Current

Key