Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r0821

D toc

Import

When data is imported:

  • Supported data types from the source are converted to corresponding data types supported by the application, based upon the conversions listed in this section.
  • Types that are not supported but are recognized by the application are converted to String types.
  • Data for types that cannot be read from the source due to technical reasons are converted to null values on import.

Type Inference

By default, the 

D s webapp
 applies type inference for imported data. The application attempts to infer a column's appropriate data type in the application based on a review of the first lines in the sample.

Info

NOTE: Mapping source data types to

D s item
itemdata types
depends on a sufficient number of values that match the criteria of the internal data type. The mapping of import types to internal data types depends on the data.

  • Type inference needs a minimum of 25 rows of data in a column to work consistently.

  • If your dataset has fewer than 20 rows, type inference may not have sufficient data to properly infer the column type.

In some datasets, the first 25 rows may be of a data type that is a subset of the best matching type. For example, if the first 25 rows in the initial same match the Integer data type, the column may be typed as Integer, even if the other 2,000 rows match for the Decimal data type. If the column data type is unmodified:

  • The data is written out from
    D s product
    as Integer data type. This works for the first 25 rows.
  • The other 2,000 rows are written out as null values, since they do not match the Integer data type. If the source data used decimal notation (e.g. 3.0 in the source), then those values are written out as null values, too.

In this case, it may be easier to disable type inference for this dataset. See below.

Tip

Tip: If you are having trouble getting your imported dataset to map to expected data types, you can disable type inference for the individual dataset. For more information, see Import Data Page.

Strong typecasting:

After data is imported, 

D s webapp
 provides some mechanisms for applying stronger typecasting to the data. Example:

  • If all input values are double-quoted, then 
    D s product
     evaluates all columns as String type. As a result, type inference cannot be applied. 
  • Since non-String data types cannot be inferred, then the first row cannot be detected as anomalous against the inferred type (String). Column headers cannot be automatically detected from double-quoted source files.

Solutions:

  • After data has been imported, you can remap individual column types through recipe steps. For more information, see Change Column Data Type.
    In the preceding example, you can apply functions to the column data to parse the values as specific data types. You can parse String values as other data types using the following functions:

    FunctionDescription
    PARSEINT Function

    D excerpt include
    pagePARSEINT Function
    nopaneltrue

    PARSEFLOAT Function

    D excerpt include
    pagePARSEFLOAT Function
    nopaneltrue

    PARSEBOOL Function

    D excerpt include
    pagePARSEBOOL Function
    nopaneltrue

    PARSEDATE Function

    D excerpt include
    pagePARSEDATE Function
    nopaneltrue

Export

On export from the

D s webapp
:

  • The application maps the internal 
    D s item
    itemdata type
     to the explicit type listed in the appropriate page in this section.
  • Unmapped types are converted to the equivalent of strings.

Tip

Tip: You can import a target schema to assist in lining up your columns with the expected target. For more information, see Overview of RapidTarget.


D generate child excerpts
pagesSupported Data Types
heading2
linktrue

For more information on the data types that are supported within the 

D s webapp
, see Supported Data Types.