Page tree

 

Support | BlogContact Us | 844.332.2821

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r0411

D toc
Excerpt

Unpacks nested data from an array Array or map Object column to create new rows or columns based on the keys in the source data.   

This transform works differently on columns of Array or Map Object type.

The  unnest  transform must include keys that you specify as part of the transform step. To unnest a column of array data that contains no keys, use the flatten transform. See Flatten Transform.

...

D s
snippetBasic

D code

unnest col: myMap myObj keys:'sourceA','sourceB' pluck:true markLineage:true

Output:

  • Extracts from the myMapmyObj column the corresponding values for the keys sourceA and sourceB into two new columns. 
  • Since markLineage is true, these new column names are prepended with the source name: sourceA_column1 and sourceB_column2
  • Any non-missing values from the source columns are added to the corresponding new columns and are removed from the source column, since pluck is true.

...

This parameter has different syntax to use for single-level and multi-level nested data. There are also variations in syntax between Map Object and Array data type.

D s
snippetusage

Required?Data Type
Yes

Comma-separated String values.

Syntax examples are provided below.

Keys for

...

Object data - single-level

Info

NOTE: Key names are case-sensitive.

For a single, top-level key in a Map fieldan Object field, you can specify the key as a simple quoted string:

D code

unnest col:myCol keys: 'myMapKeymyObjKey'

The above looks for the key myMapKey myObjKey among the top-level keys in the map Object and returns the corresponding value for the new column. You can also bracket this key in square brackets:

D code

unnest col:myCol keys: '[myMapKeymyObjKey]'

To specify multiple first-level keys, use the following:

D code

unnest col:myCol keys:'myMapKeymyObjKey','my2ndMapKeymy2ndObjKey'

The above generates two new columns ( myMapKey myObjKey and my2ndMapKeymy2ndObjKey) containing the corresponding values for the keys.

Keys for

...

Object data - multi-level

You can also reference keys that are below the first level in the MapObject

Example data:

Code Block
{ "Key1" :
  { "Key1A" :
    { "Key1A1" : "Value1" }
  }
}
{ "Key2" :
  { "Key2A" :
    { "Key2A1" : "Value2" }
  }
}
{ "Key3" :
  { "Key3A" :
    { "Key3A1" : "Value3" }
  }
}

...

The following example nested Array data matches the structure of the Map Object data in the previous example:

...

Required?Data Type
NoBoolean


D s
snippetExamples

Example - Unnest

...

an Object

You have the following dataset. The Sizes column contains map Object data on available sizes. 

...

Info

NOTE: Depending on the format of your source data, you might need to perform some replacements in the Sizes column in order to make it inferred as proper Map Object type values. The final format should look like the above.

If it is not inferred already, set the type of the Sizes column to MapObject:

D code

settype col: Sizes type: 'MapObject'

Unnest the data into separate columns. The following prepends Sizes_ to the newly generated column name.

...

If all values have been processed, the the  Sizes column now contains a set of maps missing data. You can use use the following to determine if the length of the remaining data is longer than two characters. This transform is a good one to just preview:

D code

derive value:(LEN(Sizes) > 2) as:'len_Sizes'

...

ProdIdProdNameSizes_SmallSizes_MediumSizes_LargeSizes_Extra-Large
1001HatNYYY
1002ShirtNYYN
1003PantsYYYN

Example - Unnest an array

The following example demonstrates differences between the unnest and the flatten transform, including how you use unnest to flatten array data based on specified keys.

...