Page tree

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r085


  • All datasources and all outputs specified in a job must be located within BigQuery.
  • D s dataflow
     must be selected as running environment.
  • Custom SQL datasets are not supported.
  • All recipe steps, including all

    D s lang
    functions in the recipe, must be translatable to SQL. 


    NOTE: When attempting to execute a job in BigQuery,

    D s webapp
    executes each recipe in BigQuery, until it reaches a step that cannot be executed there. At that point, data is transferred to
    D s dataflow
    , where the remainder of the job is executed.

  • Some transformations and functions are not currently supported for execution in BigQuery. See below.
  • Upserts, merges, and deletes are not supported for full execution in BigQuery.
  • Sampling jobs are not supported for execution in BigQuery.
  • If your recipe includes data quality rules, the job cannot be fully executed in BigQuery.

NOTE: In the new BigQuery console, you may not be able to see the SQL that was used to execute your job in BigQuery. This is a known issue with the new BigQuery console.

D s lang
 for BigQuery Execution


  • Regex patterns used must be valid RE2. Operations on non-RE2 regex patterns are not pushed down.
  • For more information on limitations on specific push-downs, see Flow Optimization Settings Dialog.

Unsupported data types

The following data types are not supported for execution in BigQuery.

  • Arrays
  • Objects (Maps)

Unsupported transformations


  • Search term: the value you enter in the Transform Builder
  • Transform: name of the underlying transform
Search termTransform
Unnest elementsunnest
Expand Array to rowsflatten
Extract between delimitersextractbetweendelimiters
Standardize columnstandardize
Nest columnsnest
Extract matches to Arrayextractlist
Replace between delimitersreplacebetweenpatterns
Scale to min maxscaleminmax
Scale to meanscalestandardize
Convert key/value to Objectextractkv

For more information, see Transformation Reference.