- All datasources and all outputs specified in a job must be located within BigQuery.
must be selected as running environment.
D s dataflow
- Custom SQL datasets are not supported.
All recipe steps, including all
functions in the recipe, must be translatable to SQL.
D s lang Info
NOTE: When attempting to execute a job in BigQuery,
executes each recipe in BigQuery, until it reaches a step that cannot be executed there. At that point, data is transferred to
D s webapp
, where the remainder of the job is executed.
D s dataflow
- Some transformations and functions are not currently supported for execution in BigQuery. See below.
- Upserts, merges, and deletes are not supported for full execution in BigQuery.
- Sampling jobs are not supported for execution in BigQuery.
- If your recipe includes data quality rules, the job cannot be fully executed in BigQuery.
NOTE: In the new BigQuery console, you may not be able to see the SQL that was used to execute your job in BigQuery. This is a known issue with the new BigQuery console.
|D s lang|
- Regex patterns used must be valid RE2. Operations on non-RE2 regex patterns are not pushed down.
- For more information on limitations on specific push-downs, see Flow Optimization Settings Dialog.
Unsupported data types
The following data types are not supported for execution in BigQuery.
- Objects (Maps)
- Search term: the value you enter in the Transform Builder
- Transform: name of the underlying transform
|Expand Array to rows||flatten|
|Extract between delimiters||extractbetweendelimiters|
|Extract matches to Array||extractlist|
|Replace between delimiters||replacebetweenpatterns|
|Scale to min max||scaleminmax|
|Scale to mean||scalestandardize|
|Convert key/value to Object||extractkv|
For more information, see Transformation Reference.
For more information, see Other Functions.
To verify execution in BigQuery, please do the following:
- In the left nav bar, click the Jobs link.
- In the Jobs page, select the job that you executed.
- In the Overview tab, the value for Environment under the Execution summary should be:
For more information, see Job Details Page.