...
- All datasources and all outputs specified in a job must be located within BigQuery.
must be selected as running environment.D s dataflow - Custom SQL datasets are not supported.
All recipe steps, including all
functions in the recipe, must be translatable to SQL.D s lang Info NOTE: When attempting to execute a job in BigQuery,
executes each recipe in BigQuery, until it reaches a step that cannot be executed there. At that point, data is transferred toD s webapp
, where the remainder of the job is executed.D s dataflow - Some transformations and functions are not currently supported for execution in BigQuery. See below.
- Upserts, merges, and deletes are not supported for full execution in BigQuery.
- Sampling jobs are not supported for execution in BigQuery.
- If your recipe includes data quality rules, the job cannot be fully executed in BigQuery.
| Info |
|---|
NOTE: In the new BigQuery console, you may not be able to see the SQL that was used to execute your job in BigQuery. This is a known issue with the new BigQuery console. |
Unsupported
| D s lang |
|---|
...
- Regex patterns used must be valid RE2. Operations on non-RE2 regex patterns are not pushed down.
- For more information on limitations on specific push-downs, see Flow Optimization Settings Dialog.
Unsupported data types
The following data types are not supported for execution in BigQuery.
- Arrays
- Objects (Maps)
Unsupported transformations
...
- Search term: the value you enter in the Transform Builder
- Transform: name of the underlying transform
| Search term | Transform |
|---|---|
| Unnest elements | unnest |
| Expand Array to rows | flatten |
| Extract between delimiters | extractbetweendelimiters |
| Unpivot | unpivot |
| Standardize column | standardize |
| Nest columns | nest |
| Extract matches to Array | extractlist |
| Replace between delimiters | replacebetweenpatterns |
| Scale to min max | scaleminmax |
| Scale to mean | scalestandardize |
| Convert key/value to Object | extractkv |
For more information, see Transformation Reference.
...
For more information, see Other Functions.
Verify Execution
To verify execution in BigQuery, please do the following:
Steps:
- In the left nav bar, click the Jobs link.
- In the Jobs page, select the job that you executed.
- In the Overview tab, the value for Environment under the Execution summary should be:
BigQuery.
For more information, see Job Details Page.
