Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r094

...

You can write datasets to BigQuery as part of your job definition.  

Info

NOTE: In some environments, users may not be permitted the bigquery.datasets.create permission, and

D s dataflow
jobs on BigQuery sources fail. As a workaround, an administrator can define a BigQuery dataset for use for these jobs. This dataset is used by
D s product
productdp
for storing temporary tables of intermediate query results when running
D s dataflow
jobs on BigQuery sources, which eliminates the need for the bigquery.datasets.create permission in the service account. For more information on defining this BigQuery query dataset, see Dataprep Project Settings Page.

Info

NOTE: Object and Array data types are written back to BigQuery as string values.

...

Info

NOTE: BigQuery does not support destinations with a dot (.) in the name. For more information, see https://cloud.google.com/bigquery/docs/tables#table_naming.


Tip

Tip: Optionally, you can choose to merge your results to an existing BigQuery table. Based on a specified set of matching columns, you can define the rows to update from your source.

Info

NOTE: By default, data is published to BigQuery using strict data type matching rules. When there are mismatches between

D s item
itemdata types
and BigQuery data types, the publication fails. This setting is defined on a per-output basis. For more information, see BigQuery Table Settings.


Tip

Tip: During import from

D s storage
, you can choose to show hidden files. These files include profiler files generated during a job run. These files can be re-imported into the product and then published to BigQuery for additional analysis. Please note that BigQuery does not support names with a dot in them. For more information, see Import Data Page.

For more information on creating a publishing action to BigQuery, see BigQuery Table Settings.

...

Create New Connection: n/a

D s also
inCQLtrue
label((label = "bigquery") OR (label = "connection"))