BigQuery API Connector Overview

The Workflows connector defines the built-in functions that can be used to access other Google Cloud products within a workflow.

This page provides an overview of the individual connector. There is no need to import or load connector libraries in a workflow—connectors work out of the box when used in a call step.

BigQuery API

A data platform for customers to create, manage, share and query data. To learn more, see the BigQuery API documentation.

You can use a page token in a workflow to paginate through the results of a query to a BigQuery dataset.

BigQuery connector sample

YAML

# This workflow demonstrates how to use the BigQuery connector: # Create a dataset and then insert a table with data from a public dataset # Delete both the table and dataset # Expected output: "SUCCESS" - init:  assign:  - project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}  - dataset_id: "example_dataset"  - table_id: "example_table"  - query: "SELECT * FROM `bigquery-public-data.usa_names.usa_1910_2013` LIMIT 5000;"  - create_disposition: "CREATE_IF_NEEDED" # creates table if it doesn't exist  - write_disposition: "WRITE_TRUNCATE" # truncates table if it already exists - create_dataset:  call: googleapis.bigquery.v2.datasets.insert  args:  projectId: ${project_id}  body:  datasetReference:  datasetId: ${dataset_id}  projectId: ${project_id}  access[].role: "roles/bigquery.dataViewer"  access[].specialGroup: "projectReaders" - insert_table_into_dataset:  call: googleapis.bigquery.v2.jobs.insert  args:  projectId: ${project_id}  body:  configuration:  query:  query: ${query}  destinationTable:  projectId: ${project_id}  datasetId: ${dataset_id}  tableId: ${table_id}  create_disposition: ${create_disposition}  write_disposition: ${write_disposition}  allowLargeResults: true  useLegacySql: false - delete_table_from_dataset:  call: googleapis.bigquery.v2.tables.delete  args:  projectId: ${project_id}  datasetId: ${dataset_id}  tableId: ${table_id} - delete_dataset:  call: googleapis.bigquery.v2.datasets.delete  args:  projectId: ${project_id}  datasetId: ${dataset_id} - the_end:  return: "SUCCESS"

JSON

[  {  "init": {  "assign": [  {  "project_id": "${sys.get_env(\"GOOGLE_CLOUD_PROJECT_ID\")}"  },  {  "dataset_id": "example_dataset"  },  {  "table_id": "example_table"  },  {  "query": "SELECT * FROM `bigquery-public-data.usa_names.usa_1910_2013` LIMIT 5000;"  },  {  "create_disposition": "CREATE_IF_NEEDED"  },  {  "write_disposition": "WRITE_TRUNCATE"  }  ]  }  },  {  "create_dataset": {  "call": "googleapis.bigquery.v2.datasets.insert",  "args": {  "projectId": "${project_id}",  "body": {  "datasetReference": {  "datasetId": "${dataset_id}",  "projectId": "${project_id}"  },  "access[].role": "roles/bigquery.dataViewer",  "access[].specialGroup": "projectReaders"  }  }  }  },  {  "insert_table_into_dataset": {  "call": "googleapis.bigquery.v2.jobs.insert",  "args": {  "projectId": "${project_id}",  "body": {  "configuration": {  "query": {  "query": "${query}",  "destinationTable": {  "projectId": "${project_id}",  "datasetId": "${dataset_id}",  "tableId": "${table_id}"  },  "create_disposition": "${create_disposition}",  "write_disposition": "${write_disposition}",  "allowLargeResults": true,  "useLegacySql": false  }  }  }  }  }  },  {  "delete_table_from_dataset": {  "call": "googleapis.bigquery.v2.tables.delete",  "args": {  "projectId": "${project_id}",  "datasetId": "${dataset_id}",  "tableId": "${table_id}"  }  }  },  {  "delete_dataset": {  "call": "googleapis.bigquery.v2.datasets.delete",  "args": {  "projectId": "${project_id}",  "datasetId": "${dataset_id}"  }  }  },  {  "the_end": {  "return": "SUCCESS"  }  } ]

Module: googleapis.bigquery.v2.datasets

Functions
delete Deletes the dataset specified by the datasetId value. Before you can delete a dataset, you must delete all its tables, either manually or by specifying deleteContents. Immediately after deletion, you can create another dataset with the same name.
get Returns the dataset specified by datasetID.
insert Creates a new empty dataset.
list Lists all datasets in the specified project to which you have been granted the READER dataset role.
patch Updates information in an existing dataset. The update method replaces the entire dataset resource, whereas the patch method only replaces fields that are provided in the submitted dataset resource. This method supports patch semantics.
update Updates information in an existing dataset. The update method replaces the entire dataset resource, whereas the patch method only replaces fields that are provided in the submitted dataset resource.

Module: googleapis.bigquery.v2.jobs

Functions
cancel Requests that a job be cancelled. This call will return immediately, and the client will need to poll for the job status to see if the cancel completed successfully. Cancelled jobs may still incur costs.
delete Requests the deletion of the metadata of a job. This call returns when the job's metadata is deleted.
get Returns information about a specific job. Job information is available for a six month period after creation. Requires that you're the person who ran the job, or have the Is Owner project role.
getQueryResults Retrieves the results of a query job.
insert Starts a new asynchronous job. Requires the Can View project role.
list Lists all jobs that you started in the specified project. Job information is available for a six month period after creation. The job list is sorted in reverse chronological order, by job creation time. Requires the Can View project role, or the Is Owner project role if you set the allUsers property.
query Runs a BigQuery SQL query synchronously and returns query results if the query completes within a specified timeout.

Module: googleapis.bigquery.v2.models

Functions
delete Deletes the model specified by modelId from the dataset.
get Gets the specified model resource by model ID.
list Lists all models in the specified dataset. Requires the READER dataset role. After retrieving the list of models, you can get information about a particular model by calling the models.get method.
patch Patch specific fields in the specified model.

Module: googleapis.bigquery.v2.projects

Functions
getServiceAccount Returns the email address of the service account for your project used for interactions with Google Cloud KMS.
list Lists all projects to which you have been granted any project role.

Module: googleapis.bigquery.v2.routines

Functions
delete Deletes the routine specified by routineId from the dataset.
get Gets the specified routine resource by routine ID.
insert Creates a new routine in the dataset.
list Lists all routines in the specified dataset. Requires the READER dataset role.
update Updates information in an existing routine. The update method replaces the entire Routine resource.

Module: googleapis.bigquery.v2.rowAccessPolicies

Functions
getIamPolicy Gets the access control policy for a resource. Returns an empty policy if the resource exists and does not have a policy set.
list Lists all row access policies on the specified table.
testIamPermissions Returns permissions that a caller has on the specified resource. If the resource does not exist, this will return an empty set of permissions, not a NOT_FOUND error. Note: This operation is designed to be used for building permission-aware UIs and command-line tools, not for authorization checking. This operation may "fail open" without warning.

Module: googleapis.bigquery.v2.tabledata

Functions
insertAll Streams data into BigQuery one record at a time without needing to run a load job. Requires the WRITER dataset role.
list Retrieves table data from a specified set of rows. Requires the READER dataset role.

Module: googleapis.bigquery.v2.tables

Functions
delete Deletes the table specified by tableId from the dataset. If the table contains data, all the data will be deleted.
get Gets the specified table resource by table ID. This method does not return the data in the table, it only returns the table resource, which describes the structure of this table.
getIamPolicy Gets the access control policy for a resource. Returns an empty policy if the resource exists and does not have a policy set.
insert Creates a new, empty table in the dataset.
list Lists all tables in the specified dataset. Requires the READER dataset role.
patch Updates information in an existing table. The update method replaces the entire table resource, whereas the patch method only replaces fields that are provided in the submitted table resource. This method supports patch semantics.
setIamPolicy Sets the access control policy on the specified resource. Replaces any existing policy. Can return NOT_FOUND, INVALID_ARGUMENT, and PERMISSION_DENIED errors.
testIamPermissions Returns permissions that a caller has on the specified resource. If the resource does not exist, this will return an empty set of permissions, not a NOT_FOUND error. Note: This operation is designed to be used for building permission-aware UIs and command-line tools, not for authorization checking. This operation may "fail open" without warning.
update Updates information in an existing table. The update method replaces the entire table resource, whereas the patch method only replaces fields that are provided in the submitted table resource.