Enable the BigQuery Data Transfer Service

To use the BigQuery Data Transfer Service, you must complete the following steps as a project Owner:

• Create a project and enable the BigQuery API.
• Enable the BigQuery Data Transfer Service.

For more information on Identity and Access Management (IAM) roles, see Understanding roles in the IAM documentation.

Note: If you call the BigQuery Data Transfer Service API immediately after you enable BigQuery Data Transfer Service programmatically, you should implement a retriable call with backoff delays between consecutive calls. This is necessary because API enablement is asynchronous and subject to propagation delays caused by eventual consistency.

Create a project and enable the BigQuery API

Before using the BigQuery Data Transfer Service, you must create a project and, in most cases, enable billing on that project. You can use an existing project with the BigQuery Data Transfer Service, or you can create a new one. If you are using an existing project, you may also need to enable the BigQuery API.

To create a project and enable the BigQuery API:

1. In the Google Cloud console, go to the project selector page.

   Go to project selector

2. Select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

3. Enable billing on your project for all transfers. You are billed $0 for free transfers.

   Enabling billing is only required once per project, even if you are transferring data from multiple sources. Billing must also be enabled to query the data in BigQuery, after the data is transferred.

   Learn how to confirm that billing is enabled on your project.

4. BigQuery is automatically enabled in new projects. To activate BigQuery in an existing project, enable the BigQuery API.
   
   Enable the BigQuery API

Enable the BigQuery Data Transfer Service

Before you can create a transfer, you must enable the BigQuery Data Transfer Service. To enable the BigQuery Data Transfer Service, you must be granted the Owner role for your project.

To enable the BigQuery Data Transfer Service:

1. Open the BigQuery Data Transfer API page in the API library.

2. From the dropdown menu, select the appropriate project.

3. Click the ENABLE button.

   Enable the Data Transfer API

Service Agent

The BigQuery Data Transfer Service uses a service agent to access and manage your resources. This includes, but is not limited to, the following resources:

• Retrieving an access token for the service account to use when authorizing the data transfer.
• Publishing notifications to the provided Pub/Sub topic if enabled.
• Starting BigQuery jobs.

The service agent is created automatically on your behalf after you enable the BigQuery Data Transfer Service and use the API for the first time. Upon service agent creation, Google grants the predefined service agent role automatically.

Cross-project Service Account Authorization

If you authorize the data transfer using a service account from a project that is different from the project with the BigQuery Data Transfer Service enabled, you must grant the roles/iam.serviceAccountTokenCreator role to the service agent using the following Google Cloud CLI command:

gcloud iam service-accounts add-iam-policy-binding service_account \
--member serviceAccount:[e-mail] \
--role roles/iam.serviceAccountTokenCreator

Where:

• service_account is the cross-project service account used for authorizing the data transfer.
• project_number is the project number of the project where the BigQuery Data Transfer Service is enabled.

For more information about cross-project resource configuration, see Configuring for a resource in a different project in the Identity and Access Management service account impersonation documentation.

Manual Service Agent Creation

If you want to trigger service agent creation before you interact with the API, for example, if you need to grant extra roles to the service agent, you can use one of the following approaches:

• API: services.GenerateServiceIdentity
• gcloud CLI: gcloud beta services identity create
• Terraform Provider: google_project_service_identity

When you manually trigger service agent creation, Google doesn't grant the predefined service agent role automatically. You must manually grant the service agent the predefined role using the following Google Cloud CLI command:

gcloud projects add-iam-policy-binding project_number \
--member serviceAccount:[e-mail] \
--role roles/bigquerydatatransfer.serviceAgent

Where:

• project_number is the project number of the project where the BigQuery Data Transfer Service is enabled.

Warning: Do not revoke the service agent role from the service agent. If you revoke the role, the BigQuery Data Transfer Service will no longer work.

Grant bigquery.admin access

We recommend granting the bigquery.admin predefined IAM role to users who create BigQuery Data Transfer Service transfers. The bigquery.admin role includes the IAM permissions needed to perform the most common tasks. The bigquery.admin role includes the following BigQuery Data Transfer Service permissions:

• BigQuery Data Transfer Service permissions:
  • bigquery.transfers.update
  • bigquery.transfers.get
• BigQuery permissions:
  • bigquery.datasets.get
  • bigquery.datasets.update
  • bigquery.jobs.create

Note: If the bigquery.admin role is too broad for a specific use case, you can create a custom IAM role with only the necessary permissions.

In some cases, the required permissions might differ between different data sources. Refer to the "Required permissions" section in each data source transfer guide for specific IAM information. For example, see Amazon S3 transfer permissions or Cloud Storage transfer permissions.

To grant the bigquery.admin role:

Console

1. Open the IAM page in the Google Cloud console

   Open the IAM page

2. Click Select a project.

3. Select a project and click Open.

4. Click Add to add new members to the project and set their permissions.

5. In the Add members dialog:

   • For Members, enter the email address of the user or group.
   • In the Select a role drop-down, click BigQuery > BigQuery Admin.

   • Click Add.

     

gcloud

You can use the Google Cloud CLI to grant a user or group the bigquery.admin role.

To add a single binding to your project's IAM policy, type the following command. To add a user, supply the --member flag in the format user:[e-mail]. To add a group, supply the --member flag in the format group:[e-mail].

gcloud projects add-iam-policy-binding project_id \
--member principal:address \
--role roles/bigquery.admin

Where:

• project_id is your project ID.
• principal is either group or user.
• address is the user or group's email address.

For example:

gcloud projects add-iam-policy-binding myproject \
--member group:[e-mail] \
--role roles/bigquery.admin

The command outputs the updated policy:

    bindings:
    - members:
      - group:[e-mail]
        role: roles/bigquery.admin
    

For more information on IAM roles in BigQuery, see Predefined roles and permissions.

What's next

After enabling the BigQuery Data Transfer Service, create a transfer for your data source.

• Amazon S3
• Amazon Redshift
• Azure Blob Storage
• Campaign Manager
• Cloud Storage
• Display & Video 360
• Facebook Ads (Preview)
• Google Ad Manager
• Google Ads
• Google Merchant Center (Preview)
• Google Play
• Oracle (Preview)
• Salesforce (Preview)
• Salesforce Marketing Cloud (Preview)
• Search Ads 360
• ServiceNow (Preview)
• Teradata
• YouTube Channel
• YouTube Content Owner