Manage notebooks

Note: BigQuery Studio notebooks will begin charging in General Availability after April 11, 2024. You will begin seeing usage-based charges for BigQuery Studio notebooks in your invoice.

This document describes how to manage Colab Enterprise notebooks in BigQuery, including how to view, compare, restore, schedule, and delete notebooks.

Notebooks are BigQuery Studio code assets powered by Dataform.

To use notebooks, you must first Enable BigQuery Studio for asset management.

Important: In March 2024, APIs supporting BigQuery Studio capabilities were automatically enabled for all existing BigQuery projects. New BigQuery projects have these APIs enabled by default. For more information, see Automatic enablement of BigQuery Studio APIs.

Required permissions

To share notebooks, you need the following Identity and Access Management (IAM) roles:

• BigQuery Job User (roles/bigquery.jobUser)
• BigQuery Read Session User (roles/bigquery.readSessionUser)
• Code Owner (roles/dataform.codeOwner) at the resource level.

To save and delete notebooks, you need the following IAM roles:

• BigQuery Job User (roles/bigquery.jobUser)
• BigQuery Read Session User (roles/bigquery.readSessionUser)
• Code Owner (roles/dataform.codeOwner) or Code Editor (roles/dataform.codeEditor)

To work with notebook revisions, you need the following IAM roles:

• BigQuery Job User (roles/bigquery.jobUser)
• BigQuery Read Session User (roles/bigquery.readSessionUser)

• Any one of the following roles:

  • Code Owner (roles/dataform.codeOwner)
  • Code Editor (roles/dataform.codeEditor)
  • Code Viewer (roles/dataform.codeViewer)

To create notebook schedules (Preview), you need the following roles:

• Dataform Admin (roles/dataform.admin)
• BigQuery Read Session User (roles/bigquery.readSessionUser) or BigQuery Studio User (roles/bigquery.studioUser)

To edit notebook schedules (Preview), you need the Dataform Editor (roles/dataform.editor) role.

To view notebook schedules (Preview), you need the Dataform Viewer (roles/dataform.viewer) role.

For more information about BigQuery IAM, see Access control with IAM.

Grant access to notebooks

To grant other users access to a notebook, add those users to an appropriate IAM role.

Important: Users who have access to the notebook can see all output generated by code in the notebook, even if this output contains data from tables they don't have access to. To prevent saved output from being shared, disable notebook output saving.

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Find the notebook that you want to grant access to.

3. Click View actions next to the notebook, and then click Share > Manage Permissions.

4. In the Manage permissions pane, click Add user/group.

5. In the New principals field, enter a principal.

6. In the Role list, select one of the following roles:

   • Code Owner: Can perform any action on the notebook, including deleting or sharing it.
   • Code Editor: Can edit the notebook.
   • Code Viewer: Can view the notebook.
   Note: The principal must also have the Notebook Runtime User (roles/aiplatform.notebookRuntimeUser) and BigQuery User (roles/bigquery.user) roles to run the notebook.

7. Optional: To view a complete list of roles and advanced sharing settings, click Advanced sharing.

8. Click Save.

9. To return to the notebook information page, click Close.

Share notebooks

To share a notebook with other users, you can generate and share a link to the notebook. For other users to see the notebook you share, you must first grant access to the notebook.

To run a notebook, users must have access to the data that the notebook accesses. For more information, see Grant access to a dataset.

Important: Users who have access to the notebook can see all output generated by code in the notebook, even if this output contains data from tables they don't have access to. To prevent saved output from being shared, disable notebook output saving.

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, expand the Shared notebooks folder. Find the notebook that you want to share.

3. Click View actions next to the notebook, and then click Share > Copy link.

4. Share the link with other users.

View all notebooks

To view a list of all notebooks in your project, do the following:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, click View actions next to Notebooks, and then do one of the following:

• To open the list in the current tab, click Show all.
• To open the list in a new tab, click Show all in > New tab.
• To open the list in a split tab, click Show all in > Split tab.

View notebook metadata

To view notebook metadata, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the notebook that you want to view metadata for.

3. Look at the Summary pane to see information about the notebook such as the region it uses for data and the date it was last modified.

Work with notebook versions

Use the following sections to learn how to view, compare, and restore versions of a notebook.

View notebook versions

To view notebook versions, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the notebook you want to view activity for.

3. Click the Activity tab to see a list of the notebook versions in descending order by date.

Compare notebook versions

To compare notebook versions, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the notebook that you want to compare version information for.

3. In the Activity tab, click View actions next to a notebook version and then click Compare.

4. In the Revision history pane, use the Version drop-downs to choose versions to compare.

5. Optional: To compare the notebook source code, select the Raw source checkbox.

6. Optional: To compare the versions inline instead of in separate panes, select the Inline diff checkbox.

7. Optional: To hide command output in the notebook, clear the Show output checkbox.

Restore a notebook version

Use one of the following options to restore a notebook version. Restoring from the Revision history pane lets you compare the current and previous versions of the notebook before choosing to restore the previous version. Restoring a notebook creates a new version of a notebook instead of overwriting the current version. No version history is lost.

Activity pane

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, expand the Shared notebooks folder. Click the name of the notebook that you want to restore a previous version of.

3. Select the Activity pane.

4. Click View actions next to the version of the notebook that you want to restore and then click Restore.

5. Click Confirm.

Revision history

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, expand the Shared notebooks folder. Click the name of the notebook that you want to restore a previous version of.

3. Select the Activity pane.

4. Click View actions next to a notebook version and then click Compare.

5. In the Revision history pane, click View actions next to the Version drop-down and then click Restore revision.

6. Click Confirm.

Download a notebook

To download a notebook, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the notebook that you want to download.

3. Click Download.

Schedule notebooks

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

To provide feedback for notebook scheduling, email [e-mail].

You can schedule a notebook to automatically run at a specified time and frequency—for example, to train an ML model, call external APIs, or run BigQuery DataFrames code.

Changes that you make to a notebook are automatically saved, but are available only to you and to users who have access to the notebook. To update the schedule with a new version of the notebook, you need to deploy the notebook. By deploying a notebook, you update its schedule with your current version of the notebook. Schedules run the latest deployed version of the notebook.

Each notebook schedule is run by a custom Dataform service account, which you select during schedule creation. For more information about the types of service accounts in Dataform, see About service accounts in Dataform.

Dataform writes the output of scheduled notebook runs to the Cloud Storage bucket selected during schedule creation.

Notebook schedules use a standard E2 runtime. Colab Enterprise charges for runtimes apply. You are charged for runtime processing based on the E2 machine type. For information about pricing of standard E2 runtimes, see Colab Enterprise pricing.

Enable notebook scheduling

To schedule notebooks, you must grant the following roles to the custom Dataform service account which you plan to use for notebook schedules:

Notebook Executor User (roles/aiplatform.notebookExecutorUser)

Follow Grant a single role on a project to grant the Notebook Executor User role to your custom Dataform service account on the selected project.

Storage Admin (roles/storage.admin)

Follow Add a principal to a bucket-level policy to add your custom Dataform service account as a principal to the Cloud Storage bucket which you plan to use for storing output of scheduled notebook runs, and grant the Storage Admin role to this principal.

Service Account User (roles/iam.serviceAccountUser)

Follow Grant a single role on a service account to add your custom Dataform service account as a principal to itself. In other words, add the custom Dataform service account as a principal to the same custom Dataform service account. Then, grant the Service Account User role to this principal.

Additionally, you must grant the following role to the default Dataform service account:

Service Account Token Creator(roles/iam.serviceAccountTokenCreator)

Follow Grant token creation access to a custom Dataform service account to add the default Dataform service account as a principal to your custom Dataform service account, and grant the Service Account Token Creator role to this principal.

To learn more about service accounts in Dataform, see About service accounts in Dataform.

Create a schedule

To schedule a notebook, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the notebook that you want to schedule.

3. Click Schedule.

4. In the Schedule Notebook pane, in the Schedule name field, enter a name for the schedule.

5. In the Runtime template field, select a runtime.

6. In the Service account field, select a custom Dataform service account.

7. In the Cloud Storage bucket field, click Browse and select or create a Cloud Storage bucket.

   The selected custom Dataform service account must be granted the Storage Admin IAM role on the selected bucket. For more information, see Enable notebook scheduling.

8. In the Schedule frequency section, do the following:

   1. In the Repeats drop-down, select the frequency of scheduled notebook runs.
   2. In the At time field, enter the time for scheduled notebook runs.
   3. In the Timezone drop-down, select the timezone for the schedule.

9. Click Create schedule.

When you create the schedule, the current version of the notebook is automatically deployed. To update the schedule with a new version of the notebook, deploy the notebook.

The latest deployed version of the notebook runs at the selected time and frequency.

Deploy a notebook

Deploying a notebook updates its schedule with the current version of the notebook. Schedules run the latest deployed version of the notebook.

To deploy a notebook, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the selected notebook.

3. Click Deploy.

The corresponding schedule is updated with the current version of the notebook. The latest deployed version of the notebook runs at the scheduled time.

View schedule details

To view schedule details for a selected notebook, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the selected notebook.

3. Click View schedule.

View past executions

To view past executions of a selected notebook schedule, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the selected notebook.

3. Click View schedule, and then click View past executions.

Disable a schedule

To pause scheduled runs of a selected notebook without deleting the schedule, you can disable the schedule.

To disable a schedule for a selected notebook, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the selected notebook.

3. Click View schedule.

4. In the schedule details table, in the Schedule state row, click the Schedule is enabled toggle.

Enable a schedule

To resume scheduled runs of a disabled schedule, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the selected notebook.

3. Click View schedule.

4. In the schedule details table, in the Schedule state row, click the Schedule is disabled toggle.

Edit a schedule

To edit a schedule, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the selected notebook.

3. Click View schedule, and then click Edit.

4. In the Schedule notebook dialog, edit the schedule, and then click Update schedule.

View all schedules

To view all notebook schedules in your project, follow these steps:

1. In the Google Cloud console, go to the Orchestration page.

   Go to the Orchestration page

2. Optional: To view details of a selected schedule and its past runs, click the name of the schedule.

Delete a schedule

To permanently delete a schedule for a selected notebook, follow these steps:

1. In the Google Cloud console, go to the Orchestration page.

   Go to the Orchestration page

2. In the list of all schedules, select the row that contains the schedule that you want to delete.

3. Optional: To delete multiple schedules at once, select multiple rows.

4. Click Delete.

5. In the dialog that appears, click Delete.

Delete notebooks

To delete a notebook, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, expand the Shared notebooks folder. Find the notebook that you want to delete.

3. Click View actions next to the notebook and then click Delete.

4. To confirm deletion, type delete in the dialog.

5. Click Delete.

Troubleshooting

For more information, see Troubleshoot Colab Enterprise.

What's next

• Learn more about Colab Enterprise notebooks in BigQuery.
• Learn how to create notebooks.