Work with point-in-time recovery (PITR)

This page describes how to use point-in-time recovery (PITR) to retain and recover data in Firestore in Datastore mode.

To understand PITR concepts, see Point-in-time recovery.

Permissions

To get the permissions that you need to manage PITR settings, ask your administrator to grant you the Cloud Datastore Owner (roles/datastore.owner) IAM role on the project whose PITR settings you want to enable. For more information about granting roles, see Manage access to projects, folders, and organizations.

This predefined role contains the permissions required to manage PITR settings. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to manage PITR settings:

  • To enable PITR when creating a database: datastore.databases.create
  • To update PITR settings on existing database: datastore.databases.update,datastore.databases.list
  • To perform reads from PITR data: datastore.databases.get,datastore.entities.get,datastore.entities.list,datastore.namespaces.get,datastore.namespaces.list,datastore.statistics.get,datastore.statistics.list
  • To export PITR data: datastore.databases.export
  • To import PITR data: datastore.databases.import

You might also be able to get these permissions with custom roles or other predefined roles.

Before you begin

Note the following points before you start using PITR:

  • You can't start reading from seven days in the past immediately after you enable PITR.
  • If you want to enable PITR when you create a database, you must use the gcloud firestore databases create command. Enabling PITR while creating a database using the Google Cloud console is not supported.
  • Datastore mode starts retaining versions from the point forward after enabling PITR.
  • You cannot read PITR data in the PITR window after you disable PITR.
  • If you re-enable PITR immediately after disabling it, the past PITR data is no longer available. Any PITR data created before disabling PITR will be deleted after the PITR expiration date.
  • If you accidentally deleted data in the last hour and PITR is disabled, you can restore your data by enabling PITR within one hour of deletion.
  • Any read performed on expired PITR data fails.

Enable PITR

Before using PITR, enable billing for your Google Cloud project. Only Google Cloud projects with billing enabled can use the PITR feature.

To enable PITR for your database:

Console

  1. In the Google Cloud console, go to the Databases page.

    Go to Databases

  2. Select the required database from the list of databases.

  3. In the navigation menu, click Disaster Recovery.

  4. Click Edit to edit the settings.

  5. Select the Enable point-in-time recovery checkbox, and then click Save.

Enabling PITR would incur storage costs. See Pricing for more information.

To disable PITR, clear the Enable point-in-time recovery checkbox from the Disaster Recovery page in the Google Cloud console.

gcloud

Enable PITR during database creation with the gcloud firestore databases create command as follows:

gcloud firestore databases create\
  --location=LOCATION\
  [--database=DATABASE_ID; default="(default)"]\
  [--type=TYPE; default="firestore-native"]\
  --enable-pitr

Replace the values as follows:

  • LOCATION - location where you want to create your database.
  • DATABASE_ID - set to the database ID or (default).
  • TYPE - set to datastore-mode.

You can disable PITR using the gcloud firestore databases update command as follows:

gcloud firestore databases update\
  [--database=DATABASE_ID; default="(default)"]\
  --no-enable-pitr

Replace the values as follows:

  • DATABASE_ID - set to the database ID or (default).

Get the retention period and earliest version time

Console

  1. In the Google Cloud console, go to the Databases page.

    Go to Databases

  2. Select the required database from the list of databases.

  3. In the navigation menu, click Disaster Recovery.

  4. In the Settings section, note the Retention period and Earliest version time.

    • Retention period: the period in which Datastore mode retains all versions of data for the database. The value is one hour when PITR is disabled and seven days when PITR is enabled.
    • Earliest version time: the earliest timestamp at which earlier versions of the data can be read in the PITR window. This value is continuously updated by Datastore mode and becomes stale the moment it is queried. If you are using this value to recover data, make sure to account for the time from the moment w̦hen the value is queried to the moment when you initiate the recovery.
    • Point-in-time recovery: shows Enabled, if PITR is enabled. If PITR is disabled, you will see Disabled.

gcloud

Run the gcloud firestore databases describe command as follows:

gcloud firestore databases describe --database=DATABASE_ID

Replace DATABASE_ID with the database ID or default.

Here's the output:

    appEngineIntegrationMode: ENABLED
    concurrencyMode: PESSIMISTIC
    createTime: '2021-03-24T17:02:35.234Z'
    deleteProtectionState: DELETE_PROTECTION_DISABLED
    earliestVersionTime: '2023-06-12T16:17:25.222474Z'
    etag: IIDayqOevv8CMNTvyNK4uv8C
    keyPrefix: s
    locationId: nam5
    name: projects/PROJECT_ID/databases/(default)
    pointInTimeRecoveryEnablement: POINT_IN_TIME_RECOVERY_DISABLED
    type: DATASTORE_MODE
    uid: 5230c382-dcd2-468f-8cb3-2a1acfde2b32
    updateTime: '2021-11-17T17:48:22.171180Z'
    versionRetentionPeriod: 3600s

where,

  • earliestVersionTime - timestamp of the earliest PITR data stored.
  • pointInTimeRecoveryEnablement: shows POINT_IN_TIME_RECOVERY_ENABLED, if PITR is enabled. If PITR is disabled, you will either see POINT_IN_TIME_RECOVERY_DISABLED or the pointInTimeRecoveryEnablement field might not be displayed.
  • versionRetentionPeriod - time period for which PITR data is retained in milliseconds. The value can be one hour when PITR is disabled or seven days if PITR is enabled.

Read PITR data

You can read PITR data using the client libraries, REST API methods, or FirestoreIO Apache Beam connector.

Client libraries

Java

You must use the readTime method in the ReadOption class to read PITR data. You cannot use the ReadOnly transaction to perform reads. See ReadOption sample code, for more information.

  Datastore datastore = ...
  Timestamp timestamp = ...

  // lookup
  Key key = ...
  Entity entity = datastore.get(key, ReadOption.readTime(timestamp));

  // runQuery
  Query<Entity> query = ...
  QueryResults<Entity> queryResult = datastore.run(query, ReadOption.readTime(timestamp));

  // runAggregationQuery
  AggregationQuery countAggregationQuery = ...
  Long count = getOnlyElement(datastore.runAggregation(countAggregationQuery, ReadOption.readTime(timestamp))).get("total_count");

For a complete list of readTime examples, see the GitHub repository.

Python

Use PITR read in Datastore mode Python SDK using the readTime method or use the ReadOnly transaction with readTime to perform reads.

  from datetime import datetime, timezone

  read_time = datetime.now(tz=timezone.utc)

  key = …
  # read without PITR read time
  entity = client.get(key)

  # read with PITR read time
  entity = client.get(key, read_time=read_time)

  # PITR read using read_only transaction
  with client.transaction(read_only=True, read_time=read_time):
      entity = client.get(key)

  query = client.query…
  # run query without PITR read time
  iterator = query.fetch()

  # run query with PITR read time
  iterator = query.fetch(read_time=read_time)

  # PITR read query using read_only transaction
  with client.transaction(read_only=True, read_time=read_time):
      iterator = query.fetch()

For a complete list of readTime examples, see the GitHub repository.

REST API

PITR reads are supported in the Datastore mode V1 read methods, which are lookup, runQuery, and runAggregationQuery.

To perform a read using the REST methods, try one of the following options:

  1. In the your read method request, pass the readTime value as a supported PITR timestamp in the readOptions method. A PITR timestamp can be either a microsecond precision timestamp within the past hour or a whole minute timestamp beyond the past hour, but not earlier than the earliestVersionTime.

  2. Use the readTime parameter together with the BeginTransaction method as part of a ReadOnly transaction for multiple PITR reads.

Apache Beam

Use the Datastore mode IO Apache Beam connector to read or write entities in a Datastore mode database at a large scale with Dataflow.

Specify the withReadTime(Instant readTime) method on the DatastoreV1.Read object. All the subsequent reads using the DatastoreV1.Read object read from the same readTime.

Java

The following code shows how to use the withReadTime method for PITR reads.

  com.google.datastore.v1.Query query = ...
    Instant readTime = Instant.ofEpochSecond(1684098540L);

    DatastoreV1.Read read =
            DatastoreIO.v1()
                .read()
                .withProjectId(project)
                .withQuery(query)
                .withNamespace(namespace)
                .withReadTime(readTime);

    PCollection<Entity> entities = pipeline.apply(read);
    ...

For a complete list of withReadTime examples, see the GitHub repository.

Export and import from PITR data

You can export your database to Cloud Storage from PITR data using the gcloud firestore export command. You can export PITR data where the timestamp is a whole minute timestamp within the past seven days, but not earlier than the earliestVersionTime. If data no longer exists at the specified timestamp, the export operation fails.

The PITR export operation supports all filters, including export of all entities and export of specific kinds or namespaces.

  1. Export the database, specifying the snapshot-time parameter to the required recovery timestamp.

    gcloud

    Run the following command to export the database to your bucket.

    gcloud firestore export gs://[BUCKET_NAME_PATH] \
    --snapshot-time=[PITR_TIMESTAMP] \
    --collection-ids=[COLLECTION_IDS] \
    --namespace-ids=[NAMESPACE_IDS]

    Where,

    • BUCKET_NAME_PATH - a valid Cloud Storage bucket with an optional path prefix where export files are stored.
    • PITR_TIMESTAMP - a PITR timestamp at the minute granularity, for example, 2023-05-26T10:20:00.00Z or 2023-10-19T10:30:00.00-07:00.
    • COLLECTION_IDS - a list of collection IDs or collection group IDs, for example-'specific collection group1','specific collection group2'.
    • NAMESPACE_IDS - a list of namespace IDs, for example-'customer','orders'.

    Exporting a specific subset of kinds and/or namespaces with an entity filter is also supported.

    Note the following points before exporting PITR data:

    • Specify the timestamp in RFC 3339 format. For example, 2023-05-26T10:20:00.00Z or 2023-10-19T10:30:00.00-07:00.
    • Make sure that the timestamp you specify is a whole minute timestamp within the past seven days, but not earlier than the earliestVersionTime. If data no longer exists at the specified timestamp, you will get an error. The timestamp must be a whole minute, even if the specified time is within the past hour.
    • You won't be charged for a failed PITR export.

  2. Import to a database.

    Use the steps in Import all entities to import your exported database. If any entity already exists in your database, it will be overwritten. Importing a specific subset of kinds and/or namespaces with an entity filter is also supported.