Create a private connectivity configuration

Overview

In this section, you learn how to create a private connectivity configuration. This type of configuration contains information that Datastream uses to communicate with a data source over a private network (internally within Google Cloud, or with external sources connected over VPN or Interconnect). This communication happens through a Virtual Private Cloud (VPC) peering connection.

A VPC peering connection is a networking connection between two VPCs that lets you route traffic between them using internal, private IPv4 addresses. You need to provide the private IP addresses when setting up the private connectivity configuration because Datastream doesn't support Domain Name System (DNS) resolution in private connections.

Datastream automatically deletes private connectivity configurations that haven't been used for a period of six consecutive months. To retain your private connectivity configurations, make sure they are used in at least one active connection profile.

Before you begin

Before you create a private connectivity configuration, you need to take the following steps so that Datastream can create the VPC peering connection to your project:

• Have a VPC network that can peer to Datastream's private network and that meets the requirements described as restrictions. For more information about creating this network, see Using VPC Network Peering.
• Identify an available IP range (with a CIDR block of /29) on the VPC network. This can't be an IP range that already exists as a subnet, a private services access pre-allocated IP range, or any route (other than the default 0.0.0.0 route) that includes the IP range. Datastream uses this IP range to create a subnet so that it can communicate with the source database. The following table describes valid IP ranges.

Range	Description
10.0.0.0/8 172.16.0.0/12 192.168.0.0/16	Private IP addresses RFC 1918
100.64.0.0/10	Shared address space RFC 6598
192.0.0.0/24	IETF protocol assignments RFC 6890
192.0.2.0/24 (TEST-NET-1) 198.51.100.0/24 (TEST-NET-2) 203.0.113.0/24 (TEST-NET-3)	Documentation RFC 5737
192.88.99.0/24	IPv6 to IPv4 relay (deprecated) RFC 7526
198.18.0.0/15	Benchmark testing RFC 2544

• Verify that Google Cloud and the on-premises firewall allow traffic from the selected IP range. If they don't, then create an ingress firewall rule that allows traffic on the source database port, and make sure that the IPv4 address range in the firewall rule is the same as the IP address range allocated when creating the private connectivity resource:

  gcloud compute firewall-rules create FIREWALL-RULE-NAME \
    --direction=INGRESS \
    --priority=PRIORITY \
    --network=PRIVATE_CONNECTIVITY_VPC \
    --project=VPC_PROJECT \
    --action=ALLOW \
    --rules=FIREWALL_RULES \
    --source-ranges=IP-RANGE
    

  Replace the following:

  • FIREWALL-RULE-NAME: The name of the firewall rule to create.
  • PRIORITY: The priority for the rule, expressed as an integer between 0 and 65535, inclusive. The value needs to be lower than the value set for the block traffic rule, if it exists. Lower priority values imply higher precedence.
  • PRIVATE_CONNECTIVITY_VPC: The VPC network that can peer to the Datastream private network and that meets the requirements described as restrictions. This is the VPC you specify when you create your private connectivity configuration.
  • VPC_PROJECT: The project of the VPC network.
  • FIREWALL_RULES: The list of protocols and ports to which the firewall rule applies, for example tcp:80. The rule needs to allow TCP traffic to the IP address and the port of the source database, or of the proxy. Because private connectivity can support multiple databases, the rule needs to consider the actual usage of your configuration.

  • IP-RANGE: The range of IP addresses that Datastream uses to communicate with the source database. This is the same range you indicate in the Allocate an IP range field when you create your private connectivity configuration.

    You might also need to create an identical egress firewall rule to allow traffic back to Datastream.

• Are assigned to a role that contains the compute.networks.list permission. This permission gives you the required IAM permissions to list VPC networks in your project. You can find which roles contain this permission by viewing IAM permissions reference.

You can use private connectivity to connect Datastream to any source. However, only VPC networks that are peered directly can communicate with each other.

Transitive peering isn't supported. If the network that Datastream is peered with isn't the network where your source is hosted, you're using a fully managed database (for example, Cloud SQL), or if Datastream doesn't run on the region where your source exists, then a reverse proxy is required.

For more information, see Private connectivity.

Shared VPC prerequisites

If you're using Shared VPC, then you must complete the following actions in addition to the steps described in the Before you begin section:

1. On the service project:

   1. Enable the Datastream API.

   2. Obtain the email address used for the Datastream service account. Datastream service accounts are created when you perform one of the following:

      • You create a Datastream resource, such as a connection profile or a stream.
      • You create a private connectivity configuration, select your shared VPC and click Create Datastream Service Account. The service account is created in the host project.

      To obtain the email address used for the Datastream service account, find the Project number in the Google Cloud console home page. The email address of the service account is service-[project_number]@gcp-sa-datastream.iam.gserviceaccount.com.

2. On the host project:

   1. Grant the compute.networkAdmin Identity and Access Management (IAM) role permission to the Datastream service account. This role is only required when you create the VPC peering. After the peering is established, you no longer need the role.

      If your organization doesn't allow granting the permission, create a custom role with the following minimum permissions to create and delete private connection resources:

      compute.globalAddresses.*

      • compute.globalAddresses.create
      • compute.globalAddresses.createInternal
      • compute.globalAddresses.delete
      • compute.globalAddresses.deleteInternal
      • compute.globalAddresses.get

      compute.globalOperations.*

      • compute.globalOperations.get

      compute.networks.*

      • compute.networks.addPeering
      • compute.networks.get
      • compute.networks.listPeeringRoutes
      • compute.networks.removePeering
      • compute.networks.use

      compute.routes.*

      • compute.routes.get
      • compute.routes.list

      compute.subnetworks.*

      • compute.subnetworks.get
      • compute.subnetworks.list

   For more information about custom roles, see Create and manage custom roles.

Create the configuration

1. Review the required prerequisites to reflect how the environment must be prepared for a private connectivity configuration. For more information about these prerequisites, see Before you begin.

2. Go to the Private connectivity configurations page in the Google Cloud Console.

   Go to the Private connectivity configurations page

3. Click CREATE CONFIGURATION.

4. Use the following table to populate the fields of the Configure private connectivity section of the Create private connectivity configuration page:

   Field	Description
   Configuration name	Enter the display name of the private connectivity configuration.
   Configuration ID	Datastream populates this field automatically based on the configuration name that you enter. You can keep the ID that's auto-generated or change it.
   Region	Select the region where the private connectivity configuration is stored. Private connectivity configurations are saved in a region. Region selection can impact availability if the region experiences downtime. To use a private connection, any streams and connection profiles that use this configuration must be in the same region.

5. Use the following table to populate the fields of the Set up connection section of the Create private connectivity configuration page:

   Field	Description
   Authorized VPC network	Select the VPC network that you created in Before you begin.
   Allocate an IP range	Enter an available IP range on the VPC network. You determined this IP range in Before you begin.

6. Click CREATE.

   It takes a few minutes for your private connectivity configuration to be created. Background resources need to be created so that the configuration can be created.

After creating a private connectivity configuration, you can view high-level and detailed information about it.

What's next

• Learn more about private connectivity.
• Learn how to view your private connectivity configuration.
• Find out how to delete a private connectivity configuration.