Using VPC Service Controls with AI Platform Training

VPC Service Controls can help you mitigate the risk of data exfiltration from your AI Platform Training jobs. When you run a training job from a project inside a service perimeter, VPC Service Controls ensures that your data does not leave the perimeter. This includes training data that your job accesses and artifacts that your job creates.

Creating a service perimeter

Follow the VPC Service Controls guide to creating a service perimeter. When specifying which services you want to restrict, make sure to add all of the following services:

  • AI Platform Training and Prediction API (ml.googleapis.com)
  • Pub/Sub API (pubsub.googleapis.com)
  • Cloud Storage API (storage.googleapis.com)
  • Google Kubernetes Engine API (container.googleapis.com)
  • Container Registry API (containerregistry.googleapis.com)
  • Cloud Logging API (logging.googleapis.com)

Your service perimeter must restrict all these services in order for AI Platform Training and AI Platform Prediction to work properly with VPC Service Controls.

Limitations

After you have created a service perimeter and added your Google Cloud project to it, you can run training jobs without any additional configuration. However, the following limitations apply:

  • If you submit a training job in the first few minutes after creating a service perimeter, then the job might fail. Wait approximately 15 minutes for the VPC Service Controls restrictions to propagate to all the relevant Google Cloud services, and then try again.
  • You cannot perform training with TPUs.
  • When ml.googleapis.com is protected, your training job does not have access to resources outside the perimeter. Your training code can access data in Cloud Storage and other Google Cloud services supported by VPC Service Controls in projects within the perimeter, but if your code sends requests to services outside the perimeter, those requests will fail.
  • Without additional configuration, you cannot use the Google Cloud console to manage the training jobs of a project inside a service perimeter or to view logs. Learn about accessing resources protected by a service perimeter in the Google Cloud console.

AI Platform Prediction and AI Platform Vizier

When you create a service perimeter that protects the AI Platform Training and Prediction API, VPC Service Controls protects AI Platform Training and AI Platform Prediction's online prediction functionality. Read about how to use VPC Service Controls with AI Platform Prediction.

However, batch prediction is not supported by VPC Service Controls. When the AI Platform Training and Prediction API is protected by a service perimeter, AI Platform disables the ability to create a batch prediction job in order to help protect your data from exfiltration.

AI Platform Vizier, which also uses the AI Platform Training and Prediction API, does not currently fully support VPC Service Controls. However, AI Platform Vizier remains enabled when you configure a service perimeter to protect the AI Platform Training and Prediction API.

What's next