Configure a cluster to be managed by the GKE On-Prem API

The GKE On-Prem API is a Google Cloud-hosted API that lets you manage the lifecycle of your on-premises clusters by using standard tools: the Google Cloud console, the Google Cloud CLI, or Terraform. When you create a cluster using one of these tools, the API stores metadata about your cluster's state in the Google Cloud region that you specified when creating the cluster. This metadata lets you manage the lifecycle of the cluster using the standard tools. If you want to use these tools to view cluster details or manage the lifecycle of clusters that were created using gkectl, you must enroll the clusters in the GKE On-Prem API.

Terminology

Enrolling a cluster lets you manage the cluster lifecycle by using the console, the gcloud CLI, or Terraform.

Enrolling a cluster is a separate process to registering a cluster to a fleet. A fleet is a a logical grouping of Kubernetes clusters that you can manage together. All Google Distributed Cloud clusters are registered to a fleet at cluster creation time. When you create a cluster using gkectl, the cluster is registered to the Google Cloud project that you specify in the gkeConnect.projectID field in the cluster configuration file. This project is referred to as the fleet host project. To learn more about fleets, including uses cases, best practices, and examples, see the Fleet management documentation.

View registered clusters

All your fleet clusters are displayed on the GKE Clusters pages in the console. This both gives you an overview of your entire fleet and, for Google Distributed Cloud, lets you see which clusters are managed by the GKE On-Prem API.

To view your fleet clusters:

  1. In the console, go to the GKE clusters page.
    Go to GKE clusters
  2. Select the Google Cloud project.
    • If VMware is displayed in the Type column, the cluster is managed by the GKE On-Prem API.
    • If External is displayed in the Type column, the cluster isn't managed by the GKE On-Prem API.

Requirements

  • User clusters must be version 1.11 or higher.

  • Admin clusters must be version 1.13 or higher.

  • If you aren't a project owner, minimally, you must be granted the Identity and Access Management role roles/gkeonprem.admin on the project. For details on the permissions included in this role, see GKE on-prem roles in the IAM documentation.

Before you begin

  1. Set up the gcloud CLI, if needed.

    1. If you need to install the gcloud CLI, see the gcloud CLI documentation.

    2. Update the gcloud CLI components, if needed:

      gcloud components update

  2. Make sure that the GKE On-Prem API is enabled in the project. To enable the API, run the following command:

    gcloud services enable \
    --project PROJECT_ID \
    gkeonprem.googleapis.com

    Replace PROJECT_ID with the project ID of your fleet host project. This is the project ID that was configured in the gkeconnect section of your admin cluster configuration file or user cluster configuration file.

    If you get a PERMISSION_DENIED error, double check the project ID that you entered. If the project ID is correct, run gcloud auth login to sign in to the Google Cloud CLI with the account that has access to the project.

  3. If this is the first time that you have enabled the GKE On-Prem API in your project, you might need to initialize the API. You can do this by calling a gcloud CLI command that displays available versions that you can use to create a cluster:

    gcloud container vmware clusters query-version-config \
    --project=PROJECT_ID \
    --location=REGION

    Replace REGION with us-west1 or another supported region.

  4. If your organization has set up an allowlist that lets traffic from Google APIs and other addresses pass through your proxy server, add the following to the allowlist:

    • gkeonprem.googleapis.com
    • gkeonprem.mtls.googleapis.com

    These are the service names for the GKE On-Prem API.

Enroll a cluster

To enroll a cluster, you can use the gcloud CLI. If your cluster is at 1.15.3 or higher, you can use the gkectl command-line tool.

Enroll a user cluster

gcloud CLI

Be sure to scroll over if needed to fill in the ADMIN_CLUSTER_NAME placeholder for the --admin-cluster-membership flag.

gcloud container vmware clusters enroll USER_CLUSTER_NAME  \
  --project=PROJECT_ID \
  --admin-cluster-membership=projects/PROJECT_ID/locations/global/memberships/ADMIN_CLUSTER_NAME \
  --location=REGION

Replace the following:

  • USER_CLUSTER_NAME: The name of the user cluster that you want to enroll.

  • PROJECT_ID The project ID of your fleet host project.

  • ADMIN_CLUSTER_NAME: The admin cluster that manages the user cluster. The admin cluster name is the last segment of the fully-specified cluster name that uniquely identifies the cluster in Google Cloud.

  • REGION: The Google Cloud region in which the GKE On-Prem API runs and stores cluster metadata. Specify us-west1 or another supported region. The region can't be changed after the cluster is enrolled.

gkectl 1.15.3+

Do the following steps on your admin workstation.

  1. Add the following section to the user cluster configuration file:

    gkeOnPremAPI:
  enabled:true
  location:REGION

    Replace REGION with the Google Cloud region in which the GKE On-Prem API runs and stores cluster metadata. Specify us-west1 or another supported region. We recommend that you use the same region that is configured in stackdriver.clusterLocation. The region can't be changed after the cluster is enrolled.

  2. Update the cluster:

    gkectl update cluster --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config USER_CLUSTER_CONFIG_FILE

    Replace the following:

    • ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.

    • USER_CLUSTER_CONFIG_FILE: the path of your user cluster configuration file.

Enroll an admin cluster

gcloud CLI 

gcloud container vmware admin-clusters enroll ADMIN_CLUSTER_NAME \
    --project=PROJECT_ID \
    --admin-cluster-membership=projects/PROJECT_ID/locations/global/memberships/ADMIN_CLUSTER_NAME \
    --location=REGION

Replace the following:

  • ADMIN_CLUSTER_NAME: The name of the admin cluster that you want to enroll.

  • PROJECT_ID The project ID of your fleet host project.

    The ADMIN_CLUSTER_NAME and PROJECT_ID are used to form the fully-specified cluster name for the --admin-cluster-membership flag.

  • REGION: The Google Cloud region in which the GKE On-Prem API runs and stores cluster metadata.. Specify us-west1 or another supported region. The region can't be changed after the cluster is enrolled.

gkectl 1.15.3+

Do the following steps on your admin workstation.

  1. Add the following section to the admin cluster configuration file:

    gkeOnPremAPI:
  enabled:true
  location:REGION

    Replace REGION with the Google Cloud region in which the GKE On-Prem API runs and stores cluster metadata. Specify us-west1 or another supported region. We recommend that you use the same region that is configured in stackdriver.clusterLocation. The region can't be changed after the cluster is enrolled.

  2. Update the cluster:

    gkectl update admin --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config ADMIN_CLUSTER_CONFIG_FILE

    Replace the following:

    • ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.

    • ADMIN_CLUSTER_CONFIG_FILE: the path of your admin cluster configuration file.

Get information about your cluster

After the cluster is enrolled, you can use the following commands to get information about your clusters:

User cluster

  • To describe a user cluster:
gcloud container vmware clusters describe USER_CLUSTER_NAME \
  --project=PROJECT_ID \
  --location=REGION
  • To list your user clusters:
gcloud container vmware clusters list \
  --project=PROJECT_ID \
  --location=-

When you set --location=-, that means to list all clusters in all regions. If you need to scope down the list, set --location to the region you specified when you enrolled the cluster.

Admin cluster

  • To describe an admin cluster:
gcloud container vmware admin-clusters describe ADMIN_CLUSTER_NAME \
  --project=PROJECT_ID \
  --location=REGION
  • To list your admin clusters:
gcloud container vmware admin-clusters list \
  --project=PROJECT_ID \
  --location=-

When you set --location=-, that means to list all clusters in all regions. If you need to scope down the list, set --location to the region you specified when you enrolled the cluster.

Connect to the cluster

After the cluster is enrolled with the GKE On-Prem API, you need to choose and configure an authentication method so that you can manage the cluster from the Google Cloud console. The authentication method that you select also controls access to the cluster from the command line. For more information, see the following: