This page describes the fields in the Google Distributed Cloud user cluster configuration file.
Generating a template for your configuration file
If you used gkeadm
to create your admin workstation, then gkeadm
generated
a template for your user cluster configuration file. And gkeadm
filled in
some of the fields for you.
If you did not use gkeadm
to create your admin workstation, you can use
gkectl
to generate a template for your user cluster configuration file.
To generate a template for your user cluster configuration file:
gkectl create-config cluster --config=OUTPUT_FILENAME
Replace OUTPUT_FILENAME
with a path of your choice for the
generated template. If you omit this flag, gkectl
names the file
user-cluster.yaml
and puts it in the current directory.
Template
Filling in your configuration file
In your configuration file, enter field values as described in the following sections.
name
String. A name of your choice for your user cluster. For example:
name: "my-user-cluster"
gkeOnPremVersion
String. The Google Distributed Cloud version for your user cluster. For example:
gkeOnPremVersion: 1.6.0-gke.7
vCenter
If you want all aspects of your vCenter environment to be the same as what you specified for your admin cluster, remove this section or leave it commented out.
If you want some aspects of your vCenter environment to be different from
what you specified for your admin cluster, fill in the relevant fields in this
section. Any fields that you set here in the vCenter
section override the
corresponding fields in your admin cluster configuration file.
vCenter.resourcePool
String. The name of the vCenter resource pool for your user cluster. If you are using a non-default resource pool, provide the name of your vCenter resource pool. For example:
vCenter: resourcePool: "MY-USER-POOL"
If you are using the default resource pool, provide the following value:
vCenter: resourcePool: "VCENTER_CLUSTER/Resources"
Replace VCENTER_CLUSTER
with the name of your vCenter cluster.
See Specifying the root resource pool for a standalone host.
vCenter.datastore
String. The name of the vCenter datastore for your user cluster. For example:
vCenter: datastore: "MY-USER-DATASTORE"
vCenter.caCertPath
String. When a client, like GKE on-prem, sends a request to your vCenter server, the server must prove its identity to the client by presenting a certificate or a certificate bundle. To verify the certificate or bundle, GKE on-prem must have the root certificate in the chain of trust.
Set vCenter.caCertPath
to the path of the root certificate. For example:
vCenter: caCertPath: "/usr/local/google/home/me/certs/user-vcenter-ca-cert.pem"
Your VMware installation has a certificate authority (CA) that issues a certificate to your vCenter server. The root certificate in the chain of trust is a self-signed certificate created by VMware.
If you do not want to use the VMWare CA, which is the default, you can configure VMware to use a different certificate authority.
If your vCenter server uses a certificate issued by the default VMware CA, download the certificate as follows:
curl -k "https://SERVER_ADDRESS/certs/download.zip" > download.zip
Replace SERVER_ADDRESS
with the address of your vCenter server.
Install the unzip
command and unzip the certificate file:
sudo apt-get install unzip unzip downloads.zip
If the unzip
command doesn't work the first time, enter the command again.
Find the certificate file in certs/lin
.
vCenter.credentials.fileRef.path
String. The path of a credentials configuration file that holds the username and password of your vCenter user account. The user account should have the Administrator role or equivalent privileges. See vSphere requirements. For example:
vCenter: credentials: fileRef: path: "my-config-directory/user-creds.yaml"
vCenter.credentials.fileRef.entry
String. The name of the credentials block, in your credentials configuration file, that holds the username and password of your vCenter user account. For example:
vCenter: credentials: fileRef: entry: "vcenter-creds"
enableDataplaneV2
Boolean. If you want to enable Dataplane V2
(Beta), set this to true
.
Otherwise set this to false
. For example:
enableDataplaneV2: true
network
This section holds information about your user cluster network.
network.hostConfig
This section holds information about NTP servers, DNS servers, and DNS search domains used by your cluster.
If you provided a value for one or both of the following fields, then fill in this section. Otherwise, remove this section.
loadBalancer.seesaw.ipBlockFilePath
network.ipMode.ipBlockFilePath
network.hostConfig.dnsServers
Array of strings. The addresses of DNS servers for the hosts to use. For example:
network: hostConfig: dnsServers: - "172.16.255.1" - "172.16.255.2"
network.hostConfig.ntpServers
Array of strings. The addresses of time servers for the hosts to use. For example:
network: hostConfig: ntpServers: - "216.239.35.0"
network.hostConfig.searchDomainsForDNS
Array of strings. DNS search domains for the hosts to use. These domains are used as part of a domain search list. For example:
network: hostConfig: searchDomainsForDNS: - "my.local.com"
network.ipMode.type
String. If you want your cluster nodes to get their IP address from a DHCP
server, set this to "dhcp"
. If you want your cluster nodes to have static IP
addresses chosen from a list that you provide, set this to "static"
. For
example:
network: ipMode: type: "static"
network.ipMode.ipBlockFilePath
If you set ipMode.type
to "static"
, fill in this field. If you set
ipMode.type
to "dhcp"
, remove this field.
String. The path of the IP block file for your cluster. For example:
network: ipMode: ipBlockFilePath: "/my-config-folder/user-cluster-ipblock.yaml"
network.serviceCIDR
and network.podCiDR
Strings. Your user cluster must have a range of IP addresses to use for Services
and a range of IP addresses to use for Pods. These ranges are specified by the
network.serviceCIDR
and network.podCIDR
fields. These fields are populated
with default values. If you like, you can change the populated values to values
of your choice.
The Service and Pod ranges must not overlap. Also, the Service and Pod ranges must not overlap with IP addresses that are used for nodes in any cluster.
Example:
network: serviceCIDR: "10.96.232.0/24" podCIDR: "192.168.0.0/16"
network.vCenter.networkName
String. The name of the vSphere network for your user cluster nodes.
If the name contains a special character, you must use an escape sequence for it.
Special characters | Escape sequence |
---|---|
Slash (/ ) |
%2f |
Backslash (\ ) |
%5c |
Percent sign (% ) |
%25 |
If the network name is not unique, it is possible to specify a path to the network, such as /DATACENTER/network/NETWORK_NAME.
For example:
network: vCenter: networkName: "MY-USER-CLUSTER-NETWORK"
loadBalancer
This section holds information about the load balancer for your user cluster.
loadBalancer.vips.controlPlaneVIP
The IP address that you have chosen to configure on the load balancer for the Kubernetes API server of the admin cluster. For example:
loadBalancer: vips: controlplaneVIP: "203.0.113.3"
loadBalancer.vips.ingressVIP
The IP address that you have chosen to configure on the load balancer for ingress traffic. For example:
loadBalancer: vips: ingressVIP: "203.0.113.4"
loadBalancer.kind
String. Set this to "Seesaw"
, "F5BigIP"
, or "ManualLB"
For example:
loadBalancer: kind: "Seesaw"
loadBalancer.manualLB
If you set loadbalancer.kind
to "manualLB"
, fill in this section. Otherwise,
remove this section or leave it commented out.
loadBalancer.manualLB.ingressHTTPNodePort
Integer. The ingress service in a user cluster is implemented as a
Kubernetes Services of type LoadBalancer
.
The Service has a
ServicePort
for HTTP. You must choose a nodePort
value for the HTTP ServicePorts.
Set this field to the nodePort
value. For example:
loadBalancer: manualLB: ingressHTTPNodePort: 32527
loadBalancer.manualLB.ingressHTTPSNodePort
Integer. The ingress service in a user cluster is implemented as a Service
of type LoadBalancer. The Service has a ServicePort for HTTPS. You must choose a
nodePort
value for the HTTPS ServicePort.
Set this field to the nodePort
value. For example:
loadBalancer: manualLB: ingressHTTPSNodePort: 30139
loadBalancer.manualLB.controlPlaneNodePort
Integer. The Kubernetes API server in the admin cluster is implemented as a
Service of type NodePort
. You must choose a nodePort
value for the Service.
Set this field to the nodePort
value. For example:
loadBalancer: manualLB: controlPlaneNodePort: 30968
loadBalancer.manualLB.addonsNodePort
Remove this field. It is not used in a user cluster.
loadBalancer.f5BigIP
If you set loadbalancer.kind
to "f5BigIP"
, fill in this section. Otherwise,
remove this section or leave it commented out.
loadBalancer.f5BigIP.address
String. The address of your F5 BIG-IP load balancer. For example:
loadBalancer: f5BigIP: address: "203.0.113.2"
loadBalancer.f5BigIP.fileRef.path
String. The path of a credentials configuration file that holds the username and password of an account that Google Distributed Cloud can use to connect to your F5 BIG-IP load balancer. For example:
loadBalancer: f5BigIP: fileRef: path: ""my-config-folder/user-creds.yaml"
loadBalancer.f5BigIP.fileRef.entry
String. The name of the credentials block, in your credentials configuration file, that holds the username and password of your F5 BIG-IP account. For example:
loadBalancer: f5BigIP: fileRef: entry: "f5-creds"
loadBalancer.f5BigIP.partition
String. The name of a BIG-IP partition that you created for your admin cluster. For example:
loadBalancer: f5BigIP: partition: "my-f5-admin-partition"
loadBalancer.f5BigIP.snatPoolName
String. If you are using SNAT, the name of your SNAT pool. If you are not using SNAT, remove this field or leave it commented out. For example:
loadBalancer: f5BigIP: snatPoolName: "my-snat-pool"
loadBalancer.seesaw
If you set loadbalancer.kind
to "Seesaw"
, fill in this section. Otherwise,
remove this section or leave it commented out.
loadBalancer.seesaw.ipBlockFilePath
String. Set this to the path of the IP block file for your Seesaw VM. For example:
loadbalancer: seesaw: ipBlockFilePath: "user-seesaw-ipblock.yaml"
loadBalancer.seesaw.vird
Integer. The virtual router identifier of your Seesaw VM or VM pair. This identifier must be unique in a VLAN. Valid range is 1-255. For example:
loadBalancer: seesaw: vrid: 125
loadBalancer.seesaw.masterIP
String. A VIP of your choice that your control plane Seesaw VM will advertise. For example:
loadBalancer: seesaw: masterIP: 172.16.20.21
loadBalancer.seesaw.cpus
Integer. The number of CPUs for each of your Seesaw VMs. For example:
loadBalancer:. seesaw: cpus: 8
loadBalancer.seesaw.memoryMB
Integer. The number of megabytes of memory for each of your Seesaw VMs. For example:
loadBalancer:. seesaw: memoryMB: 8192
loadBalancer.seesaw.vCenter.networkName
String. The name of the network that contains your Seesaw VMs. For example:
loadBalancer: seesaw: vCenter: networkName: "my-seesaw-network"
loadBalancer.seesaw.enableHA
Boolean. If you want to create a highly-available Seesaw load balancer, set this
to true
. Otherwise set this to false
. For example:
loadBalancer:. seesaw: enableHA: true
masterNode
This section holds information about the nodes, in the admin cluster, that serve as control plane nodes for your user cluster.
masterNode.cpus
Integer. The number of CPUs for each admin cluster node that serve as control planes for this user cluster. For example:
masterNode: cpus: 8
masterNode.memoryMB
Integer. The megabytes of memory for each admin cluster node that serves as a control plane for this user cluster. For example:
masterNode: memoryMB: 8192
masterNode.replicas
Integer. The number of control plane nodes for this user cluster. Set this field to 1 or 3. For example:
masterNode: replicas: 3
nodePools
Array of objects, each of which describes a node pool.
nodePools[i].name
String. A name of your choice for the node pool. For example:
nodePools: - name: "my-node-pool"
nodePools[i].cpus
Integer. The number of CPUs for each node in the pool. For example:
nodePools" - name: "my-node-pool" cpus: 8
nodePools[i].memoryMB
Integer. The megabytes of memory for each node in the pool. For example:
nodePools" - name: "my-node-pool" memoryMB: 8192
nodePools[i].replicas
Integer. The number of nodes in the pool. For example:
nodePools: - name: "my-node-pool" replicas: 5
noodePools[i].labels
Mapping. Labels to apply to each node in the pool. For example:
nodePools: - name: "my-node-pool" labels: environment: "production" tier: "cache"
nodePools[i].taints
Array of objects, each of which describes a taint. For example:
nodePools: - name: "my-node-pool" taints: - key: "staging" value: "true" effect: "NoSchedule"
nodePools[i].bootDiskSizeGB
Integer. The size of boot disk in gigabytes for each node in the pool. This configuration is available starting from Google Distributed Cloud version 1.5.0. For example:
nodePools" - name: "my-node-pool" bootDiskSizeGB: 40
nodePools[i].vsphere.datastore
String. The name of the vCenter datastore on which each node in the pool will be created. For example:
nodePools: - name: "my-node-pool" vsphere: datastore: "my-datastore"
antiAffinityGroups.enabled
Boolean. Set this to true
to enable DRS rule creation. Otherwise, set this to
false
. For example:
antiAffinityGroups: enabled: true
Google Distributed Cloud automatically creates VMware Distributed Resource Scheduler (DRS) anti-affinity rules for your user cluster's nodes, causing them to be spread across at least three physical hosts in your datacenter.
This feature requires that your vSphere environment meets the following conditions:
VMware DRS is enabled. VMware DRS requires vSphere Enterprise Plus license edition.
Your vSphere user account has the
Host.Inventory.Modify cluster
privilege.There are at least three physical hosts available.
Recall that if you have a vSphere Standard license, you cannot enable VMware DRS.
If you do not have DRS enabled, or if you do not have at least three hosts where
vSphere VMs can be scheduled, set antiAffinityGroups.enabled
to false
.
authentication
This section holds information about how cluster users are authenticated and authorized.
authentication.oidc
Do not use this section. Instead, after cluster creation, edit the ClientConfig custom resource as described in Configuring clusters for Anthos Identity Service with OIDC .
authentication.sni
If you want to provide an additional serving certificate for the cluster's Kubernetes API server, fill in this section. Otherwise, remove this section or leave it commented out.
authentication.sni.certPath
String. The path to a serving certificate for the Kubernetes API server. For example:
authentication: sni: certPath: "my-cert-folder/example.com.crt"
authentication.sni.keyPath
String. Path to the certificate's private key file. For example:
authentication: sni: keyPath: "my-cert-folder/example.com.key"
authentication.ldap
If you want to provide information about your LDAP provider, fill in this section. Otherwise, remove this section.
authentication.ldap.name
String. The name of your LDAP provider. For example:
authentication: ldap: name: "my-ldap-provider"
authentication.ldap.host
String. The hostname or IP address of your LDAP provider. For example:
authentication: ldap: host" "203.0.113.20"
authentication.ldap.connectionType
String. Set this to "insecure"
to use an unsecured connection for LDAP. To use
a secure connection, remove this field.
The default is to use LDAP over TLS. Set connectionType to"insecure"
to
change this and use an unsecured connection. If you set this to "insecure"
,
there is no need to supply a value for ldap.caPath
.
Example:
authentication: ldap: connectionType: "insecure"
authentication.ldap.caPath
String. The path of the root certificate for your LDAP provider. For example:
authentication: ldap: caPath: "my-cert-folder/ldap-cert.pem"
authentication.ldap.user.baseDN
String. The location for the user entries in the LDAP directory. For example:
authentication: ldap: user: baseDN: "DC=example-domain,DC=com"
authentication.ldap.user.userAttribute
String: The name of the attribute that precedes the user name in a DN. The
default is "CN"
. For example:
authentication: ldap: user: userAttribute: "CN"
authentication.ldap.user.memberAttribute
String. The name of the attribute that records a user's group membership. The
default is "memberOf"
. For example:
authentication: ldap: user: memberAttribute: "memberOf"
stackdriver
This section holds information about the Google Cloud project and service account you want to use for storing logs and metrics.
stackdriver.projectID
String. The ID of the Google Cloud project where you want to view logs. For example:
stackdriver: projectID: "my-logs-project"
stackdriver.clusterLocation
String. The Google Cloud region where you want to store logs. It is a good idea to choose a region that is near your on-premises data center. For example:
stackdriver: clusterLocation: "us-central1"
stackdriver.enableVPC
Boolean. If your cluster's network is controlled by a VPC, set this to
this field totrue
. This ensures that all telemetry flows through Google's
restricted IP addresses. Otherwise, set this field to false
. For example:
stackdriver: enableVPC: false
stackdriver.serviceAccountKeyPath
String. The path of the JSON key file for your logging-monitoring service account. For example:
stackdriver: serviceAccountKeyPath: "my-key-folder/log-mon-key.json"
stackdriver.disableVsphereResourceMetrics
Boolean. Set this to true
to disable the collection of metrics from vSphere.
Otherwise, set it to false
. For example:
stackdriver: disableVsphereResourceMetrics: true
gkeConnect
This section holds information about the Google Cloud project and service accounts you want to use to connect your cluster to Google Cloud.
gkeConnect.projectID
String. The ID of the Google Cloud project that you want to use for connecting your cluster to Google Cloud. For example:
gkeConnect: projectID: "my-connect-project-123"
gkeConnect.registerServiceAccountKeyPath
String. The path of the JSON key file for your connect-register service account. For example:
gkeConnect: registerServiceAccountKeyPath: "my-key-folder/connect-register-key.json"
gkeConnect.agentServiceAccountKeyPath
String. The path of the JSON key file for your connect-agent service account. For example:
gkeConnect: agentServiceAccountKeyPath: "my-key-folder/connect-agent-key.json"
cloudRun.enabled
Boolean. Set this to true
if you want to enable Cloud Run. Otherwise
set this to false
. For example:
cloudRun: enabled: true
usageMetering
If you want to enable usage metering for your cluster, then fill in this section. Otherwise, remove this section or leave it commented out.
usageMetering.bigQueryProjectID
String. The ID of the Google Cloud project where you want to store usage metering data. For example:
usageMetering: bigQueryProjectID: "my-bq-project"
usageMetering.bigQueryDatasetID
String. The ID of the BigQuery dataset where you want to store usage metering data. For example:
usageMetering: bigQueryDatasetID: "my-bq-dataset"
usageMetering.bigQueryServiceAccountKeyPath
String. The path of the JSON key file for your BigQuery service account. For example:
usageMetering: bigQueryServiceAccountKeyPath: "my-key-folder/bq-key.json"
usageMetering.enableConsumptionMetering
Boolean. Set this to true
if you want to enable consumption-based metering.
Otherwise set this to false. For example:
usageMetering: enableConsumptionMetering: true
cloudAuditLogging
If you want to integrate the audit logs from your cluster's Kubernetes API server with Cloud Audit Logs, fill in this section. Otherwise, remove this section or leave it commented out.
cloudAuditLogging.projectID
String. The project ID of the Google Cloud project where you want to store audit logs. For example:
cloudAuditLogging: projectID: "my-audit-project"
cloudAuditLogging.clusterLocation
String. The Google Cloud region where you want to store audit logs. It is a good idea to choose a region that is near your on-premises data center. For example:
cloudAuditLogging: clusterLocation: "us-central1"
cloudAuditLogging.serviceAccountKeyPath
String. The path of the JSON key file for your audit-logging service account. For example:
cloudAuditLogging: serviceAccountKeyPath: "my-key-folder/audit-log-key.json"
autoRepair.enabled
Boolean. Set this to true
to enable node auto repair. Otherwise, set it to
false
. For example:
autoRepair: enabled: true