Create a partial cluster

To mitigate the effects of the unavailability of user-specified VMs in specific regions at specific times (stockouts), Dataproc allows you to request the creation of a partial cluster by specifying a minimum number of primary workers that is acceptable to allow cluster creation.

Note: See Dataproc secondary workers to understand the difference between primary and secondary workers.

Standard cluster	Partial cluster
If one or more primary workers cannot be created and initialized, cluster creation fails. Workers that are created continue to run and incur charges until deleted by the user.	If the specified minimum number of workers can be created, the cluster is created. Failed (uninitialized) workers are deleted and do not incur charges. If the specified minimum number of workers cannot be created and initialized, the cluster is not created. Workers that are created are not deleted to allow for debugging.
Cluster creation time is optimized.	Longer cluster creation time can occur since all nodes must report provisioning status.
Single node clusters are available for creation.	Single node clusters are not available for creation.

Autoscaling:

Use autoscaling with partial cluster creation to help ensure that the target (full) number of primary workers is created. Autoscaling will try to acquire failed workers in the background if the workload requires them.

The following is a sample autoscaling policy that retries until the total number of primary worker instances reaches a target size of 10. The policy's minInstances and maxInstances match the minimum and total number of primary workers specified at cluster creation time (see How to create a partial cluster). Setting the scaleDownFactor to 0 prevents the cluster from scaling down from 10 to 8, and will help keep the number of workers at the maximum 10-worker limit.

workerConfig:
  minInstances: 8
  maxInstances: 10
basicAlgorithm:
  cooldownPeriod: 2m
  yarnConfig:
    scaleUpFactor: 1
    scaleDownFactor: 0
    gracefulDecommissionTimeout: 1h

How to create a partial cluster

You can use the Google Cloud CLI or the Dataproc API to create a Dataproc partial cluster.

Note: Dataproc partial cluster creation is not available in the Google Cloud console.

gcloud

To create a Dataproc partial cluster on the command line, run the following gcloud dataproc clusters create command locally in a terminal window or in Cloud Shell.

gcloud dataproc clusters create CLUSTER_NAME \
    --project=PROJECT \
    --region=REGION \
    --num-workers=NUM_WORKERS \
    --min-num-workers=MIN_NUM_WORKERS \
    other args ...

• CLUSTER_NAME: The cluster name must start with a lowercase letter followed by up to 51 lowercase letters, numbers, and hyphens, and cannot end with a hyphen.
• PROJECT: Specify the project associated with the job cluster.
• REGION: Specify the Compute Engine region where the job cluster will be located.
• NUM_WORKERS: The total number of primary workers in the cluster to create if available.
• MIN_NUM_WORKERS: The minimum number of primary workers to create if the specified total number of workers (NUM_WORKERS) cannot be created. Cluster creation fails if this minimum number of primary workers cannot be created (workers that are created are not deleted to allow for debugging). If this flag is omitted, standard cluster creation with the total number of primary workers (NUM_WORKERS) is attempted.

REST

To create a Dataproc partial cluster, specify the minimum number of primary workers in the workerConfig.minNumInstances field as part of a clusters.create request.

Note: You can click the Equivalent REST or command line links at the bottom of the left panel of the Dataproc Google Cloud console Create a cluster page to have the Console construct an equivalent API REST request or gcloud CLI command to use in your code or from the command line to create a cluster.

Display the number of provisioned workers

After creating a cluster, you can run the following gcloud CLI command to list the number of workers, including any secondary workers, provisioned in your cluster.

gcloud dataproc clusters list \
    --project=PROJECT \
    --region=REGION \
    --filter=clusterName=CLUSTER_NAME