Control access for a job using a custom service account

This document explains how to specify a Batch job's service account, which influences the resources and applications that a job's VMs can access. If you don't specify a custom service account, jobs default to using the Compute Engine default service account, which is automatically attached to all VMs in a project by default. Therefore, using a custom service account provides greater control in managing a job's permissions and is a recommended best practice for limiting privilege.

Learn more about a job's service account.

Before you begin

  1. If you haven't used Batch before, review Get started with Batch and enable Batch by completing the prerequisites for projects and users.

  2. To get the permissions that you need to control access for jobs using custom service accounts, ask your administrator to grant you the following IAM roles:

    For more information about granting roles, see Manage access to projects, folders, and organizations.

    You might also be able to get the required permissions through custom roles or other predefined roles.

  3. Identify the service account that you want to use for this job. Make sure this service account has all the necessary permissions to run your job.

    Learn more about viewing service accounts and the necessary permissions for a job's service account.

Create a job that uses a custom service account

To create a job that uses a custom service account, select one of the following methods:

  • Specify the custom service account in your job's definition, as shown in this section.
  • Use a Compute Engine instance template and specify the custom service account both in your instance template and in your job's definition.

This section provides an example for how to create a job that uses a custom service account. You can create a job that uses a custom service account using the gcloud CLI, Batch API, Java, Node.js, or Python.

gcloud

To create a job that uses a custom service account using the gcloud CLI, use the gcloud batch jobs submit command and specify the custom service account in the job's configuration file.

For example, to create a script job that uses a custom service account:

  1. Create a JSON file in the current directory named hello-world-service-account.json with the following contents:

    {
    "taskGroups": [
        {
            "taskSpec": {
                "runnables": [
                    {
                        "script": {
                            "text": "echo Hello World! This is task $BATCH_TASK_INDEX."
                        }
                    }
                ]
            }
        }
    ],
    "allocationPolicy": {
        "serviceAccount": {
            "email": "SERVICE_ACCOUNT_EMAIL"
        }
    }
}

    where SERVICE_ACCOUNT_EMAIL is the email address of your service account. If the serviceAccount field is not specified, the value is set to the default Compute Engine service account.

  2. Run the following command:

    gcloud batch jobs submit example-service-account-job \
  --location us-central1 \
  --config hello-world-service-account.json

API

To create a job that uses a custom service account using the Batch API, use the jobs.create method and specify your custom service account in the allocationPolicy field.

For example, to create a script job that uses a custom service account, make the following request:

POST https://batch.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/jobs?job_id=example-service-account-job

{
    "taskGroups": [
        {
            "taskSpec": {
                "runnables": [
                    {
                        "script": {
                            "text": "echo Hello World! This is task $BATCH_TASK_INDEX."
                        }
                    }
                ]
            }
        }
    ],
    "allocationPolicy": {
        "serviceAccount": {
            "email": "SERVICE_ACCOUNT_EMAIL"
        }
    }
}

Replace the following:

Java 


import com.google.cloud.batch.v1.AllocationPolicy;
import com.google.cloud.batch.v1.BatchServiceClient;
import com.google.cloud.batch.v1.CreateJobRequest;
import com.google.cloud.batch.v1.Job;
import com.google.cloud.batch.v1.LogsPolicy;
import com.google.cloud.batch.v1.LogsPolicy.Destination;
import com.google.cloud.batch.v1.Runnable;
import com.google.cloud.batch.v1.Runnable.Script;
import com.google.cloud.batch.v1.ServiceAccount;
import com.google.cloud.batch.v1.TaskGroup;
import com.google.cloud.batch.v1.TaskSpec;
import com.google.protobuf.Duration;
import java.io.IOException;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;

public class CreateBatchUsingServiceAccount {

  public static void main(String[] args)
      throws IOException, ExecutionException, InterruptedException, TimeoutException {
    // TODO(developer): Replace these variables before running the sample.
    // Project ID or project number of the Google Cloud project you want to use.
    String projectId = "YOUR_PROJECT_ID";
    // Name of the region you want to use to run the job. Regions that are
    // available for Batch are listed on: https://cloud.google.com/batch/docs/get-started#locations
    String region = "europe-central2";
    // The name of the job that will be created.
    // It needs to be unique for each project and region pair.
    String jobName = "JOB_NAME";
    // The email address of your service account.
    String serviceAccountEmail = "EMAIL";

    createBatchUsingServiceAccount(projectId, region, jobName, serviceAccountEmail);
  }

  // Create a job that uses a custom service account
  public static Job createBatchUsingServiceAccount(String projectId, String region, String jobName,
                                                   String serviceAccountEmail)
      throws IOException, ExecutionException, InterruptedException, TimeoutException {
    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests.
    try (BatchServiceClient batchServiceClient = BatchServiceClient.create()) {
      // Define what will be done as part of the job.
      Runnable runnable =
          Runnable.newBuilder()
              .setScript(
                  Script.newBuilder()
                      .setText(
                          "echo Hello world! This is task ${BATCH_TASK_INDEX}. "
                                  + "This job has a total of ${BATCH_TASK_COUNT} tasks.")
                      // You can also run a script from a file. Just remember, that needs to be a
                      // script that's already on the VM that will be running the job.
                      // Using setText() and setPath() is mutually exclusive.
                      // .setPath("/tmp/test.sh")
                      .build())
              .build();

      TaskSpec task = TaskSpec.newBuilder()
              // Jobs can be divided into tasks. In this case, we have only one task.
              .addRunnables(runnable)
              .setMaxRetryCount(2)
              .setMaxRunDuration(Duration.newBuilder().setSeconds(3600).build())
              .build();

      // Tasks are grouped inside a job using TaskGroups.
      // Currently, it's possible to have only one task group.
      TaskGroup taskGroup = TaskGroup.newBuilder()
          .setTaskCount(3)
          .setParallelism(1)
          .setTaskSpec(task)
          .build();

      ServiceAccount.Builder serviceAccount = ServiceAccount.newBuilder();

      // If the serviceAccount field is not specified,
      // the value is set to the default Compute Engine service account.
      if (serviceAccountEmail != null) {
        serviceAccount.setEmail(serviceAccountEmail);
      }

      // Attach service account that VMs will run as.
      AllocationPolicy allocationPolicy = AllocationPolicy.newBuilder()
              .setServiceAccount(serviceAccount)
              .build();

      Job job =
          Job.newBuilder()
              .addTaskGroups(taskGroup)
              .setAllocationPolicy(allocationPolicy)
              .putLabels("env", "testing")
              .putLabels("type", "script")
              // We use Cloud Logging as it's an out of the box available option.
              .setLogsPolicy(
                  LogsPolicy.newBuilder().setDestination(Destination.CLOUD_LOGGING))
              .build();

      CreateJobRequest createJobRequest =
          CreateJobRequest.newBuilder()
              // The job's parent is the region in which the job will run.
              .setParent(String.format("projects/%s/locations/%s", projectId, region))
              .setJob(job)
              .setJobId(jobName)
              .build();

      Job result =
          batchServiceClient
              .createJobCallable()
              .futureCall(createJobRequest)
              .get(5, TimeUnit.MINUTES);

      System.out.printf("Successfully created the job: %s", result.getName());

      return result;
    }
  }
}

Node.js 

// Imports the Batch library
const batchLib = require('@google-cloud/batch');
const batch = batchLib.protos.google.cloud.batch.v1;
// Instantiates a client
const batchClient = new batchLib.v1.BatchServiceClient();

async function main() {
  /**
   * TODO(developer): Update these variables before running the sample.
   */
  // Project ID or project number of the Google Cloud project you want to use.
  const projectId = await batchClient.getProjectId();
  // Name of the region you want to use to run the job. Regions that are
  // available for Batch are listed on: https://cloud.google.com/batch/docs/get-started#locations
  const region = 'europe-central2';
  // The name of the job that will be created.
  // It needs to be unique for each project and region pair.
  const jobName = 'batch-service-account-job';
  /**
   * TODO(developer): Uncomment and update serviceAccountEmail if you do not want to use default Compute Engine service account.
   */
  // The email address of your service account.
  // const serviceAccountEmail = 'email';
  let serviceAccountEmail;

  // Define what will be done as part of the job.
  const runnable = new batch.Runnable({
    script: new batch.Runnable.Script({
      commands: ['-c', 'echo Hello world! This is task ${BATCH_TASK_INDEX}.'],
    }),
  });

  const task = new batch.TaskSpec({
    runnables: [runnable],
    maxRetryCount: 2,
    maxRunDuration: {seconds: 3600},
  });

  // Tasks are grouped inside a job using TaskGroups.
  const group = new batch.TaskGroup({
    taskCount: 3,
    taskSpec: task,
  });

  const serviceAccount = new batch.ServiceAccount();

  if (serviceAccountEmail) {
    serviceAccount.email = serviceAccountEmail;
  }

  const allocationPolicy = new batch.AllocationPolicy({
    // If the serviceAccount field is not specified, the value is set to the default Compute Engine service account.
    serviceAccount,
  });

  const job = new batch.Job({
    name: jobName,
    taskGroups: [group],
    labels: {env: 'testing', type: 'script'},
    allocationPolicy,
    // We use Cloud Logging as it's an option available out of the box
    logsPolicy: new batch.LogsPolicy({
      destination: batch.LogsPolicy.Destination.CLOUD_LOGGING,
    }),
  });

  // The job's parent is the project and region in which the job will run
  const parent = `projects/${projectId}/locations/${region}`;

  async function callCreateBatchServiceAccountJob() {
    // Construct request
    const request = {
      parent,
      jobId: jobName,
      job,
    };

    // Run request
    const [response] = await batchClient.createJob(request);
    console.log(JSON.stringify(response));
  }
  await callCreateBatchServiceAccountJob();

Python 

from google.cloud import batch_v1


def create_with_custom_service_account_job(
    project_id: str, region: str, job_name: str, service_account_email: str
) -> batch_v1.Job:
    """
    This method shows how to create a sample Batch Job that will run
    a simple command on Cloud Compute instances with custom service account.

    Args:
        project_id: project ID or project number of the Cloud project you want to use.
        region: name of the region you want to use to run the job. Regions that are
            available for Batch are listed on: https://cloud.google.com/batch/docs/get-started#locations
        job_name: the name of the job that will be created.
            It needs to be unique for each project and region pair.
        service_account_email: custom service account email

    Returns:
        A job object representing the job created.
    """
    client = batch_v1.BatchServiceClient()

    # Define what will be done as part of the job.
    task = batch_v1.TaskSpec()
    runnable = batch_v1.Runnable()
    runnable.script = batch_v1.Runnable.Script()
    runnable.script.text = "echo Hello world! from task ${BATCH_TASK_INDEX}. This job has a total of ${BATCH_TASK_COUNT} tasks."
    task.runnables = [runnable]
    task.max_retry_count = 2
    task.max_run_duration = "3600s"

    # Tasks are grouped inside a job using TaskGroups.
    # Currently, it's possible to have only one task group.
    group = batch_v1.TaskGroup()
    group.task_count = 4
    group.task_spec = task

    # Policies are used to define on what kind of virtual machines the tasks will run on.
    # Read more about local disks here: https://cloud.google.com/compute/docs/disks/persistent-disks
    policy = batch_v1.AllocationPolicy.InstancePolicy()
    policy.machine_type = "e2-standard-4"
    instances = batch_v1.AllocationPolicy.InstancePolicyOrTemplate()
    instances.policy = policy
    allocation_policy = batch_v1.AllocationPolicy()
    allocation_policy.instances = [instances]

    # Defines the service account for Batch-created VMs. If omitted, the [default account]
    # More details: https://cloud.google.com/compute/docs/access/service-accounts#default_service_account
    service_account = batch_v1.ServiceAccount()
    service_account.email = service_account_email
    allocation_policy.service_account = service_account

    job = batch_v1.Job()
    job.task_groups = [group]
    job.allocation_policy = allocation_policy
    job.labels = {"env": "testing", "type": "script"}

    create_request = batch_v1.CreateJobRequest()
    create_request.job = job
    create_request.job_id = job_name
    # The job's parent is the region in which the job will run
    create_request.parent = f"projects/{project_id}/locations/{region}"

    return client.create_job(create_request)

What's next