Package com.google.dataflow.v1beta3 (0.10.0)

A client to Dataflow API

The interfaces provided are listed below, along with usage samples.

JobsV1Beta3Client

Service Description: Provides a method to create and modify Google Cloud Dataflow jobs. A Job is a multi-stage computation graph run by the Cloud Dataflow service.

Sample for JobsV1Beta3Client:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (JobsV1Beta3Client jobsV1Beta3Client = JobsV1Beta3Client.create()) {
   CreateJobRequest request =
       CreateJobRequest.newBuilder()
           .setProjectId("projectId-894832108")
           .setJob(Job.newBuilder().build())
           .setView(JobView.forNumber(0))
           .setReplaceJobId("replaceJobId441554788")
           .setLocation("location1901043637")
           .build();
   Job response = jobsV1Beta3Client.createJob(request);
 }

MessagesV1Beta3Client

Service Description: The Dataflow Messages API is used for monitoring the progress of Dataflow jobs.

Sample for MessagesV1Beta3Client:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (MessagesV1Beta3Client messagesV1Beta3Client = MessagesV1Beta3Client.create()) {
   ListJobMessagesRequest request =
       ListJobMessagesRequest.newBuilder()
           .setProjectId("projectId-894832108")
           .setJobId("jobId101296568")
           .setMinimumImportance(JobMessageImportance.forNumber(0))
           .setPageSize(883849137)
           .setPageToken("pageToken873572522")
           .setStartTime(Timestamp.newBuilder().build())
           .setEndTime(Timestamp.newBuilder().build())
           .setLocation("location1901043637")
           .build();
   for (JobMessage element : messagesV1Beta3Client.listJobMessages(request).iterateAll()) {
     // doThingsWith(element);
   }
 }

MetricsV1Beta3Client

Service Description: The Dataflow Metrics API lets you monitor the progress of Dataflow jobs.

Sample for MetricsV1Beta3Client:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (MetricsV1Beta3Client metricsV1Beta3Client = MetricsV1Beta3Client.create()) {
   GetJobMetricsRequest request =
       GetJobMetricsRequest.newBuilder()
           .setProjectId("projectId-894832108")
           .setJobId("jobId101296568")
           .setStartTime(Timestamp.newBuilder().build())
           .setLocation("location1901043637")
           .build();
   JobMetrics response = metricsV1Beta3Client.getJobMetrics(request);
 }

SnapshotsV1Beta3Client

Service Description: Provides methods to manage snapshots of Google Cloud Dataflow jobs.

Sample for SnapshotsV1Beta3Client:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (SnapshotsV1Beta3Client snapshotsV1Beta3Client = SnapshotsV1Beta3Client.create()) {
   GetSnapshotRequest request =
       GetSnapshotRequest.newBuilder()
           .setProjectId("projectId-894832108")
           .setSnapshotId("snapshotId-1113817601")
           .setLocation("location1901043637")
           .build();
   Snapshot response = snapshotsV1Beta3Client.getSnapshot(request);
 }

TemplatesServiceClient

Service Description: Provides a method to create Cloud Dataflow jobs from templates.

Sample for TemplatesServiceClient:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (TemplatesServiceClient templatesServiceClient = TemplatesServiceClient.create()) {
   CreateJobFromTemplateRequest request =
       CreateJobFromTemplateRequest.newBuilder()
           .setProjectId("projectId-894832108")
           .setJobName("jobName-1438096408")
           .putAllParameters(new HashMap<String, String>())
           .setEnvironment(RuntimeEnvironment.newBuilder().build())
           .setLocation("location1901043637")
           .build();
   Job response = templatesServiceClient.createJobFromTemplate(request);
 }

FlexTemplatesServiceClient

Service Description: Provides a service for Flex templates. This feature is not ready yet.

Sample for FlexTemplatesServiceClient:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (FlexTemplatesServiceClient flexTemplatesServiceClient =
     FlexTemplatesServiceClient.create()) {
   LaunchFlexTemplateRequest request =
       LaunchFlexTemplateRequest.newBuilder()
           .setProjectId("projectId-894832108")
           .setLaunchParameter(LaunchFlexTemplateParameter.newBuilder().build())
           .setLocation("location1901043637")
           .setValidateOnly(true)
           .build();
   LaunchFlexTemplateResponse response = flexTemplatesServiceClient.launchFlexTemplate(request);
 }

Classes

AutoscalingEvent

A structured message reporting an autoscaling decision made by the Dataflow service.

Protobuf type google.dataflow.v1beta3.AutoscalingEvent

AutoscalingEvent.Builder

A structured message reporting an autoscaling decision made by the Dataflow service.

Protobuf type google.dataflow.v1beta3.AutoscalingEvent

AutoscalingSettings

Settings for WorkerPool autoscaling.

Protobuf type google.dataflow.v1beta3.AutoscalingSettings

AutoscalingSettings.Builder

Settings for WorkerPool autoscaling.

Protobuf type google.dataflow.v1beta3.AutoscalingSettings

BigQueryIODetails

Metadata for a BigQuery connector used by the job.

Protobuf type google.dataflow.v1beta3.BigQueryIODetails

BigQueryIODetails.Builder

Metadata for a BigQuery connector used by the job.

Protobuf type google.dataflow.v1beta3.BigQueryIODetails

BigTableIODetails

Metadata for a Cloud Bigtable connector used by the job.

Protobuf type google.dataflow.v1beta3.BigTableIODetails

BigTableIODetails.Builder

Metadata for a Cloud Bigtable connector used by the job.

Protobuf type google.dataflow.v1beta3.BigTableIODetails

CheckActiveJobsRequest

Request to check is active jobs exists for a project

Protobuf type google.dataflow.v1beta3.CheckActiveJobsRequest

CheckActiveJobsRequest.Builder

Request to check is active jobs exists for a project

Protobuf type google.dataflow.v1beta3.CheckActiveJobsRequest

CheckActiveJobsResponse

Response for CheckActiveJobsRequest.

Protobuf type google.dataflow.v1beta3.CheckActiveJobsResponse

CheckActiveJobsResponse.Builder

Response for CheckActiveJobsRequest.

Protobuf type google.dataflow.v1beta3.CheckActiveJobsResponse

ComputationTopology

All configuration data for a particular Computation.

Protobuf type google.dataflow.v1beta3.ComputationTopology

ComputationTopology.Builder

All configuration data for a particular Computation.

Protobuf type google.dataflow.v1beta3.ComputationTopology

ContainerSpec

Container Spec.

Protobuf type google.dataflow.v1beta3.ContainerSpec

ContainerSpec.Builder

Container Spec.

Protobuf type google.dataflow.v1beta3.ContainerSpec

CreateJobFromTemplateRequest

A request to create a Cloud Dataflow job from a template.

Protobuf type google.dataflow.v1beta3.CreateJobFromTemplateRequest

CreateJobFromTemplateRequest.Builder

A request to create a Cloud Dataflow job from a template.

Protobuf type google.dataflow.v1beta3.CreateJobFromTemplateRequest

CreateJobRequest

Request to create a Cloud Dataflow job.

Protobuf type google.dataflow.v1beta3.CreateJobRequest

CreateJobRequest.Builder

Request to create a Cloud Dataflow job.

Protobuf type google.dataflow.v1beta3.CreateJobRequest

CustomSourceLocation

Identifies the location of a custom souce.

Protobuf type google.dataflow.v1beta3.CustomSourceLocation

CustomSourceLocation.Builder

Identifies the location of a custom souce.

Protobuf type google.dataflow.v1beta3.CustomSourceLocation

DataDiskAssignment

Data disk assignment for a given VM instance.

Protobuf type google.dataflow.v1beta3.DataDiskAssignment

DataDiskAssignment.Builder

Data disk assignment for a given VM instance.

Protobuf type google.dataflow.v1beta3.DataDiskAssignment

DatastoreIODetails

Metadata for a Datastore connector used by the job.

Protobuf type google.dataflow.v1beta3.DatastoreIODetails

DatastoreIODetails.Builder

Metadata for a Datastore connector used by the job.

Protobuf type google.dataflow.v1beta3.DatastoreIODetails

DebugOptions

Describes any options that have an effect on the debugging of pipelines.

Protobuf type google.dataflow.v1beta3.DebugOptions

DebugOptions.Builder

Describes any options that have an effect on the debugging of pipelines.

Protobuf type google.dataflow.v1beta3.DebugOptions

DeleteSnapshotRequest

Request to delete a snapshot.

Protobuf type google.dataflow.v1beta3.DeleteSnapshotRequest

DeleteSnapshotRequest.Builder

Request to delete a snapshot.

Protobuf type google.dataflow.v1beta3.DeleteSnapshotRequest

DeleteSnapshotResponse

Response from deleting a snapshot.

Protobuf type google.dataflow.v1beta3.DeleteSnapshotResponse

DeleteSnapshotResponse.Builder

Response from deleting a snapshot.

Protobuf type google.dataflow.v1beta3.DeleteSnapshotResponse

Disk

Describes the data disk used by a workflow job.

Protobuf type google.dataflow.v1beta3.Disk

Disk.Builder

Describes the data disk used by a workflow job.

Protobuf type google.dataflow.v1beta3.Disk

DisplayData

Data provided with a pipeline or transform to provide descriptive info.

Protobuf type google.dataflow.v1beta3.DisplayData

DisplayData.Builder

Data provided with a pipeline or transform to provide descriptive info.

Protobuf type google.dataflow.v1beta3.DisplayData

DynamicTemplateLaunchParams

Params which should be passed when launching a dynamic template.

Protobuf type google.dataflow.v1beta3.DynamicTemplateLaunchParams

DynamicTemplateLaunchParams.Builder

Params which should be passed when launching a dynamic template.

Protobuf type google.dataflow.v1beta3.DynamicTemplateLaunchParams

Environment

Describes the environment in which a Dataflow Job runs.

Protobuf type google.dataflow.v1beta3.Environment

Environment.Builder

Describes the environment in which a Dataflow Job runs.

Protobuf type google.dataflow.v1beta3.Environment

ExecutionStageState

A message describing the state of a particular execution stage.

Protobuf type google.dataflow.v1beta3.ExecutionStageState

ExecutionStageState.Builder

A message describing the state of a particular execution stage.

Protobuf type google.dataflow.v1beta3.ExecutionStageState

Description of the composing transforms, names/ids, and input/outputs of a stage of execution. Some composing transforms and sources may have been generated by the Dataflow service during execution planning.

Protobuf type google.dataflow.v1beta3.ExecutionStageSummary

ExecutionStageSummary.Builder

Protobuf type google.dataflow.v1beta3.ExecutionStageSummary

ExecutionStageSummary.ComponentSource

Description of an interstitial value between transforms in an execution stage.

Protobuf type google.dataflow.v1beta3.ExecutionStageSummary.ComponentSource

ExecutionStageSummary.ComponentSource.Builder

Description of an interstitial value between transforms in an execution stage.

Protobuf type google.dataflow.v1beta3.ExecutionStageSummary.ComponentSource

ExecutionStageSummary.ComponentTransform

Description of a transform executed as part of an execution stage.

Protobuf type google.dataflow.v1beta3.ExecutionStageSummary.ComponentTransform

ExecutionStageSummary.ComponentTransform.Builder

Description of a transform executed as part of an execution stage.

Protobuf type google.dataflow.v1beta3.ExecutionStageSummary.ComponentTransform

ExecutionStageSummary.StageSource

Description of an input or output of an execution stage.

Protobuf type google.dataflow.v1beta3.ExecutionStageSummary.StageSource

ExecutionStageSummary.StageSource.Builder

Description of an input or output of an execution stage.

Protobuf type google.dataflow.v1beta3.ExecutionStageSummary.StageSource

FailedLocation

Indicates which regional endpoint failed to respond to a request for data.

Protobuf type google.dataflow.v1beta3.FailedLocation

FailedLocation.Builder

Indicates which regional endpoint failed to respond to a request for data.

Protobuf type google.dataflow.v1beta3.FailedLocation

FileIODetails

Metadata for a File connector used by the job.

Protobuf type google.dataflow.v1beta3.FileIODetails

FileIODetails.Builder

Metadata for a File connector used by the job.

Protobuf type google.dataflow.v1beta3.FileIODetails

FlexTemplateRuntimeEnvironment

The environment values to be set at runtime for flex template.

Protobuf type google.dataflow.v1beta3.FlexTemplateRuntimeEnvironment

FlexTemplateRuntimeEnvironment.Builder

The environment values to be set at runtime for flex template.

Protobuf type google.dataflow.v1beta3.FlexTemplateRuntimeEnvironment

FlexTemplatesServiceClient

Service Description: Provides a service for Flex templates. This feature is not ready yet.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (FlexTemplatesServiceClient flexTemplatesServiceClient =
     FlexTemplatesServiceClient.create()) {
   LaunchFlexTemplateRequest request =
       LaunchFlexTemplateRequest.newBuilder()
           .setProjectId("projectId-894832108")
           .setLaunchParameter(LaunchFlexTemplateParameter.newBuilder().build())
           .setLocation("location1901043637")
           .setValidateOnly(true)
           .build();
   LaunchFlexTemplateResponse response = flexTemplatesServiceClient.launchFlexTemplate(request);
 }

Note: close() needs to be called on the FlexTemplatesServiceClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

Many parameters require resource names to be formatted in a particular way. To assist with these names, this class includes a format method for each type of name, and additionally a parse method to extract the individual identifiers contained within names that are returned.

This class can be customized by passing in a custom instance of FlexTemplatesServiceSettings to create(). For example:

To customize credentials:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 FlexTemplatesServiceSettings flexTemplatesServiceSettings =
     FlexTemplatesServiceSettings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 FlexTemplatesServiceClient flexTemplatesServiceClient =
     FlexTemplatesServiceClient.create(flexTemplatesServiceSettings);

To customize the endpoint:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 FlexTemplatesServiceSettings flexTemplatesServiceSettings =
     FlexTemplatesServiceSettings.newBuilder().setEndpoint(myEndpoint).build();
 FlexTemplatesServiceClient flexTemplatesServiceClient =
     FlexTemplatesServiceClient.create(flexTemplatesServiceSettings);

To use REST (HTTP1.1/JSON) transport (instead of gRPC) for sending and receiving requests over the wire:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 FlexTemplatesServiceSettings flexTemplatesServiceSettings =
     FlexTemplatesServiceSettings.newHttpJsonBuilder().build();
 FlexTemplatesServiceClient flexTemplatesServiceClient =
     FlexTemplatesServiceClient.create(flexTemplatesServiceSettings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

FlexTemplatesServiceGrpc

Provides a service for Flex templates. This feature is not ready yet.

FlexTemplatesServiceGrpc.FlexTemplatesServiceBlockingStub

Provides a service for Flex templates. This feature is not ready yet.

FlexTemplatesServiceGrpc.FlexTemplatesServiceFutureStub

Provides a service for Flex templates. This feature is not ready yet.

FlexTemplatesServiceGrpc.FlexTemplatesServiceImplBase

Provides a service for Flex templates. This feature is not ready yet.

FlexTemplatesServiceGrpc.FlexTemplatesServiceStub

Provides a service for Flex templates. This feature is not ready yet.

FlexTemplatesServiceSettings

Settings class to configure an instance of FlexTemplatesServiceClient.

The default instance has everything set to sensible defaults:

The default service address (dataflow.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of launchFlexTemplate to 30 seconds:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 FlexTemplatesServiceSettings.Builder flexTemplatesServiceSettingsBuilder =
     FlexTemplatesServiceSettings.newBuilder();
 flexTemplatesServiceSettingsBuilder
     .launchFlexTemplateSettings()
     .setRetrySettings(
         flexTemplatesServiceSettingsBuilder
             .launchFlexTemplateSettings()
             .getRetrySettings()
             .toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 FlexTemplatesServiceSettings flexTemplatesServiceSettings =
     flexTemplatesServiceSettingsBuilder.build();

FlexTemplatesServiceSettings.Builder

Builder for FlexTemplatesServiceSettings.

GetJobExecutionDetailsRequest

Request to get job execution details.

Protobuf type google.dataflow.v1beta3.GetJobExecutionDetailsRequest

GetJobExecutionDetailsRequest.Builder

Request to get job execution details.

Protobuf type google.dataflow.v1beta3.GetJobExecutionDetailsRequest

GetJobMetricsRequest

Request to get job metrics.

Protobuf type google.dataflow.v1beta3.GetJobMetricsRequest

GetJobMetricsRequest.Builder

Request to get job metrics.

Protobuf type google.dataflow.v1beta3.GetJobMetricsRequest

GetJobRequest

Request to get the state of a Cloud Dataflow job.

Protobuf type google.dataflow.v1beta3.GetJobRequest

GetJobRequest.Builder

Request to get the state of a Cloud Dataflow job.

Protobuf type google.dataflow.v1beta3.GetJobRequest

GetSnapshotRequest

Request to get information about a snapshot

Protobuf type google.dataflow.v1beta3.GetSnapshotRequest

GetSnapshotRequest.Builder

Request to get information about a snapshot

Protobuf type google.dataflow.v1beta3.GetSnapshotRequest

GetStageExecutionDetailsRequest

Request to get information about a particular execution stage of a job. Currently only tracked for Batch jobs.

Protobuf type google.dataflow.v1beta3.GetStageExecutionDetailsRequest

GetStageExecutionDetailsRequest.Builder

Request to get information about a particular execution stage of a job. Currently only tracked for Batch jobs.

Protobuf type google.dataflow.v1beta3.GetStageExecutionDetailsRequest

GetTemplateRequest

A request to retrieve a Cloud Dataflow job template.

Protobuf type google.dataflow.v1beta3.GetTemplateRequest

GetTemplateRequest.Builder

A request to retrieve a Cloud Dataflow job template.

Protobuf type google.dataflow.v1beta3.GetTemplateRequest

GetTemplateResponse

The response to a GetTemplate request.

Protobuf type google.dataflow.v1beta3.GetTemplateResponse

GetTemplateResponse.Builder

The response to a GetTemplate request.

Protobuf type google.dataflow.v1beta3.GetTemplateResponse

InvalidTemplateParameters

Used in the error_details field of a google.rpc.Status message, this indicates problems with the template parameter.

Protobuf type google.dataflow.v1beta3.InvalidTemplateParameters

InvalidTemplateParameters.Builder

Used in the error_details field of a google.rpc.Status message, this indicates problems with the template parameter.

Protobuf type google.dataflow.v1beta3.InvalidTemplateParameters

InvalidTemplateParameters.ParameterViolation

A specific template-parameter violation.

Protobuf type google.dataflow.v1beta3.InvalidTemplateParameters.ParameterViolation

InvalidTemplateParameters.ParameterViolation.Builder

A specific template-parameter violation.

Protobuf type google.dataflow.v1beta3.InvalidTemplateParameters.ParameterViolation

Job

Defines a job to be run by the Cloud Dataflow service.

Protobuf type google.dataflow.v1beta3.Job

Job.Builder

Defines a job to be run by the Cloud Dataflow service.

Protobuf type google.dataflow.v1beta3.Job

JobExecutionDetails

Information about the execution of a job.

Protobuf type google.dataflow.v1beta3.JobExecutionDetails

JobExecutionDetails.Builder

Information about the execution of a job.

Protobuf type google.dataflow.v1beta3.JobExecutionDetails

JobExecutionInfo

Additional information about how a Cloud Dataflow job will be executed that isn't contained in the submitted job.

Protobuf type google.dataflow.v1beta3.JobExecutionInfo

JobExecutionInfo.Builder

Additional information about how a Cloud Dataflow job will be executed that isn't contained in the submitted job.

Protobuf type google.dataflow.v1beta3.JobExecutionInfo

JobExecutionStageInfo

Contains information about how a particular google.dataflow.v1beta3.Step will be executed.

Protobuf type google.dataflow.v1beta3.JobExecutionStageInfo

JobExecutionStageInfo.Builder

Contains information about how a particular google.dataflow.v1beta3.Step will be executed.

Protobuf type google.dataflow.v1beta3.JobExecutionStageInfo

JobMessage

A particular message pertaining to a Dataflow job.

Protobuf type google.dataflow.v1beta3.JobMessage

JobMessage.Builder

A particular message pertaining to a Dataflow job.

Protobuf type google.dataflow.v1beta3.JobMessage

JobMetadata

Metadata available primarily for filtering jobs. Will be included in the ListJob response and Job SUMMARY view.

Protobuf type google.dataflow.v1beta3.JobMetadata

JobMetadata.Builder

Metadata available primarily for filtering jobs. Will be included in the ListJob response and Job SUMMARY view.

Protobuf type google.dataflow.v1beta3.JobMetadata

JobMetrics

JobMetrics contains a collection of metrics describing the detailed progress of a Dataflow job. Metrics correspond to user-defined and system-defined metrics in the job. This resource captures only the most recent values of each metric; time-series data can be queried for them (under the same metric names) from Cloud Monitoring.

Protobuf type google.dataflow.v1beta3.JobMetrics

JobMetrics.Builder

Protobuf type google.dataflow.v1beta3.JobMetrics

JobsProto

JobsV1Beta3Client

Service Description: Provides a method to create and modify Google Cloud Dataflow jobs. A Job is a multi-stage computation graph run by the Cloud Dataflow service.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (JobsV1Beta3Client jobsV1Beta3Client = JobsV1Beta3Client.create()) {
   CreateJobRequest request =
       CreateJobRequest.newBuilder()
           .setProjectId("projectId-894832108")
           .setJob(Job.newBuilder().build())
           .setView(JobView.forNumber(0))
           .setReplaceJobId("replaceJobId441554788")
           .setLocation("location1901043637")
           .build();
   Job response = jobsV1Beta3Client.createJob(request);
 }

Note: close() needs to be called on the JobsV1Beta3Client object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

This class can be customized by passing in a custom instance of JobsV1Beta3Settings to create(). For example:

To customize credentials:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 JobsV1Beta3Settings jobsV1Beta3Settings =
     JobsV1Beta3Settings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 JobsV1Beta3Client jobsV1Beta3Client = JobsV1Beta3Client.create(jobsV1Beta3Settings);

To customize the endpoint:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 JobsV1Beta3Settings jobsV1Beta3Settings =
     JobsV1Beta3Settings.newBuilder().setEndpoint(myEndpoint).build();
 JobsV1Beta3Client jobsV1Beta3Client = JobsV1Beta3Client.create(jobsV1Beta3Settings);

To use REST (HTTP1.1/JSON) transport (instead of gRPC) for sending and receiving requests over the wire:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 JobsV1Beta3Settings jobsV1Beta3Settings = JobsV1Beta3Settings.newHttpJsonBuilder().build();
 JobsV1Beta3Client jobsV1Beta3Client = JobsV1Beta3Client.create(jobsV1Beta3Settings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

JobsV1Beta3Client.AggregatedListJobsFixedSizeCollection

JobsV1Beta3Client.AggregatedListJobsPage

JobsV1Beta3Client.AggregatedListJobsPagedResponse

JobsV1Beta3Client.ListJobsFixedSizeCollection

JobsV1Beta3Client.ListJobsPage

JobsV1Beta3Client.ListJobsPagedResponse

JobsV1Beta3Grpc

Provides a method to create and modify Google Cloud Dataflow jobs. A Job is a multi-stage computation graph run by the Cloud Dataflow service.

JobsV1Beta3Grpc.JobsV1Beta3BlockingStub

Provides a method to create and modify Google Cloud Dataflow jobs. A Job is a multi-stage computation graph run by the Cloud Dataflow service.

JobsV1Beta3Grpc.JobsV1Beta3FutureStub

Provides a method to create and modify Google Cloud Dataflow jobs. A Job is a multi-stage computation graph run by the Cloud Dataflow service.

JobsV1Beta3Grpc.JobsV1Beta3ImplBase

Provides a method to create and modify Google Cloud Dataflow jobs. A Job is a multi-stage computation graph run by the Cloud Dataflow service.

JobsV1Beta3Grpc.JobsV1Beta3Stub

Provides a method to create and modify Google Cloud Dataflow jobs. A Job is a multi-stage computation graph run by the Cloud Dataflow service.

JobsV1Beta3Settings

Settings class to configure an instance of JobsV1Beta3Client.

The default instance has everything set to sensible defaults:

The default service address (dataflow.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of createJob to 30 seconds:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 JobsV1Beta3Settings.Builder jobsV1Beta3SettingsBuilder = JobsV1Beta3Settings.newBuilder();
 jobsV1Beta3SettingsBuilder
     .createJobSettings()
     .setRetrySettings(
         jobsV1Beta3SettingsBuilder.createJobSettings().getRetrySettings().toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 JobsV1Beta3Settings jobsV1Beta3Settings = jobsV1Beta3SettingsBuilder.build();

JobsV1Beta3Settings.Builder

Builder for JobsV1Beta3Settings.

KeyRangeDataDiskAssignment

Data disk assignment information for a specific key-range of a sharded computation. Currently we only support UTF-8 character splits to simplify encoding into JSON.

Protobuf type google.dataflow.v1beta3.KeyRangeDataDiskAssignment

KeyRangeDataDiskAssignment.Builder

Data disk assignment information for a specific key-range of a sharded computation. Currently we only support UTF-8 character splits to simplify encoding into JSON.

Protobuf type google.dataflow.v1beta3.KeyRangeDataDiskAssignment

KeyRangeLocation

Location information for a specific key-range of a sharded computation. Currently we only support UTF-8 character splits to simplify encoding into JSON.

Protobuf type google.dataflow.v1beta3.KeyRangeLocation

KeyRangeLocation.Builder

Location information for a specific key-range of a sharded computation. Currently we only support UTF-8 character splits to simplify encoding into JSON.

Protobuf type google.dataflow.v1beta3.KeyRangeLocation

LaunchFlexTemplateParameter

Launch FlexTemplate Parameter.

Protobuf type google.dataflow.v1beta3.LaunchFlexTemplateParameter

LaunchFlexTemplateParameter.Builder

Launch FlexTemplate Parameter.

Protobuf type google.dataflow.v1beta3.LaunchFlexTemplateParameter

LaunchFlexTemplateRequest

A request to launch a Cloud Dataflow job from a FlexTemplate.

Protobuf type google.dataflow.v1beta3.LaunchFlexTemplateRequest

LaunchFlexTemplateRequest.Builder

A request to launch a Cloud Dataflow job from a FlexTemplate.

Protobuf type google.dataflow.v1beta3.LaunchFlexTemplateRequest

LaunchFlexTemplateResponse

Response to the request to launch a job from Flex Template.

Protobuf type google.dataflow.v1beta3.LaunchFlexTemplateResponse

LaunchFlexTemplateResponse.Builder

Response to the request to launch a job from Flex Template.

Protobuf type google.dataflow.v1beta3.LaunchFlexTemplateResponse

LaunchTemplateParameters

Parameters to provide to the template being launched.

Protobuf type google.dataflow.v1beta3.LaunchTemplateParameters

LaunchTemplateParameters.Builder

Parameters to provide to the template being launched.

Protobuf type google.dataflow.v1beta3.LaunchTemplateParameters

LaunchTemplateRequest

A request to launch a template.

Protobuf type google.dataflow.v1beta3.LaunchTemplateRequest

LaunchTemplateRequest.Builder

A request to launch a template.

Protobuf type google.dataflow.v1beta3.LaunchTemplateRequest

LaunchTemplateResponse

Response to the request to launch a template.

Protobuf type google.dataflow.v1beta3.LaunchTemplateResponse

LaunchTemplateResponse.Builder

Response to the request to launch a template.

Protobuf type google.dataflow.v1beta3.LaunchTemplateResponse

ListJobMessagesRequest

Request to list job messages. Up to max_results messages will be returned in the time range specified starting with the oldest messages first. If no time range is specified the results with start with the oldest message.

Protobuf type google.dataflow.v1beta3.ListJobMessagesRequest

ListJobMessagesRequest.Builder

Protobuf type google.dataflow.v1beta3.ListJobMessagesRequest

ListJobMessagesResponse

Response to a request to list job messages.

Protobuf type google.dataflow.v1beta3.ListJobMessagesResponse

ListJobMessagesResponse.Builder

Response to a request to list job messages.

Protobuf type google.dataflow.v1beta3.ListJobMessagesResponse

ListJobsRequest

Request to list Cloud Dataflow jobs.

Protobuf type google.dataflow.v1beta3.ListJobsRequest

ListJobsRequest.Builder

Request to list Cloud Dataflow jobs.

Protobuf type google.dataflow.v1beta3.ListJobsRequest

ListJobsResponse

Response to a request to list Cloud Dataflow jobs in a project. This might be a partial response, depending on the page size in the ListJobsRequest. However, if the project does not have any jobs, an instance of ListJobsResponse is not returned and the requests's response body is empty {}.

Protobuf type google.dataflow.v1beta3.ListJobsResponse

ListJobsResponse.Builder

Protobuf type google.dataflow.v1beta3.ListJobsResponse

ListSnapshotsRequest

Request to list snapshots.

Protobuf type google.dataflow.v1beta3.ListSnapshotsRequest

ListSnapshotsRequest.Builder

Request to list snapshots.

Protobuf type google.dataflow.v1beta3.ListSnapshotsRequest

ListSnapshotsResponse

List of snapshots.

Protobuf type google.dataflow.v1beta3.ListSnapshotsResponse

ListSnapshotsResponse.Builder

List of snapshots.

Protobuf type google.dataflow.v1beta3.ListSnapshotsResponse

MessagesProto

MessagesV1Beta3Client

Service Description: The Dataflow Messages API is used for monitoring the progress of Dataflow jobs.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (MessagesV1Beta3Client messagesV1Beta3Client = MessagesV1Beta3Client.create()) {
   ListJobMessagesRequest request =
       ListJobMessagesRequest.newBuilder()
           .setProjectId("projectId-894832108")
           .setJobId("jobId101296568")
           .setMinimumImportance(JobMessageImportance.forNumber(0))
           .setPageSize(883849137)
           .setPageToken("pageToken873572522")
           .setStartTime(Timestamp.newBuilder().build())
           .setEndTime(Timestamp.newBuilder().build())
           .setLocation("location1901043637")
           .build();
   for (JobMessage element : messagesV1Beta3Client.listJobMessages(request).iterateAll()) {
     // doThingsWith(element);
   }
 }

Note: close() needs to be called on the MessagesV1Beta3Client object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

This class can be customized by passing in a custom instance of MessagesV1Beta3Settings to create(). For example:

To customize credentials:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 MessagesV1Beta3Settings messagesV1Beta3Settings =
     MessagesV1Beta3Settings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 MessagesV1Beta3Client messagesV1Beta3Client =
     MessagesV1Beta3Client.create(messagesV1Beta3Settings);

To customize the endpoint:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 MessagesV1Beta3Settings messagesV1Beta3Settings =
     MessagesV1Beta3Settings.newBuilder().setEndpoint(myEndpoint).build();
 MessagesV1Beta3Client messagesV1Beta3Client =
     MessagesV1Beta3Client.create(messagesV1Beta3Settings);

To use REST (HTTP1.1/JSON) transport (instead of gRPC) for sending and receiving requests over the wire:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 MessagesV1Beta3Settings messagesV1Beta3Settings =
     MessagesV1Beta3Settings.newHttpJsonBuilder().build();
 MessagesV1Beta3Client messagesV1Beta3Client =
     MessagesV1Beta3Client.create(messagesV1Beta3Settings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

MessagesV1Beta3Client.ListJobMessagesFixedSizeCollection

MessagesV1Beta3Client.ListJobMessagesPage

MessagesV1Beta3Client.ListJobMessagesPagedResponse

MessagesV1Beta3Grpc

The Dataflow Messages API is used for monitoring the progress of Dataflow jobs.

MessagesV1Beta3Grpc.MessagesV1Beta3BlockingStub

The Dataflow Messages API is used for monitoring the progress of Dataflow jobs.

MessagesV1Beta3Grpc.MessagesV1Beta3FutureStub

The Dataflow Messages API is used for monitoring the progress of Dataflow jobs.

MessagesV1Beta3Grpc.MessagesV1Beta3ImplBase

The Dataflow Messages API is used for monitoring the progress of Dataflow jobs.

MessagesV1Beta3Grpc.MessagesV1Beta3Stub

The Dataflow Messages API is used for monitoring the progress of Dataflow jobs.

MessagesV1Beta3Settings

Settings class to configure an instance of MessagesV1Beta3Client.

The default instance has everything set to sensible defaults:

The default service address (dataflow.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of listJobMessages to 30 seconds:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 MessagesV1Beta3Settings.Builder messagesV1Beta3SettingsBuilder =
     MessagesV1Beta3Settings.newBuilder();
 messagesV1Beta3SettingsBuilder
     .listJobMessagesSettings()
     .setRetrySettings(
         messagesV1Beta3SettingsBuilder.listJobMessagesSettings().getRetrySettings().toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 MessagesV1Beta3Settings messagesV1Beta3Settings = messagesV1Beta3SettingsBuilder.build();

MessagesV1Beta3Settings.Builder

Builder for MessagesV1Beta3Settings.

MetricStructuredName

Identifies a metric, by describing the source which generated the metric.

Protobuf type google.dataflow.v1beta3.MetricStructuredName

MetricStructuredName.Builder

Identifies a metric, by describing the source which generated the metric.

Protobuf type google.dataflow.v1beta3.MetricStructuredName

MetricUpdate

Describes the state of a metric.

Protobuf type google.dataflow.v1beta3.MetricUpdate

MetricUpdate.Builder

Describes the state of a metric.

Protobuf type google.dataflow.v1beta3.MetricUpdate

MetricsProto

MetricsV1Beta3Client

Service Description: The Dataflow Metrics API lets you monitor the progress of Dataflow jobs.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (MetricsV1Beta3Client metricsV1Beta3Client = MetricsV1Beta3Client.create()) {
   GetJobMetricsRequest request =
       GetJobMetricsRequest.newBuilder()
           .setProjectId("projectId-894832108")
           .setJobId("jobId101296568")
           .setStartTime(Timestamp.newBuilder().build())
           .setLocation("location1901043637")
           .build();
   JobMetrics response = metricsV1Beta3Client.getJobMetrics(request);
 }

Note: close() needs to be called on the MetricsV1Beta3Client object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

This class can be customized by passing in a custom instance of MetricsV1Beta3Settings to create(). For example:

To customize credentials:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 MetricsV1Beta3Settings metricsV1Beta3Settings =
     MetricsV1Beta3Settings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 MetricsV1Beta3Client metricsV1Beta3Client = MetricsV1Beta3Client.create(metricsV1Beta3Settings);

To customize the endpoint:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 MetricsV1Beta3Settings metricsV1Beta3Settings =
     MetricsV1Beta3Settings.newBuilder().setEndpoint(myEndpoint).build();
 MetricsV1Beta3Client metricsV1Beta3Client = MetricsV1Beta3Client.create(metricsV1Beta3Settings);

To use REST (HTTP1.1/JSON) transport (instead of gRPC) for sending and receiving requests over the wire:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 MetricsV1Beta3Settings metricsV1Beta3Settings =
     MetricsV1Beta3Settings.newHttpJsonBuilder().build();
 MetricsV1Beta3Client metricsV1Beta3Client = MetricsV1Beta3Client.create(metricsV1Beta3Settings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

MetricsV1Beta3Client.GetJobExecutionDetailsFixedSizeCollection

MetricsV1Beta3Client.GetJobExecutionDetailsPage

MetricsV1Beta3Client.GetJobExecutionDetailsPagedResponse

MetricsV1Beta3Client.GetStageExecutionDetailsFixedSizeCollection

MetricsV1Beta3Client.GetStageExecutionDetailsPage

MetricsV1Beta3Client.GetStageExecutionDetailsPagedResponse

MetricsV1Beta3Grpc

The Dataflow Metrics API lets you monitor the progress of Dataflow jobs.

MetricsV1Beta3Grpc.MetricsV1Beta3BlockingStub

The Dataflow Metrics API lets you monitor the progress of Dataflow jobs.

MetricsV1Beta3Grpc.MetricsV1Beta3FutureStub

The Dataflow Metrics API lets you monitor the progress of Dataflow jobs.

MetricsV1Beta3Grpc.MetricsV1Beta3ImplBase

The Dataflow Metrics API lets you monitor the progress of Dataflow jobs.

MetricsV1Beta3Grpc.MetricsV1Beta3Stub

The Dataflow Metrics API lets you monitor the progress of Dataflow jobs.

MetricsV1Beta3Settings

Settings class to configure an instance of MetricsV1Beta3Client.

The default instance has everything set to sensible defaults:

The default service address (dataflow.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of getJobMetrics to 30 seconds:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 MetricsV1Beta3Settings.Builder metricsV1Beta3SettingsBuilder =
     MetricsV1Beta3Settings.newBuilder();
 metricsV1Beta3SettingsBuilder
     .getJobMetricsSettings()
     .setRetrySettings(
         metricsV1Beta3SettingsBuilder.getJobMetricsSettings().getRetrySettings().toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 MetricsV1Beta3Settings metricsV1Beta3Settings = metricsV1Beta3SettingsBuilder.build();

MetricsV1Beta3Settings.Builder

Builder for MetricsV1Beta3Settings.

MountedDataDisk

Describes mounted data disk.

Protobuf type google.dataflow.v1beta3.MountedDataDisk

MountedDataDisk.Builder

Describes mounted data disk.

Protobuf type google.dataflow.v1beta3.MountedDataDisk

Package

The packages that must be installed in order for a worker to run the steps of the Cloud Dataflow job that will be assigned to its worker pool. This is the mechanism by which the Cloud Dataflow SDK causes code to be loaded onto the workers. For example, the Cloud Dataflow Java SDK might use this to install jars containing the user's code and all of the various dependencies (libraries, data files, etc.) required in order for that code to run.

Protobuf type google.dataflow.v1beta3.Package

Package.Builder

Protobuf type google.dataflow.v1beta3.Package

ParameterMetadata

Metadata for a specific parameter.

Protobuf type google.dataflow.v1beta3.ParameterMetadata

ParameterMetadata.Builder

Metadata for a specific parameter.

Protobuf type google.dataflow.v1beta3.ParameterMetadata

PipelineDescription

A descriptive representation of submitted pipeline as well as the executed form. This data is provided by the Dataflow service for ease of visualizing the pipeline and interpreting Dataflow provided metrics.

Protobuf type google.dataflow.v1beta3.PipelineDescription

PipelineDescription.Builder

Protobuf type google.dataflow.v1beta3.PipelineDescription

ProgressTimeseries

Information about the progress of some component of job execution.

Protobuf type google.dataflow.v1beta3.ProgressTimeseries

ProgressTimeseries.Builder

Information about the progress of some component of job execution.

Protobuf type google.dataflow.v1beta3.ProgressTimeseries

ProgressTimeseries.Point

A point in the timeseries.

Protobuf type google.dataflow.v1beta3.ProgressTimeseries.Point

ProgressTimeseries.Point.Builder

A point in the timeseries.

Protobuf type google.dataflow.v1beta3.ProgressTimeseries.Point

PubSubIODetails

Metadata for a Pub/Sub connector used by the job.

Protobuf type google.dataflow.v1beta3.PubSubIODetails

PubSubIODetails.Builder

Metadata for a Pub/Sub connector used by the job.

Protobuf type google.dataflow.v1beta3.PubSubIODetails

PubsubLocation

Identifies a pubsub location to use for transferring data into or out of a streaming Dataflow job.

Protobuf type google.dataflow.v1beta3.PubsubLocation

PubsubLocation.Builder

Identifies a pubsub location to use for transferring data into or out of a streaming Dataflow job.

Protobuf type google.dataflow.v1beta3.PubsubLocation

PubsubSnapshotMetadata

Represents a Pubsub snapshot.

Protobuf type google.dataflow.v1beta3.PubsubSnapshotMetadata

PubsubSnapshotMetadata.Builder

Represents a Pubsub snapshot.

Protobuf type google.dataflow.v1beta3.PubsubSnapshotMetadata

RuntimeEnvironment

The environment values to set at runtime.

Protobuf type google.dataflow.v1beta3.RuntimeEnvironment

RuntimeEnvironment.Builder

The environment values to set at runtime.

Protobuf type google.dataflow.v1beta3.RuntimeEnvironment

RuntimeMetadata

RuntimeMetadata describing a runtime environment.

Protobuf type google.dataflow.v1beta3.RuntimeMetadata

RuntimeMetadata.Builder

RuntimeMetadata describing a runtime environment.

Protobuf type google.dataflow.v1beta3.RuntimeMetadata

SDKInfo

SDK Information.

Protobuf type google.dataflow.v1beta3.SDKInfo

SDKInfo.Builder

SDK Information.

Protobuf type google.dataflow.v1beta3.SDKInfo

SdkHarnessContainerImage

Defines a SDK harness container for executing Dataflow pipelines.

Protobuf type google.dataflow.v1beta3.SdkHarnessContainerImage

SdkHarnessContainerImage.Builder

Defines a SDK harness container for executing Dataflow pipelines.

Protobuf type google.dataflow.v1beta3.SdkHarnessContainerImage

SdkVersion

The version of the SDK used to run the job.

Protobuf type google.dataflow.v1beta3.SdkVersion

SdkVersion.Builder

The version of the SDK used to run the job.

Protobuf type google.dataflow.v1beta3.SdkVersion

Snapshot

Represents a snapshot of a job.

Protobuf type google.dataflow.v1beta3.Snapshot

Snapshot.Builder

Represents a snapshot of a job.

Protobuf type google.dataflow.v1beta3.Snapshot

SnapshotJobRequest

Request to create a snapshot of a job.

Protobuf type google.dataflow.v1beta3.SnapshotJobRequest

SnapshotJobRequest.Builder

Request to create a snapshot of a job.

Protobuf type google.dataflow.v1beta3.SnapshotJobRequest

SnapshotsProto

SnapshotsV1Beta3Client

Service Description: Provides methods to manage snapshots of Google Cloud Dataflow jobs.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (SnapshotsV1Beta3Client snapshotsV1Beta3Client = SnapshotsV1Beta3Client.create()) {
   GetSnapshotRequest request =
       GetSnapshotRequest.newBuilder()
           .setProjectId("projectId-894832108")
           .setSnapshotId("snapshotId-1113817601")
           .setLocation("location1901043637")
           .build();
   Snapshot response = snapshotsV1Beta3Client.getSnapshot(request);
 }

Note: close() needs to be called on the SnapshotsV1Beta3Client object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

This class can be customized by passing in a custom instance of SnapshotsV1Beta3Settings to create(). For example:

To customize credentials:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 SnapshotsV1Beta3Settings snapshotsV1Beta3Settings =
     SnapshotsV1Beta3Settings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 SnapshotsV1Beta3Client snapshotsV1Beta3Client =
     SnapshotsV1Beta3Client.create(snapshotsV1Beta3Settings);

To customize the endpoint:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 SnapshotsV1Beta3Settings snapshotsV1Beta3Settings =
     SnapshotsV1Beta3Settings.newBuilder().setEndpoint(myEndpoint).build();
 SnapshotsV1Beta3Client snapshotsV1Beta3Client =
     SnapshotsV1Beta3Client.create(snapshotsV1Beta3Settings);

To use REST (HTTP1.1/JSON) transport (instead of gRPC) for sending and receiving requests over the wire:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 SnapshotsV1Beta3Settings snapshotsV1Beta3Settings =
     SnapshotsV1Beta3Settings.newHttpJsonBuilder().build();
 SnapshotsV1Beta3Client snapshotsV1Beta3Client =
     SnapshotsV1Beta3Client.create(snapshotsV1Beta3Settings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

SnapshotsV1Beta3Grpc

Provides methods to manage snapshots of Google Cloud Dataflow jobs.

SnapshotsV1Beta3Grpc.SnapshotsV1Beta3BlockingStub

Provides methods to manage snapshots of Google Cloud Dataflow jobs.

SnapshotsV1Beta3Grpc.SnapshotsV1Beta3FutureStub

Provides methods to manage snapshots of Google Cloud Dataflow jobs.

SnapshotsV1Beta3Grpc.SnapshotsV1Beta3ImplBase

Provides methods to manage snapshots of Google Cloud Dataflow jobs.

SnapshotsV1Beta3Grpc.SnapshotsV1Beta3Stub

Provides methods to manage snapshots of Google Cloud Dataflow jobs.

SnapshotsV1Beta3Settings

Settings class to configure an instance of SnapshotsV1Beta3Client.

The default instance has everything set to sensible defaults:

The default service address (dataflow.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of getSnapshot to 30 seconds:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 SnapshotsV1Beta3Settings.Builder snapshotsV1Beta3SettingsBuilder =
     SnapshotsV1Beta3Settings.newBuilder();
 snapshotsV1Beta3SettingsBuilder
     .getSnapshotSettings()
     .setRetrySettings(
         snapshotsV1Beta3SettingsBuilder.getSnapshotSettings().getRetrySettings().toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 SnapshotsV1Beta3Settings snapshotsV1Beta3Settings = snapshotsV1Beta3SettingsBuilder.build();

SnapshotsV1Beta3Settings.Builder

Builder for SnapshotsV1Beta3Settings.

SpannerIODetails

Metadata for a Spanner connector used by the job.

Protobuf type google.dataflow.v1beta3.SpannerIODetails

SpannerIODetails.Builder

Metadata for a Spanner connector used by the job.

Protobuf type google.dataflow.v1beta3.SpannerIODetails

StageExecutionDetails

Information about the workers and work items within a stage.

Protobuf type google.dataflow.v1beta3.StageExecutionDetails

StageExecutionDetails.Builder

Information about the workers and work items within a stage.

Protobuf type google.dataflow.v1beta3.StageExecutionDetails

StageSummary

Information about a particular execution stage of a job.

Protobuf type google.dataflow.v1beta3.StageSummary

StageSummary.Builder

Information about a particular execution stage of a job.

Protobuf type google.dataflow.v1beta3.StageSummary

StateFamilyConfig

State family configuration.

Protobuf type google.dataflow.v1beta3.StateFamilyConfig

StateFamilyConfig.Builder

State family configuration.

Protobuf type google.dataflow.v1beta3.StateFamilyConfig

Step

Defines a particular step within a Cloud Dataflow job. A job consists of multiple steps, each of which performs some specific operation as part of the overall job. Data is typically passed from one step to another as part of the job. Here's an example of a sequence of steps which together implement a Map-Reduce job:

Read a collection of data from some source, parsing the collection's elements.
Validate the elements.
Apply a user-defined function to map each element to some value and extract an element-specific key value.
Group elements with the same key into a single element with that key, transforming a multiply-keyed collection into a uniquely-keyed collection.
Write the elements out to some data sink. Note that the Cloud Dataflow service may be used to run many different types of jobs, not just Map-Reduce.

Protobuf type google.dataflow.v1beta3.Step

Step.Builder

Read a collection of data from some source, parsing the collection's elements.
Validate the elements.
Apply a user-defined function to map each element to some value and extract an element-specific key value.
Group elements with the same key into a single element with that key, transforming a multiply-keyed collection into a uniquely-keyed collection.
Write the elements out to some data sink. Note that the Cloud Dataflow service may be used to run many different types of jobs, not just Map-Reduce.

Protobuf type google.dataflow.v1beta3.Step

StreamLocation

Describes a stream of data, either as input to be processed or as output of a streaming Dataflow job.

Protobuf type google.dataflow.v1beta3.StreamLocation

StreamLocation.Builder

Describes a stream of data, either as input to be processed or as output of a streaming Dataflow job.

Protobuf type google.dataflow.v1beta3.StreamLocation

StreamingApplianceSnapshotConfig

Streaming appliance snapshot configuration.

Protobuf type google.dataflow.v1beta3.StreamingApplianceSnapshotConfig

StreamingApplianceSnapshotConfig.Builder

Streaming appliance snapshot configuration.

Protobuf type google.dataflow.v1beta3.StreamingApplianceSnapshotConfig

StreamingComputationRanges

Describes full or partial data disk assignment information of the computation ranges.

Protobuf type google.dataflow.v1beta3.StreamingComputationRanges

StreamingComputationRanges.Builder

Describes full or partial data disk assignment information of the computation ranges.

Protobuf type google.dataflow.v1beta3.StreamingComputationRanges

StreamingProto

StreamingSideInputLocation

Identifies the location of a streaming side input.

Protobuf type google.dataflow.v1beta3.StreamingSideInputLocation

StreamingSideInputLocation.Builder

Identifies the location of a streaming side input.

Protobuf type google.dataflow.v1beta3.StreamingSideInputLocation

StreamingStageLocation

Identifies the location of a streaming computation stage, for stage-to-stage communication.

Protobuf type google.dataflow.v1beta3.StreamingStageLocation

StreamingStageLocation.Builder

Identifies the location of a streaming computation stage, for stage-to-stage communication.

Protobuf type google.dataflow.v1beta3.StreamingStageLocation

StructuredMessage

A rich message format, including a human readable string, a key for identifying the message, and structured data associated with the message for programmatic consumption.

Protobuf type google.dataflow.v1beta3.StructuredMessage

StructuredMessage.Builder

A rich message format, including a human readable string, a key for identifying the message, and structured data associated with the message for programmatic consumption.

Protobuf type google.dataflow.v1beta3.StructuredMessage

StructuredMessage.Parameter

Structured data associated with this message.

Protobuf type google.dataflow.v1beta3.StructuredMessage.Parameter

StructuredMessage.Parameter.Builder

Structured data associated with this message.

Protobuf type google.dataflow.v1beta3.StructuredMessage.Parameter

TaskRunnerSettings

Taskrunner configuration settings.

Protobuf type google.dataflow.v1beta3.TaskRunnerSettings

TaskRunnerSettings.Builder

Taskrunner configuration settings.

Protobuf type google.dataflow.v1beta3.TaskRunnerSettings

TemplateMetadata

Metadata describing a template.

Protobuf type google.dataflow.v1beta3.TemplateMetadata

TemplateMetadata.Builder

Metadata describing a template.

Protobuf type google.dataflow.v1beta3.TemplateMetadata

TemplatesProto

TemplatesServiceClient

Service Description: Provides a method to create Cloud Dataflow jobs from templates.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (TemplatesServiceClient templatesServiceClient = TemplatesServiceClient.create()) {
   CreateJobFromTemplateRequest request =
       CreateJobFromTemplateRequest.newBuilder()
           .setProjectId("projectId-894832108")
           .setJobName("jobName-1438096408")
           .putAllParameters(new HashMap<String, String>())
           .setEnvironment(RuntimeEnvironment.newBuilder().build())
           .setLocation("location1901043637")
           .build();
   Job response = templatesServiceClient.createJobFromTemplate(request);
 }

Note: close() needs to be called on the TemplatesServiceClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

This class can be customized by passing in a custom instance of TemplatesServiceSettings to create(). For example:

To customize credentials:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 TemplatesServiceSettings templatesServiceSettings =
     TemplatesServiceSettings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 TemplatesServiceClient templatesServiceClient =
     TemplatesServiceClient.create(templatesServiceSettings);

To customize the endpoint:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 TemplatesServiceSettings templatesServiceSettings =
     TemplatesServiceSettings.newBuilder().setEndpoint(myEndpoint).build();
 TemplatesServiceClient templatesServiceClient =
     TemplatesServiceClient.create(templatesServiceSettings);

To use REST (HTTP1.1/JSON) transport (instead of gRPC) for sending and receiving requests over the wire:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 TemplatesServiceSettings templatesServiceSettings =
     TemplatesServiceSettings.newHttpJsonBuilder().build();
 TemplatesServiceClient templatesServiceClient =
     TemplatesServiceClient.create(templatesServiceSettings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

TemplatesServiceGrpc

Provides a method to create Cloud Dataflow jobs from templates.

TemplatesServiceGrpc.TemplatesServiceBlockingStub

Provides a method to create Cloud Dataflow jobs from templates.

TemplatesServiceGrpc.TemplatesServiceFutureStub

Provides a method to create Cloud Dataflow jobs from templates.

TemplatesServiceGrpc.TemplatesServiceImplBase

Provides a method to create Cloud Dataflow jobs from templates.

TemplatesServiceGrpc.TemplatesServiceStub

Provides a method to create Cloud Dataflow jobs from templates.

TemplatesServiceSettings

Settings class to configure an instance of TemplatesServiceClient.

The default instance has everything set to sensible defaults:

The default service address (dataflow.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of createJobFromTemplate to 30 seconds:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 TemplatesServiceSettings.Builder templatesServiceSettingsBuilder =
     TemplatesServiceSettings.newBuilder();
 templatesServiceSettingsBuilder
     .createJobFromTemplateSettings()
     .setRetrySettings(
         templatesServiceSettingsBuilder
             .createJobFromTemplateSettings()
             .getRetrySettings()
             .toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 TemplatesServiceSettings templatesServiceSettings = templatesServiceSettingsBuilder.build();

TemplatesServiceSettings.Builder

Builder for TemplatesServiceSettings.

TopologyConfig

Global topology of the streaming Dataflow job, including all computations and their sharded locations.

Protobuf type google.dataflow.v1beta3.TopologyConfig

TopologyConfig.Builder

Global topology of the streaming Dataflow job, including all computations and their sharded locations.

Protobuf type google.dataflow.v1beta3.TopologyConfig

TransformSummary

Description of the type, names/ids, and input/outputs for a transform.

Protobuf type google.dataflow.v1beta3.TransformSummary

TransformSummary.Builder

Description of the type, names/ids, and input/outputs for a transform.

Protobuf type google.dataflow.v1beta3.TransformSummary

UpdateJobRequest

Request to update a Cloud Dataflow job.

Protobuf type google.dataflow.v1beta3.UpdateJobRequest

UpdateJobRequest.Builder

Request to update a Cloud Dataflow job.

Protobuf type google.dataflow.v1beta3.UpdateJobRequest

WorkItemDetails

Information about an individual work item execution.

Protobuf type google.dataflow.v1beta3.WorkItemDetails

WorkItemDetails.Builder

Information about an individual work item execution.

Protobuf type google.dataflow.v1beta3.WorkItemDetails

WorkerDetails

Information about a worker

Protobuf type google.dataflow.v1beta3.WorkerDetails

WorkerDetails.Builder

Information about a worker

Protobuf type google.dataflow.v1beta3.WorkerDetails

WorkerPool

Describes one particular pool of Cloud Dataflow workers to be instantiated by the Cloud Dataflow service in order to perform the computations required by a job. Note that a workflow job may use multiple pools, in order to match the various computational requirements of the various stages of the job.

Protobuf type google.dataflow.v1beta3.WorkerPool

WorkerPool.Builder

Protobuf type google.dataflow.v1beta3.WorkerPool

WorkerSettings

Provides data to pass through to the worker harness.

Protobuf type google.dataflow.v1beta3.WorkerSettings

WorkerSettings.Builder

Provides data to pass through to the worker harness.

Protobuf type google.dataflow.v1beta3.WorkerSettings