Cloud Dataplex v1 API - Class Task.Types.SparkTaskConfig (3.3.0) 

public sealed class Task.Types.SparkTaskConfig : IMessage<Task.Types.SparkTaskConfig>, IEquatable<Task.Types.SparkTaskConfig>, IDeepCloneable<Task.Types.SparkTaskConfig>, IBufferMessage, IMessage

Reference documentation and code samples for the Cloud Dataplex v1 API class Task.Types.SparkTaskConfig.

User-specified config for running a Spark task.

Inheritance

object > Task.Types.SparkTaskConfig

Namespace

Google.Cloud.Dataplex.V1

Assembly

Google.Cloud.Dataplex.V1.dll

Constructors

SparkTaskConfig()

public SparkTaskConfig()

SparkTaskConfig(SparkTaskConfig)

public SparkTaskConfig(Task.Types.SparkTaskConfig other)
Parameter
Name Description
other TaskTypesSparkTaskConfig

Properties

ArchiveUris

public RepeatedField<string> ArchiveUris { get; }

Optional. Cloud Storage URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.

Property Value
Type Description
RepeatedFieldstring

DriverCase

public Task.Types.SparkTaskConfig.DriverOneofCase DriverCase { get; }
Property Value
Type Description
TaskTypesSparkTaskConfigDriverOneofCase

FileUris

public RepeatedField<string> FileUris { get; }

Optional. Cloud Storage URIs of files to be placed in the working directory of each executor.

Property Value
Type Description
RepeatedFieldstring

HasMainClass

public bool HasMainClass { get; }

Gets whether the "main_class" field is set

Property Value
Type Description
bool

HasMainJarFileUri

public bool HasMainJarFileUri { get; }

Gets whether the "main_jar_file_uri" field is set

Property Value
Type Description
bool

HasPythonScriptFile

public bool HasPythonScriptFile { get; }

Gets whether the "python_script_file" field is set

Property Value
Type Description
bool

HasSqlScript

public bool HasSqlScript { get; }

Gets whether the "sql_script" field is set

Property Value
Type Description
bool

HasSqlScriptFile

public bool HasSqlScriptFile { get; }

Gets whether the "sql_script_file" field is set

Property Value
Type Description
bool

InfrastructureSpec

public Task.Types.InfrastructureSpec InfrastructureSpec { get; set; }

Optional. Infrastructure specification for the execution.

Property Value
Type Description
TaskTypesInfrastructureSpec

MainClass

public string MainClass { get; set; }

The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in jar_file_uris. The execution args are passed in as a sequence of named process arguments (--key=value).

Property Value
Type Description
string

MainJarFileUri

public string MainJarFileUri { get; set; }

The Cloud Storage URI of the jar file that contains the main class. The execution args are passed in as a sequence of named process arguments (--key=value).

Property Value
Type Description
string

PythonScriptFile

public string PythonScriptFile { get; set; }

The Gcloud Storage URI of the main Python file to use as the driver. Must be a .py file. The execution args are passed in as a sequence of named process arguments (--key=value).

Property Value
Type Description
string

SqlScript

public string SqlScript { get; set; }

The query text. The execution args are used to declare a set of script variables (set key="value";).

Property Value
Type Description
string

SqlScriptFile

public string SqlScriptFile { get; set; }

A reference to a query file. This can be the Cloud Storage URI of the query file or it can the path to a SqlScript Content. The execution args are used to declare a set of script variables (set key="value";).

Property Value
Type Description
string