REST Resource: projects.locations.metadataJobs

Resource: MetadataJob

A metadata job resource.

JSON representation 
{
  "name": string,
  "uid": string,
  "createTime": string,
  "updateTime": string,
  "labels": {
    string: string,
    ...
  },
  "type": enum (Type),
  "status": {
    object (Status)
  },

  // Union field spec can be only one of the following:
  "importSpec": {
    object (ImportJobSpec)
  }
  // End of list of possible types for union field spec.

  // Union field result can be only one of the following:
  "importResult": {
    object (ImportJobResult)
  }
  // End of list of possible types for union field result.
}
Fields
name

string

Output only. Identifier. The name of the resource that the configuration is applied to, in the format projects/{project_number}/locations/{locationId}/metadataJobs/{metadataJobId}.
uid

string

Output only. A system-generated, globally unique ID for the metadata job. If the metadata job is deleted and then re-created with the same name, this ID is different.
createTime

string (Timestamp format)

Output only. The time when the metadata job was created.

Uses RFC 3339, where generated output will always be Z-normalized and uses 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
updateTime

string (Timestamp format)

Output only. The time when the metadata job was updated.

Uses RFC 3339, where generated output will always be Z-normalized and uses 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
labels

map (key: string, value: string)

Optional. User-defined labels.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
type

enum (Type)

Required. Metadata job type.
status

object (Status)

Output only. Metadata job status.

Union field spec.

spec can be only one of the following:
importSpec

object (ImportJobSpec)

Import job specification.

Union field result.

result can be only one of the following:
importResult

object (ImportJobResult)

Output only. Import job result.

Type

Metadata job type.

Enums
TYPE_UNSPECIFIED Unspecified.
IMPORT Import job.

ImportJobSpec

Job specification for a metadata import job.

You can run the following kinds of metadata import jobs:

  • Full sync of entries with incremental import of their aspects. Supported for custom entries.
  • Incremental import of aspects only. Supported for aspects that belong to custom entries and system entries. For custom entries, you can modify both optional aspects and required aspects. For system entries, you can modify optional aspects.
JSON representation 
{
  "sourceStorageUri": string,
  "sourceCreateTime": string,
  "scope": {
    object (ImportJobScope)
  },
  "entrySyncMode": enum (SyncMode),
  "aspectSyncMode": enum (SyncMode),
  "logLevel": enum (LogLevel)
}
Fields
sourceStorageUri

string

Optional. The URI of a Cloud Storage bucket or folder (beginning with gs:// and ending with /) that contains the metadata import files for this job.

A metadata import file defines the values to set for each of the entries and aspects in a metadata job. For more information about how to create a metadata import file and the file requirements, see Metadata import file.

You can provide multiple metadata import files in the same metadata job. The bucket or folder must contain at least one metadata import file, in JSON Lines format (either .json or .jsonl file extension).

In FULL entry sync mode, don't save the metadata import file in a folder named SOURCE_STORAGE_URI/deletions/.

Caution: If the metadata import file contains no data, all entries and aspects that belong to the job's scope are deleted.
sourceCreateTime

string (Timestamp format)

Optional. The time when the process that created the metadata import files began.

Uses RFC 3339, where generated output will always be Z-normalized and uses 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
scope

object (ImportJobScope)

Required. A boundary on the scope of impact that the metadata import job can have.
entrySyncMode

enum (SyncMode)

Required. The sync mode for entries.
aspectSyncMode

enum (SyncMode)

Required. The sync mode for aspects.
logLevel

enum (LogLevel)

Optional. The level of logs to write to Cloud Logging for this job.

Debug-level logs provide highly-detailed information for troubleshooting, but their increased verbosity could incur additional costs that might not be merited for all jobs.

If unspecified, defaults to INFO.

ImportJobScope

A boundary on the scope of impact that the metadata import job can have.

JSON representation 
{
  "entryGroups": [
    string
  ],
  "entryTypes": [
    string
  ],
  "aspectTypes": [
    string
  ]
}
Fields
entryGroups[]

string

Required. The entry group that is in scope for the import job, specified as a relative resource name in the format projects/{project_number_or_id}/locations/{locationId}/entryGroups/{entryGroupId}. Only entries and aspects that belong to the specified entry group are affected by the job.

Must contain exactly one element. The entry group and the job must be in the same location.
entryTypes[]

string

Required. The entry types that are in scope for the import job, specified as relative resource names in the format projects/{project_number_or_id}/locations/{locationId}/entryTypes/{entryTypeId}. The job modifies only the entries and aspects that belong to these entry types.

If the metadata import file attempts to modify an entry whose type isn't included in this list, the import job is halted before modifying any entries or aspects.

The location of an entry type must either match the location of the job, or the entry type must be global.
aspectTypes[]

string

Optional. The aspect types that are in scope for the import job, specified as relative resource names in the format projects/{project_number_or_id}/locations/{locationId}/aspectTypes/{aspectTypeId}. The job modifies only the aspects that belong to these aspect types.

This field is required when creating an aspect-only import job.

If the metadata import file attempts to modify an aspect whose type isn't included in this list, the import job is halted before modifying any entries or aspects.

The location of an aspect type must either match the location of the job, or the aspect type must be global.

SyncMode

Specifies how the entries and aspects in a metadata job are updated. For more information, see Sync mode.

Enums
SYNC_MODE_UNSPECIFIED Sync mode unspecified.
FULL

All resources in the job's scope are modified. If a resource exists in Dataplex but isn't included in the metadata import file, the resource is deleted when you run the metadata job. Use this mode to perform a full sync of the set of entries in the job scope.

This sync mode is supported for entries.
INCREMENTAL

Only the resources that are explicitly included in the metadata import file are modified. Use this mode to modify a subset of resources while leaving unreferenced resources unchanged.

This sync mode is supported for aspects.
NONE

If entry sync mode is NONE, then aspects are modified according to the aspect sync mode. Other metadata that belongs to entries in the job's scope isn't modified.

This sync mode is supported for entries.

LogLevel

The level of logs to write to Cloud Logging for this job.

Enums
LOG_LEVEL_UNSPECIFIED Log level unspecified.
DEBUG

Debug-level logging. Captures detailed logs for each import item. Use debug-level logging to troubleshoot issues with specific import items. For example, use debug-level logging to identify resources that are missing from the job scope, entries or aspects that don't conform to the associated entry type or aspect type, or other misconfigurations with the metadata import file.

Depending on the size of your metadata job and the number of logs that are generated, debug-level logging might incur additional costs.
INFO Info-level logging. Captures logs at the overall job level. Includes aggregate logs about import items, but doesn't specify which import item has an error.

ImportJobResult

Results from a metadata import job.

JSON representation 
{
  "deletedEntries": string,
  "updatedEntries": string,
  "createdEntries": string,
  "unchangedEntries": string,
  "recreatedEntries": string,
  "updateTime": string
}
Fields
deletedEntries

string (int64 format)

Output only. The total number of entries that were deleted.
updatedEntries

string (int64 format)

Output only. The total number of entries that were updated.
createdEntries

string (int64 format)

Output only. The total number of entries that were created.
unchangedEntries

string (int64 format)

Output only. The total number of entries that were unchanged.
recreatedEntries

string (int64 format)

Output only. The total number of entries that were recreated.
updateTime

string (Timestamp format)

Output only. The time when the status was updated.

Uses RFC 3339, where generated output will always be Z-normalized and uses 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".

Status

Metadata job status.

JSON representation 
{
  "state": enum (State),
  "message": string,
  "completionPercent": integer,
  "updateTime": string
}
Fields
state

enum (State)

Output only. State of the metadata job.
message

string

Output only. Message relating to the progression of a metadata job.
completionPercent

integer

Output only. Progress tracking.
updateTime

string (Timestamp format)

Output only. The time when the status was updated.

Uses RFC 3339, where generated output will always be Z-normalized and uses 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".

State

State of a metadata job.

Enums
STATE_UNSPECIFIED State unspecified.
QUEUED The job is queued.
RUNNING The job is running.
CANCELING The job is being canceled.
CANCELED The job is canceled.
SUCCEEDED The job succeeded.
FAILED The job failed.
SUCCEEDED_WITH_ERRORS The job completed with some errors.

Methods

cancel

 Cancels a metadata job.

create

 Creates a metadata job.

get

 Gets a metadata job.

list

 Lists metadata jobs.