REST Resource: projects.datasets.evaluations

Resource: Evaluation

Describes an evaluation between a machine learning model's predictions and ground truth labels. Created when an EvaluationJob runs successfully.

JSON representation 
{
  "name": string,
  "config": {
    object (EvaluationConfig)
  },
  "evaluationJobRunTime": string,
  "createTime": string,
  "evaluationMetrics": {
    object (EvaluationMetrics)
  },
  "annotationType": enum (AnnotationType),
  "evaluatedItemCount": string
}
Fields
name

string

Output only. Resource name of an evaluation. The name has the following format:

"projects/{project_id}/datasets/{dataset_id}/evaluations/{evaluation_id}'
config

object (EvaluationConfig)

Output only. Options used in the evaluation job that created this evaluation.

evaluationJobRunTime

string (Timestamp format)

Output only. Timestamp for when the evaluation job that created this evaluation ran.

A timestamp in RFC3339 UTC "Zulu" format, accurate to nanoseconds. Example: "2014-10-02T15:01:23.045123456Z".
createTime

string (Timestamp format)

Output only. Timestamp for when this evaluation was created.

A timestamp in RFC3339 UTC "Zulu" format, accurate to nanoseconds. Example: "2014-10-02T15:01:23.045123456Z".
evaluationMetrics

object (EvaluationMetrics)

Output only. Metrics comparing predictions to ground truth labels.

annotationType

enum (AnnotationType)

Output only. Type of task that the model version being evaluated performs, as defined in the

evaluationJobConfig.inputConfig.annotationType field of the evaluation job that created this evaluation.
evaluatedItemCount

string (int64 format)

Output only. The number of items in the ground truth dataset that were used for this evaluation. Only populated when the evaulation is for certain AnnotationTypes.

EvaluationMetrics

JSON representation 
{

  // Union field metrics can be only one of the following:
  "classificationMetrics": {
    object (ClassificationMetrics)
  },
  "objectDetectionMetrics": {
    object (ObjectDetectionMetrics)
  }
  // End of list of possible types for union field metrics.
}
Fields
Union field metrics. Common metrics covering most general cases. metrics can be only one of the following:
classificationMetrics

object (ClassificationMetrics)
objectDetectionMetrics

object (ObjectDetectionMetrics)

ClassificationMetrics

Metrics calculated for a classification model.

JSON representation 
{
  "prCurve": {
    object (PrCurve)
  },
  "confusionMatrix": {
    object (ConfusionMatrix)
  }
}
Fields
prCurve

object (PrCurve)

Precision-recall curve based on ground truth labels, predicted labels, and scores for the predicted labels.

confusionMatrix

object (ConfusionMatrix)

Confusion matrix of predicted labels vs. ground truth labels.

PrCurve

JSON representation 
{
  "annotationSpec": {
    object (AnnotationSpec)
  },
  "areaUnderCurve": number,
  "confidenceMetricsEntries": [
    {
      object (ConfidenceMetricsEntry)
    }
  ],
  "meanAveragePrecision": number
}
Fields
annotationSpec

object (AnnotationSpec)

The annotation spec of the label for which the precision-recall curve calculated. If this field is empty, that means the precision-recall curve is an aggregate curve for all labels.

areaUnderCurve

number

Area under the precision-recall curve. Not to be confused with area under a receiver operating characteristic (ROC) curve.

confidenceMetricsEntries[]

object (ConfidenceMetricsEntry)

Entries that make up the precision-recall graph. Each entry is a "point" on the graph drawn for a different confidenceThreshold.

meanAveragePrecision

number

Mean average prcision of this curve.

ConfidenceMetricsEntry

JSON representation 
{
  "confidenceThreshold": number,
  "recall": number,
  "precision": number,
  "f1Score": number,
  "recallAt1": number,
  "precisionAt1": number,
  "f1ScoreAt1": number,
  "recallAt5": number,
  "precisionAt5": number,
  "f1ScoreAt5": number
}
Fields
confidenceThreshold

number

Threshold used for this entry.

For classification tasks, this is a classification threshold: a predicted label is categorized as positive or negative (in the context of this point on the PR curve) based on whether the label's score meets this threshold.

For image object detection (bounding box) tasks, this is the [intersection-over-union

(IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) threshold for the context of this point on the PR curve.
recall

number

Recall value.

precision

number

Precision value.

f1Score

number

Harmonic mean of recall and precision.

recallAt1

number

Recall value for entries with label that has highest score.

precisionAt1

number

Precision value for entries with label that has highest score.

f1ScoreAt1

number

The harmonic mean of recallAt1 and precisionAt1.

recallAt5

number

Recall value for entries with label that has highest 5 scores.

precisionAt5

number

Precision value for entries with label that has highest 5 scores.

f1ScoreAt5

number

The harmonic mean of recallAt5 and precisionAt5.

ConfusionMatrix

Confusion matrix of the model running the classification. Only applicable when the metrics entry aggregates multiple labels. Not applicable when the entry is for a single label.

JSON representation 
{
  "row": [
    {
      object (Row)
    }
  ]
}
Fields
row[]

object (Row)

Row

A row in the confusion matrix. Each entry in this row has the same ground truth label.

JSON representation 
{
  "annotationSpec": {
    object (AnnotationSpec)
  },
  "entries": [
    {
      object (ConfusionMatrixEntry)
    }
  ]
}
Fields
annotationSpec

object (AnnotationSpec)

The annotation spec of the ground truth label for this row.

entries[]

object (ConfusionMatrixEntry)

A list of the confusion matrix entries. One entry for each possible predicted label.

ConfusionMatrixEntry

JSON representation 
{
  "annotationSpec": {
    object (AnnotationSpec)
  },
  "itemCount": number
}
Fields
annotationSpec

object (AnnotationSpec)

The annotation spec of a predicted label.

itemCount

number

Number of items predicted to have this label. (The ground truth label for these items is the Row.annotationSpec of this entry's parent.)

ObjectDetectionMetrics

Metrics calculated for an image object detection (bounding box) model.

JSON representation 
{
  "prCurve": {
    object (PrCurve)
  }
}
Fields
prCurve

object (PrCurve)

Precision-recall curve.

Methods

get

 Gets an evaluation by resource name (to search, use projects.evaluations.search).