Create new metrics from existing metrics

This page describes how to generate new metrics from existing metrics by deploying a MonitoringRule custom resource in Google Distributed Cloud (GDC) air-gapped.

Recording rules allow you to precompute frequently used or computationally expensive expressions, improving performance for dashboards and alerts. Defined within MonitoringRule custom resources, these rules generate new time series metrics from existing metrics, enhancing data observability.

By storing precomputed results, recording rules eliminate redundant calculations each time an expression is needed. This method significantly speeds up query times, particularly for complex dashboards and alerts that require repeated evaluations of the same expression. Essentially, MonitoringRule resources let you derive new metrics from existing ones by specifying the necessary calculations within their recording rules.

Before you begin

To get the permissions that you need to manage MonitoringRule custom resources, ask your Organization IAM Admin or Project IAM Admin to grant you one of the associated MonitoringRule roles.

Depending on the level of access and permissions you need, you might obtain creator, editor, or viewer roles for this resource in an organization or a project. For more information, see Prepare IAM permissions.

Create recording rules

To create a recording rule, define a name for the recorded metric and a valid expression that produces a numeric value. This expression, when evaluated, generates the new metric. Deploy the MonitoringRule custom resource in your project namespace on the Management API server to activate the recording rule.

For more information about recording rules, see https://grafana.com/docs/loki/latest/rules/.

Important: Before creating recording rules, ensure you have a deployed project with active workloads and configured metric scraping. In this way, the rules have data to operate on. For more information, see Collect metrics.

Follow these steps to create recording rules in your project namespace:

1. Determine the GDC project from which you want to create recording rules.

2. Create a YAML file defining the MonitoringRule custom resource.

   The complete MonitoringRule specification shows an example for metric-based recording rules. For more information, see the API reference documentation.

3. Replace the following values in the YAML file according to your needs:

   Field name	Description
   namespace	The project namespace.
   name	The name for the rule configuration.
   interval	The duration of the rule evaluation interval in seconds.
   limit	Optional. The maximum number of alerts. Set to 0 for unlimited alerts.
   recordRules	The definitions for calculating new metrics.
   recordRules.record	The record name for the new metric. The value must be a valid metric name that defines the time series where the results are stored.
   recordRules.expr	A PromQL expression for the metric rule, which must evaluate to a numeric value.
   recordRules.labels	Optional. The key-value pairs of labels to add to or overwrite the new metric.

4. Save the YAML file.

5. Apply the MonitoringRule configuration to the Management API server within the same namespace as your recording rules:

   kubectl --kubeconfig KUBECONFIG_PATH apply -f MONITORING_RULE_NAME.yaml
   

   Replace the following:

   • KUBECONFIG_PATH: the path to the kubeconfig file for the Management API server.
   • MONITORING_RULE_NAME: the name of the MonitoringRule definition file.

Complete MonitoringRule specification

A MonitoringRule custom resource contains recording rules that describe the conditions to create new metrics based on existing metrics for observability.

The following YAML file shows a template for the MonitoringRule custom resource. For more information, see the API reference documentation.

# Configures either an alert or a target record for precomputation.
apiVersion: monitoring.gdc.goog/v1
kind: MonitoringRule
metadata:
  # Choose a namespace that matches the project namespace.
  # The alert or record is produced in the same namespace.
  namespace: PROJECT_NAMESPACE
  name: MONITORING_RULE_NAME
spec:
  # Rule evaluation interval.
  interval: 60s

  # Configure the limit for the number of alerts.
  # A value of '0' means no limit.
  # Optional.
  # Default value: '0'
  limit: 0

  # Configure recording rules to generate new metrics based on existing metrics.
  # Recording rules precompute expressions that are frequently needed or computationally expensive.
  # Results are saved as a new set of time series.
  recordRules:
    # Define the time series where you want to write the recording rule.
    # The value must be a valid metric name.
  - record: MyMetricsName

    # Define the PromQL expression to evaluate for this rule.
    expr: rate({service_name="bob-service"} [1m])

    # Define labels to add or overwrite.
    # Map of key-value pairs.
    # Optional.
    labels:
      verb: read

Replace the following:

• PROJECT_NAMESPACE: your project namespace.
• MONITORING_RULE_NAME: the name of the MonitoringRule definition file.