Query and view open alerts

This page provides detailed instructions on how to query and visualize open alerts using both the GDC console and the curl tool for the Cortex endpoint to gain issue awareness and resolve problems.

After creating alert rules based on logs and metrics from Google Distributed Cloud (GDC) air-gapped environments, you can start monitoring open alerts from your project. You can visualize and filter alerts that your system events trigger on the GDC console or access them directly from Cortex using the curl tool for flexible scripting and automation.

You can access open alerts in one of the following two methods:

  • GDC console: Visualize alerting data in integrated panels containing information like the number of alerts for a specific data source, the severity level, duration, status, message, and labels. The GDC console provides a user-friendly interface for filtering and analyzing alerts from your system components.
  • Cortex Alertmanager endpoint: For more advanced use cases, query your project's Cortex instance directly using the curl tool on a command line. Cortex stores your project's Alertmanager alerts and provides an HTTP endpoint for programmatic access. This access lets you export data, automate tasks, configure cron jobs, and build custom integrations.

Before you begin

To get the permissions that you need to query and visualize alerts, ask your Project IAM Admin to grant you one of the associated Project Cortex Alertmanager roles in your project namespace. Depending on the level of access and permissions you need, you might obtain editor or viewer roles for this resource in a project.

For more information about these roles, see Prepare IAM permissions.

View and filter open alerts

Select one of the following methods to query and filter open alerts from your project namespace:

Console

View the open alerts in a project from the GDC console:

  1. Sign in to the GDC console.
  2. In the GDC console, select your project.
  3. In the navigation menu, select Operations > Alerting.
  4. Select the Alerts tab.
  5. View the list of alerts.
  6. On the Alerts opened section, click Filter to only display open alerts. You can also filter alerts by other property names or values.
  7. Click an alert name to view the alert details.

Cortex endpoint

This section describes how to access alerts using your Cortex Alertmanager endpoint.

Identify your Cortex endpoint

The following URL is the endpoint of the Cortex instance of your project:

  https://GDC_URL/PROJECT_NAMESPACE/cortex/alertmanager/

Replace the following:

  • GDC_URL: the URL of your organization in GDC.

  • PROJECT_NAMESPACE: your project namespace.

    For example, the Cortex endpoint for the platform-obs project in the org-1 organization is https://org-1/platform-obs/cortex/alertmanager/.

Authenticate the curl request

  1. Download and install the gdcloud CLI.

  2. Set the gdcloud core/organization_console_url property:

    gdcloud config set core/organization_console_url
https://GDC_URL

  3. Sign in with the configured identity provider:

    gdcloud auth login

  4. Use your username and password to authenticate and sign in.

    When the login is successful, you can use the authorization header in your cURL request through the gdcloud auth print-identity-token command. For more information, see gdcloud auth.

Call the Cortex endpoint

Complete the following steps to reach the Cortex endpoint using the curl tool:

  1. Authenticate the curl request.

  2. Use curl to call the Cortex endpoint and extend the URL using the standard Alertmanager API specification (https://prometheus.io/docs/prometheus/latest/querying/api/#alertmanagers) to query alerts.

    The following is an example of a curl request:

      curl https://GDC_URL/PROJECT_NAME/cortex/alertmanager/api/v1/alertmanagers \
  -H "Authorization: Bearer $(gdcloud auth print-identity-token \
  --audiences=https://GDC_URL)"

    You obtain the output following the command. The API response is in JSON format.