Configure notification channels for alerts

This page outlines the process for configuring notification channels to receive alerts.

Notification channels are the delivery mechanisms for your configured alerts. The system notifies designated recipients through these channels when an event triggers an alert. This process ensures that critical alerts reach the appropriate personnel through their preferred communication methods.

In Google Distributed Cloud (GDC) air-gapped environments, predefined notification channels like Slack or email are unavailable. To receive alerts, you must configure at least one custom notification channel. You edit ObservabilityPipeline custom resources in your project namespace to apply channel configurations.

By customizing notification channels, administrators can achieve the following:

Target specific recipients: Direct alerts to individuals, teams, or on-call rotations responsible for addressing issues.
Use preferred communication methods: Deliver alerts through channels like SMS, PagerDuty, webhooks, or custom integrations, catering to individual preferences and operational workflows.
Prevent alert fatigue: Reduce noise and ensure that alerts are received by those who need to take action.

Implementing notification channels in GDC requires administrators to define the channel configuration within the system. This process typically involves specifying parameters like:

Channel type: the type of channel being used.
Destination: the endpoint where the system should send notifications.
Authentication: any required credentials for accessing the destination.

By configuring notification channels, administrators can ensure that the monitoring platform effectively delivers critical alerts, enabling prompt responses to potential issues and maintaining the stability and performance of GDC environments.

Before you begin

To get the permissions that you need to manage ObservabilityPipeline custom resources, ask your Organization IAM Admin or Project IAM Admin to grant you one of the associated ObservabilityPipeline roles.

Additionally, to get the permissions that you need to manage ConfigMap objects in your project namespace required to define configuration rules, ask your Organization IAM Admin or Project IAM Admin to grant you the ConfigMap Creator role.

Depending on the level of access and permissions you need, you might obtain creator, editor, or viewer roles for these resources in an organization or a project. For more information, see Prepare IAM permissions.

Configure notification channels

Configure notification channels for alerts in your project namespace:

Define your custom configuration for notification channels in a YAML file named alertmanager.yml. You must follow the same syntax as the Alertmanager specification:

https://prometheus.io/docs/alerting/latest/configuration/.

Create a ConfigMap object and include your custom configuration from the alertmanager.yml file in the data field.

The following example shows how the ConfigMap object must look with the alertmanager.yml file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: CONFIGMAP_NAME
  # The namespace must match your project namespace.
  namespace: PROJECT_NAMESPACE
data:
  # The file name must be alertmanager.yml
  alertmanager.yml: |
    # Define your notification channels.
    # Add the custom configuration in the following lines of this file.
    # Follow the same syntax as in https://prometheus.io/docs/alerting/latest/configuration

  [...]

Replace the following:

CONFIGMAP_NAME: the name of the ConfigMap definition file. You use this name later in the ObservabilityPipeline custom resource.
PROJECT_NAMESPACE: your project namespace.

Apply the ConfigMap object to the Management API server within the same namespace as your configured alerts:
```
kubectl --kubeconfig KUBECONFIG_PATH apply -f CONFIGMAP_NAME.yaml
```
Replace the following:
- KUBECONFIG_PATH: the path to the kubeconfig file for the Management API server.
- CONFIGMAP_NAME: the name of the ConfigMap definition file.
Edit the ObservabilityPipeline custom resource specification to import your configuration to the alertmanagerConfig field:
```
 # Configure the observability pipeline.
apiVersion: observability.gdc.goog/v1
kind: ObservabilityPipeline
metadata:
  # The namespace must match your project namespace.
  namespace: PROJECT_NAMESPACE
  name: OBSERVABILITY_PIPELINE_NAME
spec:
  # Configure alerts.
  alerting:
    # The alerting configuration must be in the ConfigMap.
    # The value must match the ConfigMap name exactly.
    alertmanagerConfig: CONFIGMAP_NAME

  [...]
```
Replace the following:
- PROJECT_NAMESPACE: your project namespace.
- OBSERVABILITY_PIPELINE_NAME: the name of the ObservabilityPipeline custom resource.
- CONFIGMAP_NAME: the name you used for the ConfigMap definition file. The name must match exactly.
Note: In the alerting field of the ObservabilityPipeline custom resource, you can also define other configurations for the project namespace such as volumes (volumes) and volume mounts (volumeMounts). You reference volume mount names in the alertmanager.yml file.
Save the file.
Apply the changes of the ObservabilityPipeline custom resource to the Management API server within the same namespace as your configured alerts:
```
kubectl --kubeconfig KUBECONFIG_PATH apply -f OBSERVABILITY_PIPELINE_NAME.yaml
```
Replace the following:
- KUBECONFIG_PATH: the path to the kubeconfig file for the Management API server.
- OBSERVABILITY_PIPELINE_NAME: the name of the ObservabilityPipeline definition file.