Production systems need more than dashboards—they need alerts that reach the right people at the right time through the right channels. GCP Cloud Monitoring provides the detection layer, but the path from “metric crossed a threshold” to “engineer sees a Discord notification with actionable context” requires a carefully designed pipeline. Each stage—detection, routing, deduplication, and delivery—introduces its own failure modes and design decisions.
In a typical production infrastructure repository, the alert pipeline follows a four-stage architecture: Cloud Monitoring alerting policies detect anomalies and publish structured notifications to Pub/Sub topics, Cloud Functions subscribe to those topics and enrich the alert payload with context (affected service, runbook links, severity), Firestore provides incident deduplication with TTL-based document expiry to prevent notification storms, and finally a webhook delivers the formatted alert to a Discord channel where the on-call team monitors it.
This serverless pipeline scales automatically with alert volume, costs nothing when idle, and decouples detection from delivery so that adding new notification channels (Slack, PagerDuty, email) requires only a new Cloud Function subscriber—no changes to the detection policies.
An alerting policy defines a condition (e.g., “CPU utilization > 80% for 5 minutes”), a notification channel, and optional documentation. Policies use Monitoring Query Language (MQL) or the legacy filter syntax to specify which metric time series to watch. When the condition triggers, Cloud Monitoring creates an incident and pushes a notification to all configured channels, including Pub/Sub topics.
Pub/Sub decouples alert producers (Cloud Monitoring) from consumers (Cloud Functions). A single topic can fan out to multiple subscriptions, each with independent delivery guarantees. If a Cloud Function is temporarily down, Pub/Sub retries delivery with exponential backoff, ensuring no alerts are silently dropped. This decoupling is what makes the pipeline extensible—adding a new consumer requires only a new subscription.
When a condition remains in a violated state, Cloud Monitoring continues to send notifications at the alerting policy’s re-notification interval. Without deduplication, the Discord channel would be flooded with repeated alerts for the same incident. The pipeline uses Firestore documents keyed by incident ID with a TTL field. If a document for that incident already exists, the Cloud Function skips the notification. The TTL ensures old incidents are automatically cleaned up.
Discord webhooks accept structured JSON payloads that render as rich embeds—colored sidebars, fields, timestamps, and links. The Cloud Function formats each alert into an embed with severity-based color coding (red for critical, yellow for warning, green for resolved), the affected resource name, a summary of the violated condition, and a link to the Cloud Monitoring incident page for one-click triage.
The end-to-end flow starts when a Cloud Monitoring alerting policy detects that a metric condition has been violated. The policy publishes a JSON notification to a Pub/Sub topic. A Cloud Function with a Pub/Sub trigger receives the message, parses the notification payload, checks Firestore for a deduplication record, and—if the alert is new—formats and sends a Discord webhook request.
// Terraform: alerting policy that publishes to Pub/Sub
resource "google_monitoring_alert_policy" "high_cpu" {
display_name = "High CPU Utilization"
combiner = "OR"
conditions {
display_name = "CPU > 80% for 5 min"
condition_threshold {
filter = "metric.type=\"compute.googleapis.com/instance/cpu/utilization\""
comparison = "COMPARISON_GT"
threshold_value = 0.8
duration = "300s"
aggregations {
alignment_period = "60s"
per_series_aligner = "ALIGN_MEAN"
}
}
}
notification_channels = [
google_monitoring_notification_channel.pubsub.name
]
}
resource "google_monitoring_notification_channel" "pubsub" {
type = "pubsub"
display_name = "Alert Pub/Sub Channel"
labels = {
topic = google_pubsub_topic.alerts.id
}
}The Cloud Function is the core of the pipeline. It receives the Pub/Sub message, deserializes the Cloud Monitoring notification, checks Firestore for deduplication, and sends the Discord webhook.
import json
import hashlib
from datetime import datetime, timedelta
from google.cloud import firestore
import requests
db = firestore.Client()
DISCORD_WEBHOOK_URL = "https://discord.com/api/webhooks/..."
DEDUP_TTL_HOURS = 4
def process_alert(event, context):
"""Pub/Sub triggered Cloud Function for alert routing."""
import base64
payload = json.loads(base64.b64decode(event["data"]).decode("utf-8"))
incident = payload.get("incident", {})
incident_id = incident.get("incident_id", "unknown")
state = incident.get("state", "open")
policy_name = incident.get("policy_name", "Unknown Policy")
summary = incident.get("summary", "No summary available")
resource = incident.get("resource_name", "unknown")
# Deduplication check
dedup_ref = db.collection("alert_dedup").document(incident_id)
dedup_doc = dedup_ref.get()
if dedup_doc.exists and state == "open":
return "duplicate, skipping"
# Write dedup record with TTL
dedup_ref.set({
"incident_id": incident_id,
"first_seen": datetime.utcnow(),
"expire_at": datetime.utcnow() + timedelta(hours=DEDUP_TTL_HOURS),
"state": state,
})
# Format and send Discord embed
color = 0xFF0000 if state == "open" else 0x00FF00
embed = {
"embeds": [{
"title": f"{'🔴' if state == 'open' else '🟢'} {policy_name}",
"description": summary,
"color": color,
"fields": [
{"name": "Resource", "value": resource, "inline": True},
{"name": "State", "value": state.upper(), "inline": True},
],
"timestamp": datetime.utcnow().isoformat(),
}]
}
resp = requests.post(DISCORD_WEBHOOK_URL, json=embed, timeout=10)
resp.raise_for_status()
return f"sent alert for {incident_id}"Firestore supports TTL policies that automatically delete documents whose designated timestamp field has passed. By setting an expire_at field on each deduplication document, old incident records are cleaned up without manual intervention or scheduled jobs.
# Configure Firestore TTL policy via gcloud
gcloud firestore fields ttls update expire_at \
--collection-group=alert_dedup \
--enable-ttl \
--project=my-project-prodWhen a condition returns to a healthy state, Cloud Monitoring sends a “closed” notification. The pipeline treats resolved alerts differently: it always sends them (no dedup suppression for resolutions), uses a green color code in the Discord embed, and updates the Firestore document’s state to “closed.” This ensures the team sees both the alert and its resolution, providing closure without requiring manual incident tracking.
Not all alerts deserve the same treatment. Critical alerts (database down, data loss risk) should page immediately via PagerDuty, while warnings (elevated latency, approaching disk capacity) can go to Discord for asynchronous triage. Implement this by publishing all alerts to a single Pub/Sub topic and using multiple Cloud Function subscribers that inspect the severity field and route to the appropriate channel.
Raw Cloud Monitoring notifications contain metric names and resource IDs, but engineers need context: which service is affected, who owns it, and what runbook to follow. The Cloud Function can enrich alerts by querying a service catalog (a Firestore collection or a static JSON config) that maps resource labels to human-readable service names, owner teams, and runbook URLs.
Publish a test message directly to the Pub/Sub topic with a synthetic incident payload. This exercises the full pipeline—Cloud Function trigger, Firestore dedup, Discord delivery—without waiting for a real metric violation. Include a "test": true field so the Cloud Function can tag the Discord embed as a test and avoid confusion.