Use Case

Datadog monitors trigger alerts. Before routing to PagerDuty, you want to:

• Deduplicate noisy alerts
• Enrich with internal context
• Apply custom escalation rules
• Track alert patterns

Skooh forwards Datadog webhooks to your internal alert processor.

Setup

1. Create Skooh Endpoint

Create "Datadog Alerts" in Skooh.

2. Run the Skooh Proxy

Using the Desktop App:

1. Download the Skooh Desktop App
2. Enter your API key and hook token
3. Set forwarding URL (e.g., `http://localhost:3000/alerts/datadog\`)
4. Click Connect

Using Docker:

docker run -d \\
  --name skooh-datadog \\
  -e SKOOH_TOKEN=datadog-alerts \\
  -e SKOOH_API_KEY=your_api_key \\
  -e FORWARD_URL=http://alert-processor:3000/alerts/datadog \\
  --restart unless-stopped \\
  skooh/proxy

3. Configure Datadog

Note: See Datadog's Webhooks documentation for current setup instructions.

In Datadog:

1. Go to Integrations → Webhooks
2. Create new webhook
3. Enter your Skooh URL
4. Customize payload (see below)
5. Reference webhook in monitor notification: `@webhook-skooh`

Custom Payload

{
  "alert_id": "$ALERT_ID",
  "alert_title": "$ALERT_TITLE",
  "alert_status": "$ALERT_STATUS",
  "alert_type": "$ALERT_TYPE",
  "alert_metric": "$ALERT_METRIC",
  "alert_query": "$ALERT_QUERY",
  "alert_scope": "$ALERT_SCOPE",
  "hostname": "$HOSTNAME",
  "org_name": "$ORG_NAME",
  "tags": "$TAGS",
  "link": "$LINK",
  "last_updated": "$LAST_UPDATED",
  "event_msg": "$EVENT_MSG"
}

Alert Processing

app.post('/alerts/datadog', express.json(), async (req, res) => {
  const alert = req.body;
  
  // 1. Dedupe check
  const isDuplicate = await alertStore.isDuplicate(alert.alert_id);
  if (isDuplicate) {
    console.log(\`Duplicate alert \${alert.alert_id}, skipping\`);
    return res.status(200).send('OK');
  }
  
  // 2. Enrich with internal context
  const enrichedAlert = await enrichAlert(alert);
  
  // 3. Apply escalation rules
  const escalation = determineEscalation(enrichedAlert);
  
  // 4. Route alert
  await routeAlert(enrichedAlert, escalation);
  
  res.status(200).send('OK');
});

async function enrichAlert(alert) {
  // Add team ownership
  const service = parseServiceFromTags(alert.tags);
  const team = await teamDirectory.getOwner(service);
  
  // Add recent deploy info
  const recentDeploys = await deployDb.getRecent(service, '1h');
  
  // Add similar past alerts
  const similarAlerts = await alertStore.findSimilar(alert.alert_title);
  
  return {
    ...alert,
    team,
    recentDeploys,
    similarAlerts,
    possibleDeployRelated: recentDeploys.length > 0
  };
}

Deduplication

class AlertStore {
  constructor() {
    this.alerts = new Map(); // Use Redis in production
  }
  
  async isDuplicate(alertId) {
    const existing = this.alerts.get(alertId);
    if (existing && Date.now() - existing.timestamp < 5 * 60 * 1000) {
      // Same alert within 5 minutes = duplicate
      return true;
    }
    this.alerts.set(alertId, { timestamp: Date.now() });
    return false;
  }
  
  async findSimilar(title) {
    // Find alerts with similar titles in last 24h
    const recent = Array.from(this.alerts.entries())
      .filter(([_, v]) => Date.now() - v.timestamp < 24 * 60 * 60 * 1000);
    
    // Fuzzy match title
    return recent.filter(([id, _]) => similarity(id, title) > 0.8);
  }
}

Custom Escalation Rules

function determineEscalation(alert) {
  // Deploy-related alerts go to deployer first
  if (alert.possibleDeployRelated) {
    return {
      primary: alert.recentDeploys[0]?.author,
      backup: alert.team?.oncall,
      delay: '5m'
    };
  }
  
  // Critical alerts skip delay
  if (alert.alert_status === 'error') {
    return {
      primary: alert.team?.oncall,
      backup: 'platform-oncall',
      delay: '0m'
    };
  }
  
  // Default escalation
  return {
    primary: alert.team?.oncall,
    backup: 'platform-oncall',
    delay: '15m'
  };
}

Testing Alert Logic

Use Skooh replay to test:

1. Deduplication logic
2. Enrichment accuracy
3. Escalation rules
4. PagerDuty integration

Without generating real alerts!

Set up Datadog alerts →