Linear Incident Runbook

Overview

Step-by-step procedures for handling production incidents with Linear integrations.

Prerequisites

• Production access credentials
• Monitoring dashboard access
• Communication channels configured
• Escalation paths defined

Incident Classification

Immediate Actions

Step 1: Confirm the Issue

# Check Linear API status
curl -s https://status.linear.app/api/v2/status.json | jq '.status'

# Quick health check
curl -s -H "Authorization: $LINEAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "{ viewer { name } }"}' \
  https://api.linear.app/graphql | jq

# Check your application health endpoint
curl -s https://yourapp.com/api/health | jq

Step 2: Gather Initial Information

// scripts/incident-info.ts
import { LinearClient } from "@linear/sdk";

async function gatherIncidentInfo() {
  const client = new LinearClient({ apiKey: process.env.LINEAR_API_KEY! });

  console.log("=== Linear Incident Information ===\n");

  // 1. Test authentication
  console.log("1. Authentication:");
  try {
    const viewer = await client.viewer;
    console.log(`   Status: OK (${viewer.name})`);
  } catch (error) {
    console.log(`   Status: FAILED - ${error}`);
  }

  // 2. Check teams access
  console.log("\n2. Team Access:");
  try {
    const teams = await client.teams();
    console.log(`   Accessible teams: ${teams.nodes.length}`);
  } catch (error) {
    console.log(`   Status: FAILED - ${error}`);
  }

  // 3. Test issue creation (dry run)
  console.log("\n3. Write Capability:");
  try {
    const teams = await client.teams();
    const result = await client.createIssue({
      teamId: teams.nodes[0].id,
      title: "[INCIDENT TEST] Delete immediately",
    });
    if (result.success) {
      const issue = await result.issue;
      await issue?.delete();
      console.log("   Status: OK (created and deleted test issue)");
    }
  } catch (error) {
    console.log(`   Status: FAILED - ${error}`);
  }

  console.log("\n=== End Information ===");
}

gatherIncidentInfo();

Runbook: API Authentication Failure

Symptoms

• All API calls returning 401/403
• "Authentication required" errors
• Sudden spike in auth errors

Diagnosis

# Test API key directly
curl -I -H "Authorization: $LINEAR_API_KEY" \
  https://api.linear.app/graphql

# Check for key format issues
echo $LINEAR_API_KEY | head -c 8
# Should output: lin_api_

# Verify key in secrets manager
vault read secret/data/linear/production
# or
aws secretsmanager get-secret-value --secret-id linear/production

Resolution Steps

1. Verify API key is loaded correctly bash # Check env var is set (don't print actual key) [ -n "$LINEAR_API_KEY" ] && echo "Key is set" || echo "Key is NOT set"

2. Check if key was rotated/revoked - Log into Linear dashboard - Navigate to Settings > API > Personal API keys - Verify key exists and is active

3. Generate new API key if needed - Create new key in Linear dashboard - Update secrets manager - Restart affected services

4. Rollback if recent deployment ```bash # Check last deployment git log --oneline -5

# Rollback to previous version git revert HEAD ```

Runbook: Rate Limiting Issues

Symptoms

• HTTP 429 responses
• "Rate limit exceeded" errors
• Degraded performance

Diagnosis

# Check current rate limit status
curl -I -H "Authorization: $LINEAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "{ viewer { name } }"}' \
  https://api.linear.app/graphql 2>&1 | grep -i ratelimit

# Check application metrics
curl -s http://localhost:9090/api/v1/query?query=linear_rate_limit_remaining | jq

Resolution Steps

1. Identify rate limit cause bash # Check request patterns grep "linear" /var/log/app/*.log | grep -E "[0-9]{4}-[0-9]{2}-[0-9]{2}" | wc -l

2. Implement emergency throttling ```typescript // Emergency rate limiter const EMERGENCY_MODE = true; const MIN_DELAY_MS = 5000;

async function emergencyThrottle(fn: () => Promise): Promise { if (EMERGENCY_MODE) { await new Promise(r => setTimeout(r, MIN_DELAY_MS)); } return fn(); } ```

3. Disable non-critical operations - Stop background sync jobs - Disable polling (if using) - Queue non-urgent requests

4. Wait for rate limit reset - Linear resets every minute - Monitor X-RateLimit-Reset header

Runbook: Webhook Failures

Symptoms

• Events not being received
• Webhook signature validation failing
• Processing timeouts

Diagnosis

# Check webhook endpoint is reachable
curl -I https://yourapp.com/api/webhooks/linear

# Check recent webhook logs
tail -100 /var/log/webhooks.log | grep linear

# Verify webhook secret
echo $LINEAR_WEBHOOK_SECRET | wc -c
# Should be > 20 characters

Resolution Steps

1. Verify endpoint health typescript app.get("/api/webhooks/linear/health", (req, res) => { res.json({ status: "ok", timestamp: new Date().toISOString() }); });

2. Check signature verification ```typescript // Debug signature verification function debugVerifySignature(payload: string, signature: string): boolean { const secret = process.env.LINEAR_WEBHOOK_SECRET!; const expected = crypto.createHmac("sha256", secret).update(payload).digest("hex");

   console.log("Debug: Received signature:", signature); console.log("Debug: Expected signature:", expected); console.log("Debug: Secret length:", secret.length);

   return signature === expected; } ```

3. Recreate webhook if needed - Go to Linear Settings > API > Webhooks - Delete existing webhook - Create new webhook with same URL - Update webhook secret in secrets manager

Communication Templates

Initial Incident Announcement

**INCIDENT: Linear Integration Issue**
Severity: SEVX
Status: Investigating
Impact: [Description of user impact]
Start Time: [UTC timestamp]

We are investigating issues with our Linear integration. Updates will follow.

Status Update

**UPDATE: Linear Integration Issue**
Status: [Investigating/Identified/Mitigating/Resolved]
Time: [UTC timestamp]

Update: [What we know/did]
Next Steps: [What we're doing next]
ETA: [If known]

Resolution Notice

**RESOLVED: Linear Integration Issue**
Duration: [X hours Y minutes]
Root Cause: [Brief description]
Impact: [What was affected]

A full post-mortem will follow within 48 hours.

Post-Incident

Immediate Actions

[ ] Verify all systems are healthy
[ ] Clear any queued/stuck jobs
[ ] Validate data consistency
[ ] Notify stakeholders of resolution

Post-Mortem Checklist

[ ] Timeline of events
[ ] Root cause analysis
[ ] Impact assessment
[ ] What went well
[ ] What could be improved
[ ] Action items with owners

Resources

• Linear Status Page
• Linear API Documentation
• Internal: On-call runbook wiki
• Internal: Escalation contacts

Next Steps

Learn data handling patterns with linear-data-handling.