{
  "skill_name": "incident-triage-brief",
  "evals": [
    {
      "id": 1,
      "prompt": "I am a Technical Analyst helping the on-call lead with an active API incident. Please turn these notes into a triage brief for SRE and the backend owner. Service: subscription-api, production. Alert fired at 14:06 UTC because POST /v1/subscriptions 500s hit 11%, baseline is under 0.2%. p95 latency rose from 350 ms to 2.9 s. Deploy v2026.07.31.9 went out at 13:58 UTC with a change to coupon validation. Logs from 14:04-14:18 show many `CouponServiceTimeout after 1800ms`, then `SubscriptionCreateFailed`, request IDs mostly from us-east-1. Coupon service dashboard shows normal CPU but queue depth climbed from 12 to 1,900 starting at 14:02. Support has 7 tickets from self-serve customers unable to start paid plans. Enterprise renewals are not affected because they use a different workflow. Rollback started 14:22 and 500s dropped to 2.4% by 14:30, but queue depth is still above 800.",
      "expected_output": "A Technical Analyst triage brief that builds a timeline, assesses self-serve subscription impact, treats coupon validation/queue backpressure as the leading hypothesis with rollback evidence but not final root cause, notes enterprise workflow exclusion, lists residual queue-depth risk, and recommends next checks for coupon service queue, traces, deploy diff, retries, and rollback recovery."
    },
    {
      "id": 2,
      "prompt": "Create an incident triage brief from these mixed signals. Audience: data platform owner and support leadership. At 08:42 CET our daily account-health dashboard showed stale data for EMEA customers. Support says CSMs are seeing yesterday's health scores, and one renewal-risk report exported at 08:55 used old data. Airflow job `health_score_rollup` started at 01:00 CET and normally finishes by 01:40. Today's run says success at 01:37, but the warehouse table `account_health_daily` has max partition date yesterday. Logs show `0 rows written` after `source_customer_events unavailable, using cached source`. There was a warehouse permission change at 00:48 CET for the dbt service account. No API errors are showing in the dashboard app. A backfill was triggered at 09:20 and is still running. Need a brief that does not blame the dashboard team if the evidence points elsewhere.",
      "expected_output": "A triage brief that separates dashboard symptom from data freshness cause, includes timeline and impact for EMEA CSMs/renewal reports, identifies warehouse/dbt permission or cached source fallback as leading hypotheses, notes that Airflow success is misleading because zero rows were written, avoids blaming the dashboard app, and recommends checks for service-account permissions, source availability, partition freshness, backfill completion, and report re-export."
    },
    {
      "id": 3,
      "prompt": "I need a concise incident handoff for mobile engineering. Incident: push notifications delayed. Alert: notification delivery p95 exceeded 20 minutes at 19:12 PDT. Normal is under 90 seconds. Customer reports started around 19:20 from West Coast users saying order-ready alerts came late. Metrics: enqueue rate normal, worker CPU 92%, queue depth from 4k to 185k between 19:05 and 19:30, provider API status page green. Logs: many `DeviceTokenBatchTooLarge` warnings after mobile app release 8.14.0 began rollout at 18:50 to 10% iOS users. Android delivery looks normal. A worker autoscale policy increased replicas from 12 to 28 at 19:35, p95 improved to 8 minutes by 19:48 but queue depth is still 74k. There was also a marketing campaign at 19:00, but send volume was within forecast.",
      "expected_output": "A concise handoff that timelines release, marketing campaign, alert, reports, autoscale mitigation, and partial recovery; identifies iOS token batch size plus worker saturation/backlog as likely hypotheses; treats the provider status page and forecasted campaign as lower-support causes; states West Coast/iOS/order alert impact and residual backlog; and recommends next checks for app release batching behavior, worker throughput, queue drain ETA, token batch logs, and whether to pause rollout."
    }
  ]
}