AI SRE · evidence-cited RCA

Read your production.
Explain your incidents.

Autonomous SRE agent that investigates alerts, cites its evidence, and generates the dashboards you never got around to building. Ships as a Helm chart. Runs in your VPC.

Prometheus · OpenTelemetry · Loki · Tempo · Datadog exports · Splunk HEC

Legend — INC-4471service callanomaly propagationroot causeedge-lbok · 12.4k req/scheckout-apip99 1,183mspayments-apierr 6.2%payments-dbpool 98/100cart-cachehit-rate 41.2%stripe-proxytimeouts 89/minorders-workerqueue 12,847z=6.4 · conn-pool exhaustion4 signals citedp99 +412% vs baselinekpi orders/min −17.4%▲ deploy a3f91c2 · 41m agocorr 0.82 with #01ROOT CAUSEconfidence 0.87INFRASAGE — INCIDENT DRAWINGDWG NO. INC-4471TRIGGER MANUAL · OPERATOR ON-CALLREV 03 · 6.2sDATE 2026-08-28 · 03:14 UTCSHEET 1/1 · SCALE 1:1INC-4471 · anomaly propagationpropagationroot causeedge-lbokcheckout-apip99 1,183mspayments-apierr 6.2%payments-dbpool 98/100p99 +412%deploy a3f91c2z=6.4ROOT CAUSEconfidence 0.87INFRASAGE — DWG INC-4471MANUAL TRIGGER · 03:14 UTC · SCALE 1:1
Fig. 01 — Detect

Anomaly Detection

Multi-modal baselines detect what's out of pattern across metrics, logs, traces, and business KPIs, with no thresholds to babysit per service. Every score is explainable: z-score, changepoint, seasonal residual, or KPI-drift, plus the exact window it fired on.

Detail A — Anomaly Scorer03:14:03 UTC
payments-dbconnection_pool_utilization
trailing 15m · 1s bucket
detector: z-score · window 60s · min-samples 30
score6.42confidence0.94first bad sample03:14:03.212Zgrouped incidentINC-4470 →
Fig. 02 — Triage

Signal Triage

Dedup, prioritize, and causal-cluster raw telemetry so the pager fires on one root event, not fifty symptoms.

Detail B — Signal Triage03:14:21 UTC
CRITpayments-api p99 > 800ms · breached 4m12s
CRITpayments-db connection pool 98/100
WARNstripe-proxy timeout burst 89/min
WARNcheckout-api error ratio 2.1%
INFOdeploy payments-db@a3f91c2 · 41m ago
+ 7 more suppressed as duplicates · pager fired once
CRITcart-cache hit-rate 41.2% · floor 70%
WARNcart-cache evictions 12,040/min
WARNcheckout-api p95 +38% vs baseline
INFOcart-cache pod restart ×2 · 22m ago
+ 5 more suppressed as duplicates · pager fired once
WARNemail-relay send failures 312/min
WARNsmtp-egress 429 responses climbing
INFOqueue depth 8,912 · draining slowly
+ 5 more suppressed as duplicates · pager fired once
WARNsearch-index p50 flapping · 6 cycles/h
INFOshard 3 rollover retry ×14
+ 3 more suppressed as duplicates · pager fired once
Fig. 03 — Explain

Evidence-cited RCA

The agent walks your telemetry, cites every tool call, and commits a root-cause narrative you can read in a scrollback.

Detail C — Evidence-cited RCArca-session · INC-4472
rca — payments-api — manual trigger — 08:14:22 UTCoperator: on-call
01rca.start--incident INC-4471--scope payments-api// operator-triggered
02toolprometheus_range("rate(http_5xx[1m])", "-15m..now")→ 118 series · anomaly on payments-api
03Error ratio on payments-api climbs 0.4% → 6.8% at 08:11:47. Upstreams checkout-api and order-svc are quiet. Fault likely local.
04toolget_dependency_graph("payments-api", depth=2)→ 4 nodes · payments-db, stripe-proxy, redis-tx, kafka-events
05toolget_anomaly_scores(nodes=4, window="15m")→ payments-db z=3.14 · stripe-proxy z=0.42
06payments-db dominates the anomaly hemisphere. Checking recent mutations before implicating storage.
07toolget_deploy_events(service="payments-db", since="-45m")→ 1 event · migration 20260828-013 at 08:09:12
08Migration 20260828-013 landed 2m35s before error ratio inflected. Signal alignment across metric, dependency, and change surface.
Verdictconfidence 0.87
Root cause · payments-db migration 20260828-013
Schema change added a non-covering index on tx_events(tenant_id, created_at), degrading tail latency on the checkout write path. Error ratio trails deploy by 2m35s across 4 cited signals.
payments-dbstripe-proxycheckout-apiorder-svcOpen trace
Manual trigger · 4 tools cited · 8 lines
Fig. 04 — Remediate

Approval-Gated Runbooks

Every destructive action lands in Slack for a human approval first. The agent proposes; on-call decides.

Detail D — Approval-Gated Runbooks
#agent-approvals14 members · pinned by @on-call-sre
runbook: rollback deploy payments-db@a3f91c2
impact
63 pods · ~2,781 req/min
blast radius
payments-api · stripe-proxy · checkout-api
evidence
INC-4473 · Tempo trace 8f2c · Loki lines 214
confidence
0.87
Waiting for a human decision · requested 14s ago · you're the approver — pick one
Approved by @you · just now · rolled back in 47s · error ratio recovering
Rejected by @you · runbook cancelled · zero writes executed · incident stays open
Modify requested by @you · scope narrowed to payments-db only · re-queued for approval
Fig. 05 — Prioritize

Business KPI Guarding

Baseline your revenue-critical KPIs by hour-of-week. Anomalies tag which incidents are actually costing money.

Detail E — Business KPI Guardingbaseline: hour-of-week · window: 24h
checkout success ratepayments-api · checkout-api
baseline band (p10–p90)liveanomaly
03:14 UTC
checkout drop 2.3%·$12k/min at risk
03:14 UTC
AOV −4.1%·$3.2k/min at risk
not armed yet
4 more days of baseline needed
Fig. 06 — Access

Natural-language Production Chat

Ask InfraSage in plain English. It answers grounded in live telemetry, code, and tribal knowledge from your runbooks.

Learn more
Detail F — Production Chat

why is checkout error rate up in eu-west-1?

Checkout error climbed 0.3% → 6.2% in eu-west-1 starting 13:47 UTC, correlated with a payment-gateway timeout spike (p99: 220ms → 8.4s). See: INC-4472. Trace →

  • prometheus.range(checkout_error_ratio, 30m)
  • tempo.trace(stripe-proxy, p99)
  • loki.grep(payments-api, "gateway timeout")
  • evidence_graph.lookup(INC-4472)
Design partners · In private beta

Placeholders shown until partners approve public attribution. Names withheld under NDA.

  • DESIGN PARTNER · Fortune 500 e-commerce
  • DESIGN PARTNER · Series C fintech
  • DESIGN PARTNER · Global SaaS · $1B+ ARR
  • DESIGN PARTNER · Payments platform
  • DESIGN PARTNER · Public cloud provider (private beta)

FIELD NOTE · DESIGN PARTNER

“The RCA agent surfaced the deploy-related root cause on INC-4471 before I finished loading the second Grafana tab. That’s the honest headline — every alert I’ve triaged since has felt like a downgrade.”

Head of SRE · Fortune 500 e-commerce design partner

  • 3m 42smedian resolution
  • 92%alert dedup
  • 87%top-1 RCA