JACK SIU ← Back to portfolio

SRE case study · Oracle Cloud Infrastructure

Incident Automation Engine

An event-driven system that creates, routes, resolves, and notifies on production incidents without a human in the loop — cutting triage from about thirty minutes per ticket to under a minute, and removing roughly ten hours a week of on-call toil across 20+ weekly incidents. Built, shipped, and operated in production for multiple years.

~30m → <1m
Triage MTTR per ticket
~10 hrs
On-call toil removed per week
20+
Incidents automated per week
Multi-year
Continuously live in production

Architecture

How an alert becomes a resolved ticket

Select any component to inspect its role and the technology behind it. Keyboard accessible — tab to a node and press Enter.

SOURCES INGESTION PROCESSING CONSUMERS AUTO-RESOLVE Monitoring Alerts MONITORING Grafana alerts System Events SYSTEM Service events Health Checks HEALTH Failed probes Event Bus EVENT BUS Pub/sub · at-least-once Routing Engine ROUTER Config-driven rules Deduplication Engine DEDUP Noise suppression Enrichment Service ENRICHMENT Owner · runbook Ticket Engine TICKET ENGINE Create · route · resolve · notify Ticketing System TICKETING Internal system Real-time Data Stream STREAM Real-time telemetry Notification Service NOTIFY On-call paging

The write-up

Problem, approach, results, and what it cost

The problem

Triage was predictable toil

An on-call engineer gets paged, looks up the affected service, checks the runbook, manually files a ticket, routes it to the right team, and notifies the rotation — about thirty minutes per incident. Across 20+ incidents a week that's ~10 hours of engineering time spent on a task that follows the same shape every time. Textbook toil: manual, repetitive, automatable, and scaling linearly with the service.

The approach

Event-driven, decoupled, inspectable

Alerts, system events, and failed health checks publish to a central bus with at-least-once delivery, so producers and consumers deploy independently and spikes buffer instead of dropping. A config-driven router classifies each event, deduplication collapses flapping services into one incident, and enrichment attaches ownership and runbook context before anything reaches the ticket engine.

The results

MTTR and toil, measurably down

Triage dropped from ~30 minutes to under a minute per ticket, removing ~10 hours a week of on-call toil. Tickets now arrive pre-populated with severity, owner, and runbook links, so responders start acting instead of investigating. The system auto-resolves incidents when the underlying signal clears, and has run continuously through service migrations and rotation handoffs for years.

Tradeoffs

What operating it actually taught me

Rules-based routing is fast and inspectable but doesn't generalize to novel incident types — it works because the service catalog is stable, and that's a bet, not a guarantee. Deduplication uses a fixed time window, which occasionally collapses related-but-distinct incidents into one. Building it was the easy part; keeping it correct through schema changes and migrations is where the reliability lessons lived.