An AI monitoring system where agents track token usage, error spikes and downtime across LLM applications and raise alerts on anomalies — a multi-agent setup with Prometheus and LLM-based log summarizers.
As clients shipped more LLM features, their observability stack couldn't keep up: dashboards showed everything and explained nothing, and on-call engineers drowned in noisy alerts.
The challenge
LLM apps fail in fuzzy ways — a quiet quality regression, a slow token-cost creep, an intermittent timeout. Threshold alerts either miss these or fire constantly, and raw logs take an engineer an hour to read.
Our approach
We deployed monitoring agents that watch each metric stream and reason about what's normal, so alerts fire on genuine anomalies rather than fixed thresholds. Prometheus handles collection; an LLM-based summarizer turns a wall of logs into a two-line incident brief the moment something breaks.
Under the hood
- Agents — Per-metric monitoring agents, multi-agent coordination
- Metrics — Prometheus collection & storage
- Signals — Token usage, error spikes, downtime, latency
- Summaries — LLM-based log summarizers for incident briefs
What we built
- Monitoring agents that learn each metric's normal range
- Anomaly alerts that cut through threshold noise
- LLM log summaries that explain incidents in seconds
- A multi-agent setup spanning many LLM applications
Results
- On-call engineers triage from a brief, not a log dump
- Silent quality and cost regressions get caught early
A Lightning Leap case study — one of the AI systems we've designed, built, and shipped.