IT operations: AI agents as first responders

The alert came in at two and the fix started at four

The problem

Most of an incident is not the repair. It is the search. An engineer wakes up to an alert, and before touching anything has to work out what is actually broken: pull logs from several services, check which metrics moved and when, find out what deployed in the last few hours, look at whether an upstream dependency is degraded, and rule out the three things that looked like this last time and were not. That work is mechanical, it is the same work every time, and it happens while the clock is running and the person doing it is half awake.

The usual answer

It also happens badly at three in the morning. The engineer who knows the system best is not always the one on the rota, and the one who is on the rota spends the first stretch reconstructing context that already exists in the tooling.

How we approach it

We put agents on that stretch. When an alert fires, they gather the logs, metrics, traces and recent deploys that relate to it, correlate what moved with what changed, check the alert against previous incidents that looked similar, and write an initial read: what is failing, when it started, what changed just before, and what has been ruled out. That lands with the alert.

What changes

The engineer opens a summary instead of a terminal. Judgement stays with the person, because deciding what to do about a failing service is not a task to hand over, but they start from a position it would have taken them ninety minutes to reach. Time to resolution drops without adding headcount to the rota, and the write-up afterwards is easier because the timeline was assembled as it happened.