Agentic AI for Incident Response: Where It Actually Helps
Agentic AI for Incident Response: Where It Actually Helps
I'm currently doing two things that keep colliding with each other. At work, I'm using agentic AI to speed up incident investigation and root-cause analysis. Outside of work, I'm leading a small team building an enterprise AI governance platform — the thing that watches how AI tools get used across an org and flags when sensitive data leaks into them. Spend enough time in both seats and the hype around "agentic ops" starts sounding very different from the reality of running it against production systems.
Here's what I've actually found useful, and where I still keep a human firmly in the loop.
What "Agentic" Actually Means Here
Worth being precise, because the term gets stretched. A chatbot that answers "what does this error mean" isn't agentic — it's a lookup with better phrasing. An agent is something that can take a sequence of actions on its own: pull logs, correlate them against a deploy timeline, form a hypothesis, and decide what to check next — without a human writing each query by hand.
That autonomy is exactly what makes it useful for the tedious parts of incident response, and exactly what makes it dangerous if you give it too much rope.
Where It Genuinely Helps
Context assembly, but smarter than a static script. My observability post covered auto-attaching metrics and logs when an alert fires — that's deterministic automation, not agentic. The agentic version goes further: instead of just fetching the last 30 minutes of metrics, it reasons across sources — "latency spiked on service B two minutes after a deploy to service A, and B calls A's new endpoint" — and proposes that as a starting hypothesis, not just a pile of data.
Correlation a human would take longer to spot. Cross-referencing a deploy event three services upstream with a downstream symptom is exactly the kind of connective work that's slow for a tired human at 3 AM and fast for something that can query five systems in parallel and hold all the context at once.
A first-draft postmortem. Timeline reconstruction — when did the alert fire, when did the deploy go out, when was the fix applied — is mechanical. Having a draft timeline waiting when the human starts writing the actual postmortem saves real time without touching judgment calls.
Suggesting the next diagnostic step. Not executing it — suggesting it. "Check whether the connection pool is exhausted" is a good agent output. Running the command that resets the connection pool is not, until a human says so.
Where It Doesn't Help (Yet)
Anything with a blast radius. An agent should never be the one that restarts a production database, deletes a resource, or rolls back a deploy without a human explicitly approving that specific action. Confidence in an LLM's output is not the same as correctness, and the failure mode isn't "it does nothing" — it's "it does something plausible and wrong."
Novel incidents. Agents are pattern-matchers dressed up as reasoners. A failure mode that looks like fifty things in the training data gets handled well. A genuinely new failure mode — the kind that becomes next quarter's "we've never seen this before" postmortem — is where a human's actual understanding of the system still wins.
Judgment calls. Roll back or fix forward? How do we word the customer status page update? Is this severity 1 or severity 2? These aren't information-retrieval problems. They're calls that carry organizational and reputational weight, and they stay human.
The Architecture: An Agent on a Leash
The pattern I use is a strict split between what the agent can read and what it can do:
READ_ONLY_TOOLS = [
"query_metrics",
"query_logs",
"get_deploy_history",
"get_dependent_services",
]
WRITE_TOOLS = [
"draft_incident_report", # produces text, changes nothing
"propose_next_step", # suggests an action, doesn't take it
]
# Anything beyond this list requires an explicit human approval gate:
GATED_ACTIONS = [
"restart_service",
"rollback_deploy",
"scale_resource",
]
def execute_agent_action(action: str, params: dict, approved_by: str | None):
if action in READ_ONLY_TOOLS or action in WRITE_TOOLS:
return run_tool(action, params)
if action in GATED_ACTIONS:
if not approved_by:
raise PermissionError(f"{action} requires human approval before execution")
audit_log(action, params, approved_by)
return run_tool(action, params)
raise ValueError(f"Unrecognized action: {action}")
The agent can look at everything and propose anything. It can execute nothing consequential without a named human attached to the approval. That one line — approved_by — is the difference between a genuinely useful tool and an incident waiting to happen.
The Part That Connects to My Other Project
Here's the thing that doesn't get talked about enough: an agent with read access to production logs, metrics, and deploy history is itself a data exposure surface. It's seeing customer data, internal architecture details, sometimes secrets that leaked into a log line by accident. If you wouldn't hand a new contractor unrestricted read access to every system without an access review, don't hand it to an agent either.
That's not a hypothetical for me — it's literally what the governance platform I'm building is meant to catch: monitoring what data flows into AI tools, enforcing access policy, and flagging when something sensitive crosses a boundary it shouldn't. Agentic ops tools need the same governance as the humans who'd otherwise be running those queries by hand. Skipping that step because "it's just an AI" is how you get an incident review of your own.
Where This Is Going
The discipline that made the observability stack work — automate the diagnostic legwork, keep humans in charge of consequential decisions, log everything — is the same discipline that decides whether agentic AI in ops becomes a trustworthy tool or a liability with good PR. Right now it's genuinely useful for the parts that were always mechanical. It's not ready to hold the pager. Treat it accordingly, and it earns its keep.
