Detection Is Not Response

Last week Uber open-sourced ADR, short for "Agentic AI Detection and Response," with an accompanying MLSys 2026 paper and production deployment behind it. This is exactly the kind of release the field needs more of: a credible team, real telemetry engineering, a labeled benchmark, and documentation that is honest about its own limits. If you care about securing AI agents, you should read the repo.

What actually shipped

ADR is three things bundled under one name.

A Sensor that reads the local artifacts AI coding tools leave on disk (Claude Code's JSONL files under ~/.claude/projects/, Cursor's state.vscdb SQLite store, Codex sessions, Cline, Warp, Claude Desktop) and normalizes them into a unified AgentEvent schema.

A Detection system: a dual-agent LLM pipeline in which GPT-4o triages a session and Claude Sonnet, invoked via the Claude CLI with MCP, delivers the verdict of benign or malicious.

And ADR-Bench, a benchmark of 303 tasks (261 benign, 42 malicious) spanning 133 verified MCP servers and 17 attack techniques, with a clean task schema and AgentDojo integration.

Two more components are named in the paper but explicitly withheld from the release: Prevention, the blocking component, and Explorer, the offline red-teaming harness. The repo says "stay tuned." Until then, what is open source is detection and evaluation. The "Response" in the name is aspirational. What shipped is a DR product with the R removed.

The benchmark is truly valuable

Before the critique, credit where it is due. ADR-Bench is real infrastructure the field lacks: a labeled corpus with user_prompt, mcp_servers, expected_tools, ground_truth, and threat_technique per task, over a large set of actually-verified MCP servers. Anyone building agent security tooling can and should run against it.

That framing also recasts the headline numbers. The paper reports precision 1.000, recall 0.667, F1 0.800, zero false positives. Strong, but some caveats:

• The result is on the authors' own benchmark, scored by the authors' own detector. Normal for a systems paper, but it is a property of this benchmark, not a promise about your traffic.
• The positive class is 42 attacks. Recall of 0.667 means 28 caught and 14 sophisticated attacks missed, and every metric swings by roughly 2.4 points per task at that sample size. No confidence intervals are reported.
• Of the four baselines in the comparison table, only LlamaFirewall is reproducible from the repo. The ALRPHFS and GuardAgent rows are those papers' published numbers, not re-runs on the same harness, because baseline code was removed for licensing.

None of this is hidden. The docs say all of it plainly, which is to the team's credit. But "zero false positives" will get quoted without the context, and it should not be.

The Sensor detects the past

The Sensor is a post-hoc log parser. It ingests session artifacts after the agent has finished writing them.

Think about what that means for the attacks that matter. The canonical agent compromise is the lethal trifecta: private data, untrusted content, external communication, ending in exfiltration. Exfiltration is irreversible the instant it succeeds. So are credential theft and destructive commands. By the time a malicious session appears in a JSONL file or a SQLite store, the bytes have left, the secret is read, the command has run. This architecture can raise a good alert. It cannot, by construction, prevent the harm.

For malware on an endpoint, retrospective detection still has enormous value, because the attacker usually needs persistence, lateral movement, time. An agent compromise often needs none of that. One injected session, one tool call, one HTTP request, done. Detection latency is not a performance metric for agent security. It is the difference between an alert and a prevention.

There is a second problem: the collection contract is fragile. The Sensor depends on the undocumented, private on-disk formats of third-party tools. Anthropic, Cursor, or OpenAI can rename a field, change a schema, or encrypt a store in any release and silently break a parser. Anything that never writes to the watched file (a tool executed outside the instrumented app, a shelled-out subprocess, a redirected log) is invisible. An adversary who knows what the sensor reads knows exactly where not to write.

The Detector is part of the attack surface

The detector is two frontier-model calls per suspicious session, throttled to single-digit concurrency, with the docs candidly warning to "budget API spend accordingly." Three problems follow.

Scale. Two frontier LLM calls per flagged session is a research-evaluation cadence, not a SOC pipeline over thousands of developers. The paper measures accuracy; it does not demonstrate throughput or unit economics.

Injectability. The system asks an LLM, connected over MCP, to judge sessions that may themselves contain prompt-injection payloads. The obvious adversarial move is to craft the malicious session so that its own text steers the judge toward "benign." A benchmark of known attack techniques will not surface this, because the technique targets the evaluator, not the agent. If your security control reads attacker-controlled text with a language model, your security control is inside the threat model.

Auditability. Two LLMs deciding benign or malicious means the same session can score differently across runs, and the rationale is a generated explanation rather than a deterministic rule trace. A control whose verdicts are not reproducible is hard to tune, hard to regression-test, and hard to defend to an auditor.

Narrower than the name

"Agentic AI Detection and Response" sounds like coverage for autonomous agents broadly. The mechanism covers developer coding assistants on endpoints: Claude Code, Cursor, Codex, Cline, Warp. That is a real attack surface, and insider risk from prompt-injected coding agents deserves tooling. But a customer-facing support agent, a backend automation pipeline, or a multi-agent system running server-side has no ~/.claude/projects/ directory on an engineer's laptop to parse. The workforce-endpoint slice is what is covered; the production-agent estate, arguably the higher-stakes surface, is out of mechanistic reach.

One more caveat the repo states itself and a reader should not miss: the release is explicitly "not intended for production deployment," with dependencies pinned to versions carrying known CVEs deemed acceptable only in an isolated benchmark environment. This is research code, and says so.

Why this matters beyond one repo

I am not writing this to dunk on Uber. The release is more honest about its limits than most commercial products in this space, and the benchmark alone justifies its existence. I am writing it because ADR is the first big-name entrant in a category that is forming right now, and the category is forming around a comfortable pattern: collect logs, ask a model whether the session looked malicious, call it detection and response.

That pattern is comfortable because we already know how to build it. It is also structurally wrong for the failure modes that make agents dangerous, the irreversible ones. So when you evaluate anything wearing the "agentic detection and response" label, ask four questions:

1. Where does it sit? Reading artifacts after the session, or on the boundary where the model call, the tool call, and the network egress actually happen? Only the second position can turn a detection into a prevention.
2. Can it act before the irreversible step? Ask for a demonstration of a block, not an alert about something that already happened.
3. Are verdicts reproducible? Deterministic rules and detectors can be tuned, tested, and audited. A generated explanation cannot.
4. Where is it blind? Every sensor has coverage gaps. The trustworthy ones enumerate their own.

Uber gave the field a benchmark and an honest paper. The field should return the favor by being honest about the architecture: a log parser and an LLM judge, however well engineered, detect the past. Agents commit their harms in the present tense.