Coding Agents Are Not Enough: Why Security Investigation Demands a Context Layer

Most cloud security questions ask about a whole population: which roles can read your storage buckets, which resources fail a control, what another account can reach. Coding agents can now answer these questions by calling cloud APIs directly, so we tested what a security context layer still adds on top of that access. We gave the Sola Security Brain, Claude Code and OpenAI Codex the same 28 analyst-style questions and read-only access to the same live AWS environment. Then we scored each system on coverage: its share of the security-relevant claims all three produced together, graded blind and weighted by security consequence.

Where the gap comes from

The Security Brain surfaced 1.6x what Claude Code did and nearly 2x what Codex did, and it led all six themes. Its lead was widest on blast radius (+116% over Claude Code) and cross-account exposure (+104%), where answers depend on multi-hop relationships across the whole estate. It ran on Claude Sonnet 4.6, a cheaper model than either agent’s, and cost 17.8x and 20.8x less per task. It was slower on raw wall-clock time, but per unit of coverage it was 1.4x and 1.5x faster. The four tasks it didn’t lead all involved relationships the layer doesn’t model yet, such as multi-reader secret sets. On the 24 tasks where the graph holds the relationship, Sola outperforms both agents by a wide margin. On the other 4, where the graph doesn’t yet pre-index that reach, all three land in the same range. Same models, same environment, and the advantage appears exactly where the context layer does, and nowhere else.

A clean negative that wasn’t

We also noticed that the coding agents fell short in one very specific way. When we asked who can read the company’s storage buckets, Claude Code checked 40 out of roughly 5,000 buckets, then reported that none had access policies. The Security Brain, on the other hand, checked all of them. It found 65 buckets with policies granting read access to anyone, plus five outside accounts with read access to company data. In other words, a false negative from a coding agent reads exactly like a true one, so a reader cannot identify the error from the answer alone. These results come from AWS, the cloud environment where coding agents perform best. We expect the gap to widen in multi-vendor environments, where identities, workloads and data span systems that share no common identifiers and were never designed to interoperate.

What this means for your team

The broader implication concerns how security teams evaluate the answers their tools return. A clean negative about an environment carries weight only when the system behind it examined the entire population in question. An agent that inspects a sample can produce an answer that is fluent, well reasoned and honest about its own limits, yet still wrong in the direction that matters most, because the exposure it never surfaced is the one an attacker is most likely to find. Completeness, in other words, has to exist in the underlying representation before the question is asked, since an agent working live against raw APIs cannot reliably assemble it within a finite budget. So when a tool reports that a risk does not exist in your environment, the first question to ask is whether it examined everything, or only enough to reach a confident conclusion.