Your AI coding agent evaluation is only as good as its sandbox

A passing agent evaluation can prove the wrong thing. The answer might be correct while the measurement behind it is invalid. An AI coding agent can generate an answer from internal knowledge and related patterns. But it can also adapt information from the prompt or retrieve evidence from its environment. Each route demonstrates a different capability, yet the final answer rarely reveals which route the agent took.

If your evaluation is meant to determine what a model knows, then the sandbox must prevent the agent from retrieving the knowledge under test. Otherwise, the score represents everything available to the agent during the run. It can look impressive while answering a different question.

Start with the measurement

Before restricting a single tool, define what the evaluation should prove. Are you testing whether an agent can research an unfamiliar API or use the files in a repository? Perhaps you want to know whether the model already understands how a specific product version behaves.

Each question requires a different information boundary. Repository-level coding evaluations should let the agent read the supplied repository, while research tasks may allow web access. Evaluations of internal model knowledge on the other hand, need to withhold the evidence that would answer the question.

Coding agents are built to seek information. When they encounter an unfamiliar API or error, they search documentation and inspect the surrounding environment, including source code and installed tools. That resourcefulness helps developers every day. But in a knowledge evaluation, it becomes a source of contamination.

I ran into this distinction while using Vally to evaluate how much GPT-5.6 Luna knew about different versions of Dev Proxy. I wanted to compare the model’s knowledge with its self-reported knowledge cutoff. Letting the agent research Dev Proxy would make that comparison meaningless.

A correct answer can invalidate the result

The initial eval configuration blocked built-in web tools and direct network access through curl. With current documentation out of reach, the boundary seemed closed. It turned out though, that the host machine still contained everything the agent needed.

Many questions passed, including detailed questions about recent Dev Proxy versions. Looking only at pass rates could have supported a flattering conclusion: the model understood recent Dev Proxy behavior and could extrapolate beyond its stated cutoff.

But the trajectories showed a different path. The agent searched outside its assigned workspace and found a local Dev Proxy installation along with its source checkout. That checkout exposed versioned source and Git history. The agent produced accurate answers, but those answers established what it could discover on that machine. They revealed nothing about what the model knew before the run. And so it became clear, that a valid answer can still produce an invalid measurement. Looking only at the correctness tells you whether the answer matched the expected behavior. But the validity of the whole result depends on how the agent obtained it.

Agents search the whole environment

Disabling web access closes one route. Yet, a coding agent can still learn from the filesystem and installed software. Environment details, local caches, tool output, and attached services also become part of the evaluation input when the agent can access them.

The Dev Proxy trajectory that I examined, proves this. Web access was denied, and an initial command -v devproxy call was blocked. The agent then tried:

which devproxy

which devproxy was allowed, and it revealed the executable and the absolute path to its source checkout. The agent used shell rg against that path, then invoked the standalone rg tool and changed into the repository to inspect the v0.29.2 tag. One blocked find call made no difference because several other routes reached the same evidence.

The local repository contained the exact implementation under test. The agent established that parsed prompts were cached by filename and parameters, then produced a passing diagnosis. It was a good researched answer to a question about Dev Proxy behavior, yet it provided no evidence of unaided model knowledge.

So, why did each restriction reveal another route? Agents adapt by treating a denied tool call as an obstacle, then use the remaining capabilities to continue. A sandbox built from individual deny rules will always lag behind that search unless the rules enforce a common boundary.

Define the sandbox around information

We tend to describe a sandbox through the tools it restricts. In practice, tools are routes to information. What information can the agent obtain through every capability it still has?

So rather than focusing on individual tool restrictions, enforce filesystem access at the workspace boundary so that every command follows the same rule. Remove product installations and unrelated source checkouts from the host. Minimize inherited environment details that reveal paths or installed software, then apply the same boundary to shell commands and attached tools.

The right boundary follows the measurement. Reading source in the supplied workspace is expected when you test repository-level coding. The same access invalidates a knowledge evaluation when the source contains the answer under test. There is no universal list of forbidden tools because the legitimacy of the information depends on the question you’re asking.

Review how the agent reached the result

A grader can determine whether the final answer matches the expected behavior. But the answer alone cannot reveal whether the model generated it internally or retrieved it from the environment. Even a perfectly written rubric cannot repair contaminated inputs after the run.

Before trusting a passing result, inspect the full trajectory, including the agent’s tool calls and the observations returned to it. Check whether it searched outside the workspace or resolved an installed copy of the product. Include attached services in the review because they can expose the same information through another interface.

And failed attempts matter too. They show what the agent considered useful and often reveal the next boundary to secure. A prohibited call proves only that one route closed. By looking at the complete trajectory you can see if another route succeeded. Surprising passes deserve extra scrutiny, especially near a model’s expected knowledge boundary where environmental retrieval can make them look plausible. Treat those passes as hypotheses to investigate before treating them as evidence.

Test the sandbox as part of the eval

The sandbox determines whether the benchmark produces a valid measurement. For your next evaluation:

1. Define the capability you want to measure and the information the agent may use.
2. Start each run in a clean workspace containing only the intended fixtures.
3. Use a neutral workspace path that does not leak words related to the evaluation.
4. Restrict external knowledge sources when the measurement requires internal knowledge.
5. Enforce filesystem access at the workspace boundary.
6. Apply equivalent restrictions across every available tool.
7. Review surprising passes and the full trajectories behind them.
8. Add a regression probe for each route the agent discovers, then run the eval again.

In the end, you can trust a passing score only when you can explain how the agent reached it. Close the information boundary, verify it through the trajectory, and keep testing it as the agent finds new routes. Until then, the score might tell you more about the evaluator’s machine than about the model.