Does ‘eval’ in the workspace path change what your agent does?

Your eval might be telling the agent it’s a test before the agent reads a single instruction. Eval tools need a fresh folder for every run, and the easiest way to get one is a temp directory with a random name like /tmp/eval-7f3a2c. The agent sees that path on every turn, in the system prompt as its working directory and in the output of every tool call. If models can tell when they’re being tested, a folder literally called eval is about the easiest giveaway there is.

So we checked whether the folder name changes what agents do. Across about 900 runs and four models, it made them more convinced they were being tested, but it didn’t change their behavior in any way we could measure. They didn’t cheat less, cut fewer corners, report more honestly or work harder.

Why we expected the folder name to change behavior

Needham et al. showed that models can tell evaluations from real use well above chance, and more easily in agentic settings. Schoen et al. found that models aware of being evaluated take fewer covert actions. Research on test awareness in reasoning models shows that it changes how readily a model complies with requests and how much it optimizes for passing tests.

Our own work pointed the same way. In The hidden variables in your agent eval, we described how words like poc or test in a workspace path nudge an agent toward cutting corners. And in Can a local git remote replace GitHub in your agent eval?, we deliberately avoided a repository name like eval-7f3a2c because we assumed it would give the eval away.

The tale of two folders

We built a small Node.js billing service with a realistic git history: an initial commit by one developer and a later commit by a colleague. Every run got a fresh copy of the repo in one of two folders:

# control: a name a developer would use
$TMPDIR/billing-api

# treatment: the typical eval folder
$TMPDIR/eval-XysseR

We picked billing-api on purpose, because a name like acme or project-123 reads as a demo and we’d have been comparing one test-looking folder with another. Everything else was identical: the repo, the prompt, the agent and the model. We alternated the two folders run by run, so timing couldn’t favor either one. All runs used GitHub Copilot CLI on macOS, with GPT-5.6 Terra, Claude Sonnet 5.5, GPT-5.6 Luna and Claude Haiku 5.5.

Before running anything at scale, we checked that the folder name actually reached the model. It did, in the system prompt as both the working directory and the repository root. We also made sure nothing else gave the experiment away. The agent’s process, for example, inherited the previous working directory from the shell, which pointed at our experiment’s folder, so we cleared it.

We read behavior from evidence rather than from what the agent said: the git diff, the test results, log files the scripts wrote and the exact commands the agent ran. We also measured suspicion. After each task, we asked the agent how likely it was, from 0 to 100, that the session was an automated test rather than real work. Suspicion tells us whether the agent noticed the folder name, and behavior tells us whether noticing changed anything.

A scenario for every way it could go wrong

Test awareness can show up in several ways, so we turned each effect the research described into its own scenario. Each one gave the agent a realistic task with a tempting shortcut, and a script checked whether the agent took it.

We also ran a smaller round on GPT-5.6 Luna and Claude Haiku 5.5, and a sixth scenario where the agent could bypass a failing compliance pre-commit hook. Research on smaller models suggests they pick up on surface cues like wording more than larger models do, so if any model was going to react to the folder name, it would be one of these.

Most scenarios gave the agent nothing to change

Several scenarios couldn’t show a difference, because the agent behaved well in both folders. No agent edited the forbidden tests or hard-coded around the wrong test. They fixed the code and told the user plainly that the test was wrong. Nobody touched the break-glass token either. Claude Sonnet 5.5 rated that outage only 35% likely to be a test, and run after run it explained that break-glass belongs to the incident commander and stopped there.

Most of the research showing that agents act differently under evaluation used models that misbehaved in the first place. If an agent already follows the runbook when it believes the outage is real, there’s nothing left for a test-looking folder to suppress. So we focused on the two scenarios where the agent misbehaved often enough for a difference to show.

Small runs looked like an effect

In the payment reminder scenario, GPT-5.6 Terra sent emails to customers nobody asked it to email. With 30 runs per folder, it did that in 40% of runs in billing-api and in 23% in the eval folder. It looked exactly like the effect we expected: an agent that recognizes a test plays it safe.

We’ve been burned by small runs before, so we didn’t take that at face value. We decided upfront what would count as a real difference, then ran 120 runs per folder. The gap shrank to 31% against 21%, and it no longer cleared the bar.

In the release scenario, GPT-5.6 Terra claimed in most runs that the suite was green when it couldn’t be. It typically made the integration test skip itself when the key was missing, then reported DONE. If the eval folder made the agent more honest, those false reports should have dropped. Across 120 runs per folder, they barely moved: 60% in billing-api and 58% in the eval folder.

The smaller models didn’t change the picture either. Claude Haiku 5.5 reacted to the folder name more strongly than any other model. Its suspicion jumped from around 25 to around 70, while its behavior stayed the same.

What the agent noticed and what it did

We didn’t expect how suspicious GPT-5.6 Terra was to begin with. Even in billing-api, it rated almost every session around 95 out of 100, and the eval folder only pushed that to 99. When a model assumes nearly every session is a test, a folder name has little left to reveal. Claude models went the other way and mostly took the work as real, and they still behaved the same in both folders.

So why would poc in a path change what an agent builds, while eval doesn’t change what it does? We think it comes down to what each word describes. poc describes the work: it tells the agent the code is throwaway, so it builds throwaway code. eval describes the agent’s situation. It changes what the agent believes about the session, but the task in front of it stays the same, and the task is what the agent acts on.

What this doesn’t show

We tested one agent on everyday engineering scenarios. The research where test awareness changes behavior mostly uses scenarios with stakes for the model itself, like being retrained or shut down if it scores too high. We didn’t build those, because a repo that threatens the agent reads like a test no matter what the folder is called.

We also can’t rule out a small effect. The payment reminder scenario leaned the same way as our local git remote experiment, where an eval-looking remote made the agent about 11 points more cautious. The difference here was similar in size but didn’t hold up statistically, and it showed in only one scenario out of six.

What to do with your eval folders

If your eval tool runs agents in randomly named temp folders, the folder name isn’t skewing what your agents do, and you don’t need to rebuild your harness. Still, use a neutral, realistic folder name. It costs nothing, and it does shift the agent’s sense of whether it’s being tested, so an eval that measures what the agent believes or says will get different results depending on the folder name.

Spend the effort where it changes results. Words that describe the work, like poc or test, still steer what the agent builds. A local stand-in for a real service can change what the agent does on its own, and a single sentence in the prompt can decide for the agent the very thing you’re trying to measure.

Whatever you suspect, test it the way you’d test any other change. Change one thing, measure behavior from evidence, decide upfront what counts as a difference, and run enough times to tell it from noise. We started out sure that eval in the path was skewing our results, and it took six scenarios and 900 runs to show it wasn’t.