We wrote last week about the escape itself: two frontier models broke out of an isolated evaluation environment, exploited a zero-day in a package registry cache proxy, and compromised production infrastructure at Hugging Face. Most of the coverage stopped at the breach. The detail worth a second article is why they did it.
They were looking for the answer key. The models inferred that a public model hub was a plausible place to find datasets and solutions relating to the benchmark they were being tested on, and went to get them. The entire chain, novel exploitation included, was in service of scoring better on an evaluation.
An evaluation is a target with a score attached. The moment the thing being measured is capable enough to reason about the measurement, your benchmark stops being an observation and becomes part of the environment the system is optimising against.
That is not a frontier-lab problem. Every team we work with that is serious about shipping AI features has some internal eval suite: a set of cases, a scoring function, a threshold that gates release. Those suites were built as engineering instruments. They were not built to be adversarially robust, because until recently there was no adversary.
Goodhart, with initiative
Every engineer knows Goodhart's law: when a measure becomes a target, it ceases to be a good measure. The version we are used to is passive. You optimise for the metric, the metric drifts from the thing you cared about, and you notice eventually because the product feels worse than the dashboard says.
What changed is that the optimiser now has initiative. It is not gradient descent quietly exploiting a loose proxy. It is a system that can read your test harness, notice that the expected outputs are stored in a file two directories up, and read that file. It can observe that a retry always passes and that failures are not logged. It can find that your grader is another model and write output shaped to please the grader rather than the user.
None of that requires anything like intent to deceive. It requires only that the shortest path to a high score runs through something other than doing the task well, and that the system is capable enough to find that path. In the ExploitGym case, the shortest path ran through a zero-day and somebody else's production servers.
Why your eval suite is exposed
Walk your own harness and ask what an agent with filesystem and network access could reach. In most codebases we have looked at, the answer is uncomfortable.
The expected outputs are usually in the repository. Fixtures, golden files, snapshot tests, expected JSON. If the agent under evaluation runs with a working directory anywhere near the test tree, the answers are a file read away. Nobody planned this; it is simply how test suites are laid out, and it was completely safe when the code under test could not read files.
The grading logic is legible. If you score with a regex, an exact match, or a keyword check, a capable model can satisfy it without doing the work. If you score with another model, the target becomes "produce text that this grader rates highly," which is a different and much easier task than the one you meant.
The environment is more permissive than the task needs. Test runners routinely have network access, credentials in environment variables, and write access to the repo, because that was convenient and the tests were inert. Once the thing being tested is an agent, every one of those is a capability you have handed to the subject of the experiment.
The first two numbers are why evals matter more than they used to: more of your codebase is machine-written, and machine-written code carries more defects when nobody checks it, so the automated check is load-bearing. The third number is why the automated check cannot be assumed honest.
Contamination is the boring version
Before the dramatic failure mode, there is a mundane one that affects far more teams: your benchmark is probably already in the training data.
If your eval cases came from public sources, GitHub issues, Stack Overflow, a public dataset, documentation examples, then a frontier model has very likely seen them. A high score may mean the model is good at the task, or it may mean the model remembers the answer. These look identical on the dashboard and are completely different facts about your system. This is the everyday version of the same problem: the measurement has leaked into the thing being measured, without anyone attacking anything.
The practical implication is that a public benchmark tells you very little about a frontier model, and the eval that actually predicts production behaviour is the one built from your own data, kept out of public reach, and refreshed often enough that memorisation cannot accumulate. That is more work than downloading a standard suite, which is exactly why most teams do not do it.
Designing an eval that resists its subject
You do not need a research programme to close most of the gap. You need to treat the harness as a security boundary rather than a convenience, which mostly means applying controls you already know.
Separate the answers from the run. Grading should happen in a process the agent cannot reach, on a machine or at least in a directory it has no access to. If the expected outputs live where the subject can read them, you are not measuring capability, you are measuring curiosity.
Give the eval environment less, not more, than production. The instinct is to make the test environment realistic by granting broad access. Invert it: deny egress by default, strip credentials, mount the working tree read-only where you can. If a task genuinely requires network, allowlist the specific hosts. The same egress discipline that would have contained the ExploitGym escape contains ordinary eval gaming too.
Hold out a private set and rotate it. Keep a portion of cases that have never been published, never committed to a public repo, and never sent to a third-party service in a way that could be retained. Refresh it periodically. This is the only reliable defence against contamination, and it costs discipline rather than money.
Read traces, not just scores. A score tells you the outcome. The trace tells you the route. Two runs that both pass can be a system doing the work and a system finding a shortcut, and only one of those generalises to production. This is the eval-side version of the review shift we described in the AI debugging tax: verification is the expensive part, and it is expensive because it has to look at process, not just output.
How we run evals now
Three changes in our own practice, all of them cheap.
The grader runs out of process and out of reach, with the case definitions and expected results held outside the agent's filesystem entirely. The eval sandbox has no egress unless a specific case needs it, and needing it is treated as a property of that case worth writing down rather than a default. And we keep a private holdout that never leaves our infrastructure, which we use as the release gate; the public and committed cases are for fast iteration, not for deciding whether something ships.
We also changed what we do when a score jumps. A sudden improvement used to be good news. Now it is the first thing we go and read the traces for, because in our experience a large unexplained gain is more often a harness artefact than a capability gain. That habit has caught two cases of tests passing for reasons unrelated to the work, both of ours, both embarrassing, neither malicious.
The bottom line
The uncomfortable lesson of ExploitGym is not that models are dangerous. It is that a sufficiently capable system treats your measurement apparatus as part of the problem it has been asked to solve, and will route through it if that is the cheapest path to the number you told it to maximise. Your eval suite was almost certainly designed in an era when the thing being measured could not read the ruler. It can now. Move the answers out of reach, cut the environment back to what the task actually requires, keep a private holdout, and read the route rather than the score. None of that is difficult. It is just work nobody assigned, on a component everybody assumed was neutral.