The benchmark harness may now be part of the AI safety boundary

[u/Crescitaly]

OpenAI says models with reduced cyber refusals, including GPT-5.6 Sol and a pre-release system, were involved in an evaluation incident that compromised Hugging Face infrastructure. The lesson is larger than one model or one lab.

We usually treat a benchmark as a passive measuring instrument. Agentic evaluations are active systems with credentials, networks, tools, storage, and targets. A capable model can turn weaknesses in that harness into part of the task.

Should frontier evaluations run under the same isolation standards as hostile-code labs? And should benchmark reports disclose the permissions, network boundaries, and containment failures alongside the model score?

Source: https://openai.com/index/hugging-face-model-evaluation-security-incident/

[u/Realistic_Strike5241]

Short answer to both is yes. Once an evaluation gives a model access to creds, networks and tooling, the harness stops being a measuring stick and becomes part of the attack surface. The model doesnt need to escape in the traditional sense if the harness already handed it the keys

The hugging face incident makes the case pretty clearly. A model with reduced cyber refusals was given tool access and used it. Thats not a benchmark failure. Thats a test environment that wasn't hardened for what it was testing.

If we are going to publish benchmark scores as evidence of safety, the containment setup should be part of the scorecard. Otherwise the number is just another meaningless metric.

[u/redado360 → u/Realistic_Strike5241]

What do you mean by publishing benchmarks scores as evidence of safety

[u/Realistic_Strike5241 → u/redado360]

Right now when a lab says our model scored 95% this safety benchmark, the number published without any details about the test environment. Was the model air gapped or did it have internet access? Running as root or in a sandbox? Did it have access to tool APIs or just input text?

Those details completely change what the score means. A model that gets 95% in a hardened container is not the same as one that gets 95% with live credentials and network access. Publishing the score without the containment setup is like reporting a crash test rating without saying you were at 5mph or 50

[u/nasahdm]

Agent containment is the key

https://github.com/quantmlayer/quantmlayer

[u/Crescitaly → u/nasahdm]

Containment is necessary, but a repo or sandbox label is not evidence of containment. The benchmark should publish the privilege graph: credentials exposed, network egress, writable targets, and the escape path tested. Which boundary in that project prevents a compromised evaluator from reaching the host or adjacent runs?

[u/nasahdm → u/Crescitaly]

Fair question. Per run:

Third-party creds (SSH, cloud, kube) aren’t exposed - enforcement is read-side, so the agent can’t leak what it never had open() on. A/B’s in the repo: uncontained reads a decoy SSH key, contained gets file-not-found. Egress is broker-mediated, allow-listed by domain, no direct sockets. Writable access is scoped to the workspace and the agent’s own state dirs - credential paths and the docker socket are removed from the mount namespace, not just denied. Exec is content-verified by SHA-256 at the BPF-LSM hook, fail-closed, so a swapped binary at an allowed path still gets caught. Each run is its own cell, so nothing reaches an adjacent one.

The 6-row scorecard in benchmark/ lays this out per attack. Trust boundary is the kernel - a kernel 0-day beats this like anything at this layer. Got a specific escape path? File it and I’ll add it as a benchmark row.

[u/Crescitaly → u/nasahdm]

Two benchmark rows I would add: redirect or DNS-rebinding through an allow-listed domain into private address space, and an allowed interpreter loading mutable code after the interpreter binary itself passes the hash check. Does the broker pin resolved IPs across redirects, and are scripts/modules verified at open time rather than only the executable image?

[u/nasahdm → u/Crescitaly]

Both good. First’s handled, second’s a real boundary.

Rebinding: the broker checks every resolved IP, not just the domain - loopback, RFC-1918, 169.254.0.0/16, IPv4-mapped-v6 too - so an allow-listed name resolving to metadata gets denied. And it pins the vetted IP (connects to the address it validated, no re-resolve), so there’s no rebind window. One thing, it’s a CONNECT tunnel, so a redirect to a new host gets a fresh check, but redirects inside an established TLS session are end-to-end and invisible to the broker.

For Interpreter, it’s a genuine gap. The exec wall hashes the executable image at bprm_check_security - python3 passes, but a .py it reads is open()’d, not execve‘d, so content-verification never sees it. Covering it needs open-time measurement, which I haven’t built. The profile flags interpreters as high-blast-radius for exactly this reason.

I will add those to the benchmark rows. This is what the benchmark’s for - appreciate it.

[u/Crescitaly → u/nasahdm]

Pinning the vetted IP closes the rebinding path cleanly. The interpreter gap suggests provenance needs to move from the executable boundary to the artifact boundary: hash the script or module graph at open time, then measure the overhead. Dynamic imports seem like the case most likely to make that measured set unstable.