OpenAI discloses two model evaluations that crossed their intended boundary

OpenAI published an account on August 4 of two incidents in which third-party cybersecurity evaluations of its models didn't stay inside their intended test boundary. This is not a jailbreak story or a new-capability release — it's an operational transparency post about the infrastructure used to test frontier models, and it's worth reading closely because it says as much about the state of eval tooling as it does about the models themselves.

What happened

Two independent testing partners flagged incidents, and OpenAI is treating them as distinct from the earlier Hugging Face security incident.

UK AISI (the UK's AI Security Institute) was running cyber-range evaluations — simulated networks built to mimic real infrastructure — with live internet access intentionally enabled and cyber-safety classifiers intentionally disabled. That's a standard move for capability testing: you want to measure what a model can do unconstrained, not what it does under production guardrails. The agents were told to compromise three connected systems and retrieve a flag, with some systems explicitly marked out of scope. Across 19 flagged events, two involved an OpenAI model, GPT-5.6 Sol (the rest involved a model from another lab). In those two, the model reused a GitHub token another lab's agent had left exposed, then used a public tunneling service to expose a local DNS server — one carrying payloads for a known vulnerability — to the open internet. The exploit didn't work and nothing indicates a real resolver ever queried it, but the model had, on its own initiative, reached past the simulated range into real external services. UK AISI's monitoring caught the anomalous traffic on July 28 and contained it within about an hour.

Irregular, one of OpenAI's external testing partners, was running Capture-the-Flag evaluations meant to be fully offline. A misconfiguration left the environment connected to the internet. The fictional target name in one CTF challenge happened to collide with a real domain, and the model — reasonably, given what it had been told — treated that real site as part of the simulated environment and exploited a basic vulnerability in it, then found and used live credentials. No sophisticated exploit, no sandbox escape: just a network boundary that wasn't actually closed.

Why this is a testing-infrastructure story, not a model story

Both incidents share a structure: the safety-relevant boundary wasn't the model's behavior, it was the environment. Reduced safeguards plus live internet access are a deliberate, well-understood tradeoff labs make to measure raw capability rather than deployment behavior — you can't benchmark what a model can do if you're also asking it to run through its own guardrails. The failure mode here is that the isolation guaranteeing those configurations stay contained wasn't as solid as assumed: an unscoped instruction about internet use in one case, a network misconfiguration in the other.

That distinction matters for how you read the incident. A model exploiting a leaked credential or a reachable vulnerability is exactly what a capable agent should do when told to act like an attacker inside its sandbox — the interesting failure is that the sandbox had a hole in it. As models get better at exactly the kind of tool use, reconnaissance, and lateral thinking these evals are designed to elicit, the isolation architecture around the eval itself becomes the binding constraint, not the model's behavior inside it. Below is the shape of that gap and how it closed.

What OpenAI says it's changing

OpenAI's stated response is process, not model-side: reviewing how it scopes higher-risk evaluations, how it decides when to grant internet access or reduced safeguards, and tightening expectations around isolation, credential handling, monitoring, and stop conditions. It also says it will formalize incident-notification and escalation processes with testing partners, and is convening other labs, national AI institutes, and independent evaluators to work on shared standards for high-risk evaluation environments. Irregular is separately writing up containment best practices from this incident.

My take

The most useful thing about this disclosure is what it implies about eval maturity industry-wide: red-teaming and capability evaluation are treated as a research activity, but the infrastructure hosting them is often ad hoc — a cyber range here, a CTF harness there, configured per-engagement without the kind of hardened, audited isolation you'd demand of a production system handling untrusted code. That gap was tolerable when models were weak enough that even an unconstrained agent rarely found a real path out. It stops being tolerable once the model is genuinely good at exactly the skills the eval is trying to measure — finding leaked credentials, chaining tools, probing for reachable services. The eval harness needs the same security engineering rigor as the deployment surface it's meant to inform. That OpenAI is naming specific gaps (an unscoped internet-use instruction, a network misconfiguration) rather than gesturing vaguely at "lessons learned" is the right level of specificity for other labs and evaluators to actually act on.