OpenAI's distillation disclosure: when the model becomes the decryption oracle

On September 30, OpenAI published Disrupting a coordinated model-distillation campaign. It is OpenAI's own account of a campaign to pull protected reasoning out of its models so that, by OpenAI's assessment, someone else could train on it. It is a security disclosure, not a model release, but labs rarely describe reasoning-trace extraction in this much detail.

What happened

OpenAI defines adversarial distillation as "the systematic and unauthorized use of one model's outputs or reasoning to help train, reproduce, or improve another model." The target was protected reasoning: the model's internal record of working through a task, which is kept out of the final answer.

The timeline:

• July 1: activity begins at low volume.
• July 24–25: spikes totalling 16,000 requests that used an extraction pattern, coming from more than 4,000 users. A footnote says these were attempts, not necessarily successful extractions.
• By July 28: OpenAI had fully disrupted a related cluster of more than 15,000 users.

OpenAI attributes a core cluster of the activity to individuals associated with Moonshot AI, the developer of Kimi. It also says it is unclear whether all the operators were a single actor.

The mechanism: no breach required

The key sentence describes what did not happen. The operators "did not break our encryption, compromise a database, or gain direct access to stored user conversations." Instead, they manipulated their conversations with the model until the reasoning came back in a form they could read.

One technique: copy encrypted reasoning out of one conversation, paste it into another, and ask the model to decrypt and transcribe it. Nobody cracked the encryption. The model decrypted it on request, acting as a decryption oracle. Independent researchers separately disclosed related cross-model and conversation-compaction vulnerabilities, which OpenAI confirmed.

What OpenAI changed

• Account enforcement: banned or restricted fraudulent accounts, tightened signup and infrastructure controls, and widened monitoring for related networks.
• Replay binding: strengthened protections for hidden reasoning "across users, workspaces, organizations, and model families." It also closed the path that let someone holding another user's encrypted reasoning replay it and recover the contents.
• Stream checks: added checks that catch streamed output that might expose reasoning and hold it back.
• Coordination: worked with the third-party services the activity passed through, and shared findings through the Frontier Model Forum and government information-sharing channels.

OpenAI says the work is unfinished: partner-hosted deployments need the same protections, and tool-output attacks need defenses that check more than visible text.

Why reasoning is the prize

Distill from final answers and a smaller model learns where to end up. Distill from reasoning traces and it learns how to get there: the intermediate steps, the dead ends, the corrections. In physics terms, that is the difference between a system's final state and its whole trajectory. The trajectory is a much richer training signal, which is why labs hide it. OpenAI's safety concern follows: a model trained on extracted reasoning gets the capability without the original's safeguards.

My take

The lesson is not specific to OpenAI. The post says the technique does not rely on a weakness unique to OpenAI's models, and that "systems that support portable or replayable reasoning artifacts may face related risks." Handing encrypted reasoning back to the client for the next turn is convenient for stateless deployments, but it makes the reasoning a bearer token: whoever holds it can use it. If anyone can hand it to a model that has the key, encrypting it protects nothing.

My rule for production agent systems is to treat these artifacts like session credentials. Bind each one to the user, tenant, and model that produced it, and reject it anywhere else. Treat compaction summaries, tool outputs, and streamed tokens as leak paths until shown otherwise. Checks on the output side matter because the attack works through the model's willingness to help.

I have two caveats. The volume figures count attempts, so we do not know how much reasoning actually got out. And OpenAI has not published the evidence behind its attribution. Even so, a write-up that names the mechanism, the timeline, and the fix is more useful than another general warning about distillation. I would expect other frontier labs to be auditing their own replay paths this week.