The replay flaw behind OpenAI's distillation campaign was reported in May — and called unreproducible

OpenAI says that in July 2026 a coordinated campaign of more than 15,000 users tried to pull its models' hidden reasoning out by copying encrypted reasoning from one conversation and asking a model in another conversation to decrypt it, and it attributes a core cluster of that activity to individuals associated with Moonshot AI, the developer of Kimi. The weakness the campaign leaned on — encrypted reasoning blocks that could be replayed across sessions, users and models — had been reported to OpenAI and Anthropic through bug-bounty channels in May, and was publicly demonstrated against OpenAI, Anthropic and Google in August. The lesson reaches beyond model theft: anything an AI provider hands back to clients "encrypted" is only as safe as what that ciphertext is bound to.

The short version

Modern "reasoning" models think through a problem step by step before answering. Providers hide that internal working — they call it chain-of-thought or protected reasoning — because it can contain things withheld from the final answer and because it is valuable training material for competitors. To let applications keep a multi-turn conversation going without exposing the reasoning, the APIs hand the client an encrypted copy of it, which the client sends back on the next turn.

On September 30, 2026 OpenAI said it had disrupted a campaign that abused exactly that mechanism. According to OpenAI, operators copied encrypted reasoning from one conversation and asked a model in a different conversation to "decrypt and transcribe" it. Nobody broke the encryption: the model itself was talked into reading out what the ciphertext contained. OpenAI calls this "adversarial distillation" — harvesting one model's outputs or reasoning to train a rival model — and says it has closed the pathway.

The part OpenAI's post does not dwell on is the history. A Johns Hopkins cryptographer, Matthew Green, wrote on May 29, 2026 that the same encrypted reasoning blocks could be replayed across API calls, sessions and accounts, and said OpenAI described his report as unreproducible. Academic researchers then turned the observation into a working attack against all three major providers and published it on August 10. The campaign OpenAI describes sits in between: it ran in July.

Timeline

• February 23, 2026 — Anthropic publishes "Detecting and preventing distillation attacks", attributing campaigns against Claude to DeepSeek, Moonshot AI and MiniMax; it says Moonshot's accounted for over 3.4 million exchanges. (The page shows a modification date of September 9, 2026.)
• May 29, 2026 — Matthew Green publishes "Let's talk about encrypted reasoning", reporting that OpenAI and Anthropic reasoning blobs could be replayed across sessions, accounts and models; he had reported it through bug-bounty programmes.
• Early July 2026 — Researchers from ELLIS Institute Tübingen, the Max Planck Institute for Intelligent Systems, Snyk and MATS run their experiments against Anthropic, OpenAI and Google APIs (the paper gives "early July 2026" as its testing period).
• First week of July 2026 — OpenAI's earliest observed activity from the campaign.
• July 22, 2026 — White House technology policy director Michael Kratsios publicly accuses Moonshot of large-scale distillation of US models, according to The New Stack and Implicator; neither outlet reports public evidence.
• July 24–25, 2026 — OpenAI sees "high-volume spikes" of 16,000 requests using an extraction pattern from more than 4,000 users.
• July 28, 2026 — OpenAI says it had "fully disrupted" related activity across a cluster of more than 15,000 users.
• August 10, 2026 — arXiv 2608.09867, "Stealing Reasoning Traces from Proprietary LLM APIs", is published; it states the providers mitigated after disclosure and the authors could no longer launch the same attacks.
• August 11, 2026 — Green adds an update to his post pointing to the paper.
• September 10, 2026 — Anthropic's September threat report lists distillation as one of seven harm areas it disrupted between December 2025 and August 2026.
• September 30, 2026 — OpenAI publishes "Disrupting a coordinated model distillation campaign".

What we know

From OpenAI's post (September 30, 2026). OpenAI says it "identified and disrupted a coordinated campaign designed to extract protected reasoning from our models, with the earliest observed activity occurring in the first week of July." (Some coverage gives July 1.) It defines adversarial distillation as "the systematic and unauthorized use of one model's outputs or reasoning to help train, reproduce, or improve another model," and says the operators attempted "to extract protected reasoning in novel ways, including by copying encrypted reasoning from one conversation and asking a model in another conversation to decrypt and transcribe the hidden reasoning content."

The numbers are OpenAI's: 16,000 requests from more than 4,000 users on July 24 and 25, and related "prompt-pattern activity across a cluster of more than 15,000 users, which we fully disrupted by July 28." A footnote says these describe "attempted, not necessarily successful, extractions." The post gives no success rate, does not name the models or products targeted, and does not say whether the accounts were ChatGPT or API customers.

On attribution, OpenAI writes: "It is unclear whether all operators we observed during the relevant time period originated from a single actor. However, we attribute a core cluster of the activity to individuals associated with Moonshot AI, the developer of Kimi." CyberScoop and The Hacker News both note that the post offers no technical evidence for the attribution. CyberScoop says it asked Moonshot AI for comment; we found no public response from Moonshot in any outlet we checked as of October 5, and a further search on October 12 turned up none.

OpenAI is explicit about what did not happen: "The operators did not break our encryption, compromise a database, or gain direct access to stored user conversations." It says it banned or restricted fraudulent accounts, "strengthened signup and infrastructure controls, and expanded monitoring for related networks," "closed a pathway that allowed someone who already possessed another user's encrypted reasoning to replay it and recover its contents," added "checks to detect and hold streamed output that might expose reasoning," and "strengthened protections for hidden reasoning across users, workspaces, organizations, and model families." It adds: "This manipulation is not a vulnerability unique to OpenAI's models," and says it shared findings through the Frontier Model Forum and "appropriate government information-sharing channels." It also credits outside work: "Independent security researchers also brought related cross-model and conversation-compaction vulnerabilities to our attention through responsible disclosure," linking the August paper. The post does not say when the replay pathway was closed.

From Matthew Green (May 29, 2026, updated August 11). Green tested OpenAI and Anthropic reasoning blobs and found that unmodified blocks could be replayed across sessions, accounts and models without error — consistent, he wrote, with a single key protecting all of them. By his account, OpenAI called the report "unreproducible" and Anthropic saw "no security implications in side channels or replays." He also described size and timing side channels in the blobs.

From the August paper (arXiv 2608.09867). The authors (Panfilov, Schmotz, Shumailov, Beurer-Kellner, Schaeffer, Prabhu, Geiping, Andriushchenko) assume "a standard, unprivileged API adversary" with no insider access. Their core finding, as summarised by the Cloud Security Alliance's research note of August 12: encrypted reasoning blocks from Anthropic, OpenAI and Google were "interchangeable across sessions, users, and models." Their attack takes a block produced by a strong model and replays it to a weaker sibling model in the same family, which acts as "an unwitting decryption oracle." The paper names the weak models used: Claude Haiku 4.5 (receiving Opus 4.8 blocks), GPT-5.6 Luna, and Gemini Robotics 1.6. It cites Green's May post as the original disclosure of interchangeability, and states: "All model providers acknowledged the receipt of our report and subsequently we were unable to launch the same attacks."

The paper also went looking for exposed reasoning in the wild. Per the CSA note, from 6,708 public GitHub and Hugging Face transcripts the authors recovered 315,320 reasoning blocks, and in them "367 personally identifiable information artifacts and 182 credentials, including 62 API keys and 33 passwords" — much of it present only inside the encrypted reasoning, never in the visible chat.

Context on Moonshot. Anthropic's February post attributed a distillation campaign against Claude to Moonshot "with high confidence", citing IP correlation, request metadata and infrastructure indicators. Press reports on September 10 described a further Anthropic allegation that Moonshot routed Kimi user requests through Claude; we could not read Anthropic's own text for that case, so we do not repeat its figures here. Kratsios's July 22 accusation is known to us only through press coverage of his post.

Technical analysis

The root cause is a binding failure, not a cryptographic one. Encrypted reasoning is a capability token: whoever holds the blob can ask the provider's infrastructure to put it back into a model's context. If the token is bound only to "this model family" rather than to a specific user, conversation and model, then any party who obtains a blob — by scraping a public transcript, from a shared log, or by generating it themselves — can feed it to any model in that family. Encryption kept the blob opaque to the client; it never constrained where the blob could be used. This is the same class of mistake as a session cookie that is valid for every user, or a signed URL with no audience.

The model is the decryption oracle. Once a valid blob is in a model's context, the plaintext reasoning is, functionally, part of what the model "remembers". Asking the model to repeat it is a prompt, and prompts are policed by the model's own safeguards. A smaller, cheaper sibling with weaker refusal behaviour is the obvious target — which is the paper's cross-model attack. OpenAI's description of the campaign ("copying encrypted reasoning from one conversation and asking a model in another conversation to decrypt and transcribe") is the same shape.

Our assessment: OpenAI does not say the campaign used the exact technique Green or the Tübingen-led team described, and we do not claim it did. But the mechanism OpenAI describes and the flaw the researchers published are, conceptually, one and the same: reasoning state that could leave its original conversation and be read back elsewhere. What is notable is the sequence. The replay behaviour was reported in late May; by Green's account it was dismissed; the campaign ran through July; the providers' mitigations are reported as in place by August 10. Whether OpenAI closed the pathway before, during or after the July activity, its post does not say.

Why controls failed. Three layers were in play. Account and signup controls did not stop thousands of coordinated accounts appearing — OpenAI's own remedies start with "strengthened signup and infrastructure controls". Model-level refusal was the only thing standing between a replayed blob and its plaintext, and refusal is probabilistic. And the architectural control that would have made the whole class moot — binding each blob to its user, session and model — was, per the paper, missing at all three providers. OpenAI's new "checks to detect and hold streamed output that might expose reasoning" is a fourth, output-side layer: inspect what the model is about to stream and stop it if it looks like hidden reasoning.

What is novel about the AI angle. Classic model extraction queries a model and trains on its answers. This is extraction of state the provider deliberately withheld, achieved by making the model betray its own encrypted context. It also shows that "hidden" reasoning has two value profiles: to a rival lab it is training signal; to everyone else, the paper's credential and PII findings show it can be a leak channel for secrets that users and agents put into prompts and never saw echoed back.

The attribution question. OpenAI is careful: a "core cluster", "individuals associated with" Moonshot, and an admission that not all operators may be one actor. That is weaker than Anthropic's "high confidence" wording in February and comes with no published indicators. Our assessment: readers should treat it as a company's attribution, consistent with prior claims by Anthropic and a US official, but not independently verifiable from what has been published.

What remains unclear

• When the pathway was closed. OpenAI gives the disruption date of the accounts (July 28) but no date for the replay fix. The paper says mitigations came after its disclosure; Green says his May report was called unreproducible. Whether the campaign's July activity succeeded against an unfixed system is not stated.
• How much was extracted. OpenAI's figures are attempts. There is no success rate and no statement on whether extracted reasoning was used to train any model.
• Which models and products. Not named — ChatGPT, API, or both; which reasoning models.
• Evidence for attribution. None published; Moonshot has not, to our knowledge, responded publicly.
• The "conversation-compaction vulnerabilities." OpenAI mentions these alongside cross-model issues but does not describe them.
• Other providers' state. OpenAI says the manipulation is "not a vulnerability unique to OpenAI's models". The paper reports mitigations at Anthropic and Google too, but neither has published a detailed account of the binding now in place, and the CSA note says the underlying architectural problem remains unresolved industry-wide.
• Overlap between the campaigns. OpenAI's July activity, Anthropic's distillation cases and the White House claim all point at Moonshot; whether they describe one operation, several, or overlapping infrastructure is not public.

Lessons and what to do

For security teams using AI APIs and agents

• Treat encrypted reasoning fields in logs, agent trajectories, evaluation outputs and shared transcripts as sensitive data — the CSA recommends handling them "as though it were plaintext regardless of its encrypted appearance."
• Strip reasoning fields before publishing transcripts, bug reports or datasets, and extend secret scanning to agent trajectory exports.
• If agent transcripts containing reasoning blocks were ever pushed to public repositories, rotate any credentials that could have passed through those sessions.
• Ask your AI vendors directly what replay protection is in place: is reasoning state bound to your organisation, user, session and model? Put the answer in vendor-risk records.

For AI builders and platform teams

• Any opaque token you return to clients — encrypted reasoning, cached context, tool state, conversation summaries — needs an audience: bind it cryptographically to the tenant, conversation and model that produced it, and reject it anywhere else.
• Do not rely on a model's refusal to protect data the model can see. If plaintext must never leave, it must never re-enter a context the caller controls.
• Take "unreproducible" bug reports about cryptographic design seriously; replay is a design property, not a flaky bug.
• Add output-side checks for content that should never be streamed, as OpenAI now does.

For leadership

• Model distillation is now an openly named threat category at OpenAI and Anthropic, and a US policy issue. If your organisation builds or fine-tunes models, its reasoning outputs and training data are assets with the same theft profile.
• If you procure foreign or open-weight models, follow the attribution debate — but recognise that, as of today, the public record contains companies' and officials' claims, not published evidence.

Sources

• OpenAI, "Disrupting a coordinated model distillation campaign", September 30, 2026.
• Panfilov et al., "Stealing Reasoning Traces from Proprietary LLM APIs", arXiv 2608.09867, August 10, 2026.
• Matthew Green, "Let's talk about encrypted reasoning", A Few Thoughts on Cryptographic Engineering, May 29, 2026 (updated August 11).
• Anthropic, "Detecting and preventing distillation attacks", February 23, 2026.
• Anthropic, "Detecting and countering misuse of AI: September 2026".
• Cloud Security Alliance Labs, research note on reasoning trace theft, August 12, 2026.
• Cloud Security Alliance Labs, "Reasoning Trace Theft: A Shared Flaw Across AI Vendors", August 14, 2026.

Coverage:

• CyberScoop.
• The Hacker News, October 1, 2026.
• Implicator, on the August paper and vendor responses.
• The New Stack, on the July 22 White House accusation.