OpenAI Thwarts Massive Campaign to Steal AI Model Secrets
OpenAI has disrupted a massive campaign involving adversarial distillation aimed at stealing protected reasoning to train rival AI models.
OpenAI has announced the successful disruption of a highly coordinated campaign aimed at extracting "protected reasoning" from its artificial intelligence models. This sophisticated attempt was designed to harvest the internal logic used by OpenAI's systems to help competitors train, reproduce, or enhance their own AI models.
The company identified this activity as adversarial distillation—a practice where one model's outputs and internal reasoning are systematically and unauthorisedly used to develop a different model.
A Massive Surge in Extraction Requests
The investigation revealed that the operation began quietly on July 1, but the scale of the activity escalated rapidly. OpenAI reported a significant spike in activity during late July:
- July 24 and 25: The company detected 16,000 requests using specific extraction patterns.
- User Scale: These requests originated from more than 4,000 different users.
- Total Impact: Further investigation uncovered related prompt-pattern activity involving a cluster of over 15,000 users.
OpenAI confirmed that it had fully disrupted the campaign by July 28.
How the Attack Worked: Manipulating Reasoning
Crucially, the attackers did not breach OpenAI's encryption, compromise any databases, or gain direct access to stored user conversations. Instead, they employed a more subtle method of manipulation.
The operators manipulated interactions with the models to force the reproduction of protected reasoning. This refers to the model's internal "thought process" or record used to work through complex tasks. While this reasoning is typically withheld from the final answer provided to a user, extracting it allows others to see the underlying logic, making it easier to mimic the model's advanced capabilities.
OpenAI also noted a highly creative and novel attack method: operators would copy encrypted reasoning from one conversation and then ask a model in a completely different conversation to decrypt and transcribe that hidden information.
Attribution and National Security Risks
While OpenAI stated it is unclear if all the operators belonged to a single entity, it identified a core cluster of the activity as being associated with individuals linked to Moonshot AI, the developer of the Kimi model.
The company warned that adversarial distillation is not just a corporate issue but a significant safety and national security concern. The risks include:
- Bypassing Safeguards: Extracted reasoning can be used to train new models without the safety protocols and guardrails present in the original model.
- Dual-Use Risks: As AI models gain capabilities in sensitive "dual-use" areas, the ability to mimic them without safety constraints becomes increasingly dangerous.
Strengthening Defences Against AI Mimicry
To halt the campaign, OpenAI implemented a multi-layered response involving account enforcement, technical controls, and coordination with various partners. Key actions included:
- Banning or restricting fraudulent accounts.
- Strengthening infrastructure and sign-up controls.
- Expanding monitoring for suspicious networks.
- Closing a specific pathway that allowed users to "replay" and recover the contents of another user's encrypted reasoning.
OpenAI expects these types of adversarial distillation attempts to grow more sophisticated as frontier models evolve and bad actors seek cheaper ways to replicate high-level AI intelligence.