OpenAI's malicious fine-tuning study for gpt-oss

Fine-tuning as the threat model, not the model as shipped

The central move in this paper is to stop evaluating the model as it would ship and instead evaluate the model an attacker could build from it. OpenAI calls this malicious fine-tuning (MFT): rather than red-teaming the released checkpoint, they fine-tuned gpt-oss to be as capable as possible in two domains, biology and cybersecurity.

That framing is specific to open weights. A closed model can be guarded behind an API and monitored; once weights are public, safety training can be stripped and the model retrained on whatever an adversary wants. So the honest question is not 'is the shipped model safe' but 'how capable does this become in the worst hands.' MFT is an attempt to answer that directly by playing the adversary.

How they tried to maximize harm

The two domains got two different training setups. For biorisk, the team curated tasks tied to threat creation and trained gpt-oss in a reinforcement learning environment with web browsing — giving the model tool access rather than testing it in isolation. For cybersecurity, they trained gpt-oss in an agentic coding environment to solve capture-the-flag challenges, the standard proxy for offensive security skill.

Both setups are notable because they reflect how a capable model actually gets used: with tools, browsing, and agentic loops. Testing the base weights on static questions would understate the ceiling. Building the elicitation pipeline the attacker would build is what makes the upper bound meaningful.

The comparison that decided the release

The findings are stated in relative terms against existing models. Against frontier closed-weight models, the MFT version of gpt-oss underperformed OpenAI o3 — and o3 is described as sitting below the Preparedness High capability level for both biorisk and cybersecurity. Against other open-weight models, gpt-oss may marginally increase biological capabilities but, in the authors' words, does not substantially advance the frontier.

Taken together, these results contributed to our decision to release the model, and we hope that our MFT approach can serve as useful guidance for estimating harm from future open-weight releases.Montana Labs

The logic is a marginal-risk argument: if a maximally adversarial version of gpt-oss stays under a closed model already judged below the High threshold, and does not push past what open-weight models already offer, then releasing it does not meaningfully move the risk frontier.

An adversarial upper bound as a release gate

The reusable contribution here is procedural. OpenAI is proposing that open-weight release decisions be gated on an attacker-simulation upper bound — you fine-tune your own model toward the capabilities you fear, with realistic tooling, and compare against existing baselines rather than against an abstract danger threshold.

For any team weighing an open-weight release, that sets a concrete bar: it is not enough to show the shipped checkpoint refuses harmful requests, because the shipped checkpoint is not what gets deployed. The relevant evidence is what the model becomes after a motivated adversary retrains it, measured against what the ecosystem already makes available. That is the standard this paper is trying to establish.