OpenAI's evals primer: a Specify-Measure-Improve loop for business automation

Frontier evals ship models; contextual evals run workflows

The most useful distinction in OpenAI's primer is one it draws about its own practice. OpenAI says its researchers use rigorous frontier evals to measure how models perform across domains, because at OpenAI "our models are our products." But it concedes those frontier evals "cannot reveal all the nuances required to ensure the model will perform on a specific workflow in a specific business setting."

That admission is the whole argument. The company reports that its internal teams have built "dozens of contextual evals" for specific products and internal workflows, and it tells business leaders they need to do the same. The benchmark that qualifies a model as state-of-the-art is not the benchmark that tells you whether it converts your inbound emails into demos.

OpenAI is also candid that this is unsettled territory: contextual evals are "an active area of development and definitive processes have yet to emerge." The primer offers a framework it has "seen work across many situations" rather than a finished methodology.

The loop: specify a golden set, measure against edge cases, improve on a flywheel

The framework is three steps: Specify, Measure, Improve. Specify begins with a small cross-functional team writing the system's purpose in plain terms — the example given is "Convert qualified inbound emails into scheduled demos while staying on brand" — then mapping dozens of example inputs to desired outputs. OpenAI calls this a "golden set," described as a living reference of expert judgment about what great looks like.

A practical detail stands out here: the primer recommends reviewing 50 to 100 outputs from an early system version to produce a taxonomy of errors and their frequencies. That is a concrete starting scale, and it reframes evaluation as error analysis rather than scorekeeping.

Measure calls for a test environment that mirrors real conditions — "not just a demo or prompt playground" — including rare-but-costly edge cases. LLM graders can scale the work, but OpenAI insists a domain expert "regularly audit LLM graders for accuracy" and directly review behavior logs. Improve builds a data flywheel: log inputs, outputs and outcomes, route ambiguous or costly cases to expert review, and feed those judgments back into prompts, tools, or models.

The claim that the flywheel is a defensible asset

OpenAI's economic pitch is that running this loop at scale "yields a large, differentiated, context-specific dataset that is hard to copy." The argument is that in a world where models and expertise are broadly available, advantage shifts to how well your systems "execute inside your context."

In a world where information is freely available across the world and expertise is democratized, your advantage hinges on how well your systems can execute inside your context.Montana Labs

This is a notable position from a model vendor: it locates durable value not in the model but in the accumulated evaluation data and institutional judgment a customer builds around it. The primer positions evals as the successor to OKRs and KPIs — "measuring what matters" for probabilistic systems.

What automating a standard operating procedure now requires

For teams automating workflows, the primer's operative implication is that specification is the bottleneck, not model capability. OpenAI states plainly: "If you cannot define what 'great' means for your use case, you're unlikely to achieve it." It frames this as "management skills are AI skills" — clear goals, direct feedback, and judgment about when precision matters versus when velocity does.

Two caveats in the text deserve attention before anyone treats evals as a launch gate. OpenAI warns that rubrics can "over-emphasize superficial items at the expense of your overall goals," and that some qualities are "difficult or impossible to measure." It also says evals do not replace A/B tests for external deployments — they complement traditional experimentation rather than substitute for it.

The honest takeaway is that OpenAI is telling businesses the hard part of automation is not calling the API. It is the cross-functional, iterative, "messy" work of defining desired outcomes and maintaining that definition as models and goals evolve. The eval is where that work either exists or doesn't.