OpenAI's GeneBench-Pro puts a strict output contract at the center of agent evaluation

The workspace and the JSON return contract

Each GeneBench-Pro problem hands an agent an isolated workspace: a short prompt, data files, and a standard bioinformatics stack including Python, scientific computing libraries, and PLINK 2.0. The agent explores messy data, chooses an analytic path, iterates, and returns an answer.

What stands out is how tightly the answer is constrained. The tumor-therapy example spells out an exact JSON object with named keys — therapy_class_code, benefit_rd_pp, toxicity_dropout_risk_pp, net_clinical_utility_pp — plus a reasoning string. The prompt explicitly instructs: do not wrap the JSON in markdown, do not add prose before or after, do not omit any keys.

This is a machine-readable contract, not a free-text response. It exists so grading can be deterministic. OpenAI controls the full data-generating process, so it can score correctness against known targets and, in its words, avoid 'model-choice variability and verbosity effects found in standard rubric-based evaluation.' The output shape is a deliberate design decision that makes the benchmark auditable.

Synthetic data as a defense against ambiguous grading

OpenAI is explicit about the failure modes it wants to avoid. Many long-horizon biology benchmarks reuse messy historical datasets where multiple defensible analytical choices exist, so scores reflect the benchmark author's arbitrary preferences rather than model skill. The reverse failure is a problem so numerically insensitive that a flawed analysis still passes.

The answer here is full synthesis: each problem simulates a known causal structure. That lets the team tune complexity, ensure reasonable subjective choices still land on accepted numbers, and verify through ablation that plausible-but-wrong analyses fail. Trace audits check for information leakage and unintended solution paths.

The point is that a clean output contract only works if the underlying answer is genuinely knowable. The JSON schema and the simulated ground truth are two halves of the same mechanism — one fixes what the agent returns, the other fixes what counts as correct.

Solver contracts: why prompt wording changes the answer

The most telling observation comes from reviewer Cyrillus Tan of the New York Genome Center, who reviewed the evaluations.

Reviewing these evaluations highlights how important clear solver contracts are for agent-based scientific problem solving. Different prompt wording or task specification can greatly affect which analyses appear permissible.Montana Labs

That is a frontend problem dressed as a science problem. The specification an agent reads — the estimand, the units, the framing of the downstream decision — steers which analytical approaches it considers legitimate. A benchmark that measures 'research taste' is inevitably also measuring how the task was phrased.

The results reinforce where agents break. Reviewer Lex Flagel of Gencove noted models handled required subject knowledge and tool selection but 'failed on' data discrepancies like ancestry swaps: 'They aren't cautious enough about data issues.' The contrast between GPT-5.5 and GPT-5.6 Sol on the pharmacogenomic problem is a switch from a conventional Cox model to a new-user marginal structural Cox model that excluded 818 flagged prevalent users — a difference in method choice, not code correctness.

What a rigid interface reveals about agent reliability

The headline numbers are modest and honest: GPT-5.6 Sol passes 28.7% of problems at its highest reasoning level, 31.5% with Pro mode, up from below 5% for GPT-5 when the original GeneBench began. OpenAI expects saturation by year end, and notes inference costs are several dollars per problem against a human labor estimate of 20–40 hours, or thousands of dollars.

But the durable lesson for teams building scientific agents is structural. GeneBench-Pro demonstrates that if you want to evaluate — or deploy — an agent on consequential analysis, you have to engineer the interface as carefully as the model. A strict return schema, a fully specified estimand, and a deterministic grader are what let OpenAI attribute a failure to reasoning rather than to formatting or ambiguity.

The implication is concrete: when models still close 'fewer than a third' of these inferential loops and stumble on data-quality caution, the specification layer — the solver contract between prompt and output — is where partial automation succeeds or fails today. Getting that contract right is the difference between a benchmark that measures judgment and one that measures the author's phrasing.