Jul, 134 Min. Lesezeit
Frontend
News · OpenAI's LifeSciBench grades free-response science answers against 19,020 expert rubric criteria
Jul, 84 min to read
FrontendOpenAI's LifeSciBench grades free-response science answers against 19,020 expert rubric criteria
The benchmark drops clean multiple-choice formats for open-ended collaborator requests, then scores them line by line — and the results expose where frontier models still stall.
The request replaces the question
Most life science evaluations, as OpenAI describes them, reduce research to fact-recall or clean prediction problems with structured formats and tidy reference answers. LifeSciBench discards that shape. Each of its 750 tasks is built to look like something a scientist would actually say to a capable colleague: a scientific prompt, any relevant context or attached artifacts, and a free-response answer.
The published example makes the format concrete. A team preparing for a Type B FDA meeting asks for a 'hard-nosed critique' of whether their AAV9 micro-dystrophin package supports accelerated approval on a surrogate endpoint. The prompt supplies biopsy Western blot numbers, immunofluorescence readings, a 48-week NSAA change, safety events, and eligibility criteria, then asks the model to pressure-test the package item by item. There is no checkbox answer — the output is expected to read like a reviewer's memo.
This is a deliberate interface decision. By framing the input as a working request rather than a quiz item, LifeSciBench measures whether a model can operate inside the messy front-end of research — incomplete evidence, conflicting results, decisions under uncertainty — instead of the sanitized version benchmarks usually test.
Grading that mirrors how scientists judge each other
The other half of this design is the rubric. Across the benchmark, expert-developed rubrics carry 19,020 criteria — an average of 25 per task. The DMD example is scored against six weighted criteria totaling 100 points, with 24 points for catching the assay-quantification problems (MANEX1A epitope sharing, invalid full-length dystrophin standards), 22 for explaining why expression level is not automatically a valid clinical surrogate, and only 8 for noting patient-selection and sample-size gaps.
OpenAI's stated reason is that a correct high-level conclusion can still be judged incomplete — if a response overlooks a key assay limitation or fails to raise a consequential biological nuance. Conversely, a partial answer can contain high-quality reasoning without solving the task. The benchmark reports two metrics to capture this: pass rate, the share of tasks clearing a 70% task-level threshold, and score, the average rubric reward that grants partial credit.
The validation numbers suggest the format holds up. Feedback came from 453 reviewers not involved in writing tasks; 97% held a doctorate, with an average of 12 years of experience and 14 peer-reviewed publications. Agreement exceeded 96% in every category, with 90.4% strongly agreeing that tasks reflect real-world work.
The artifact gap
The most telling result sits at the seam between the prompt and its attachments. LifeSciBench includes 1,062 attached artifacts — figures, PDFs, tables, sequence files, structure and chemical files, web references — and 53% of tasks require interpreting or synthesizing at least one of them.
Performance falls off sharply when models must read those artifacts rather than the prompt text alone. GPT-Rosalind drops from 45.1% on text-only tasks to 28.1% when artifacts or URLs are involved; GPT-5.5 shows the same shape, 29.9% down to 21.9%. OpenAI attributes the gap to models struggling to extract information from complex figures or large sequence files and fold it into the final answer.
Exact-output formats are worse still. GPT-Rosalind reaches only 14.8% on numeric tasks and 24.0% on sequence or structure outputs, with construct-generation tasks at 27.3% and barely improved over GPT-5.5. These are the outputs that would be used directly — CRISPR/HDR donor design, siRNA design — where a small formatting or calculation slip drops a response below threshold.
Partial credit is not a usable deliverable
GPT-Rosalind lifts overall exact pass rate from 25.7% to 36.1%, with its strongest gains in scientific communication (56.3% to 71.1%, though n=9) and translation (36.8% to 57.7%). Those are the categories with a clear evidence boundary that reward organizing and explaining. Design, optimization, and prediction (30.7%) and analysis (30.3%) remain the hardest.
The specific implication of this benchmark's rubric-plus-pass-rate design is what it exposes about the space between the two scores. In roughly 14% of tasks, models earned substantial rubric credit while failing the exact-pass threshold; for GPT-Rosalind, 109 tasks scored under 20% pass rate yet earned at least 50% rubric reward. A model can surface the right evidence and write a plausible partial answer, then miss a single constraint, use the wrong data, or fail to connect its reasoning to a usable decision.
For anyone building on these systems, that is the number to internalize: a response that looks 50% correct on a rubric is still, in operational terms, an unfinished draft a scientist must audit. LifeSciBench measures the front-end interaction — request in, expert-facing answer out — and its verdict is that frontier models are becoming persuasive collaborators well before they become dependable ones.
Find this story relevant to you?
Contact us to find a unique solution
Brauchen Sie einen KI-Engineering-Partner, der auch liefern kann?
Wir helfen Unternehmen dabei, KI in Produkte und Prozesse zu integrieren, wertvolle Workflows zu automatisieren und die zugrunde liegenden Softwaresysteme zu modernisieren.
Weiterführende Beiträge
Weitere Analysen rund um Produktbereitstellung, operative KI und die Systemarbeit, die dafür sorgt, dass der Einsatz in der Praxis Bestand hat.