OpenAI's political bias evaluation decomposes tone, not just facts

Bias defined as communication style, not stated belief

The core move in this post is defining political bias as something observable in output behavior rather than in a model's professed positions. OpenAI names five axes: user invalidation, user escalation, personal political expression, asymmetric coverage, and political refusals.

That decomposition matters because it targets how a response is written, not only what facts it contains. The post is explicit that a model can get individual facts right and still be biased through one-sided framing, selective evidence, or language that amplifies a user's slant.

Human bias isn't only "what one believes"; it's also how one communicates through what is emphasized, excluded, or implied. The same is true for models.Montana Labs

The worked example makes this concrete. A 'biased' response to a question about war spending scored 0.67, driven by 0.75 on personal political expression and 0.75 on asymmetric coverage — including a closing line validating the user ('the fact you're asking this shows you're paying attention'). The reference response covered similar ground but scored 0.00 by attributing arguments to actors rather than adopting them.

An LLM grader anchored to reference answers

The evaluation runs on approximately 500 prompts across 100 topics, each written from five perspectives: liberal charged, liberal neutral, neutral, conservative neutral, and conservative charged. Topics come from U.S. party platforms and culturally salient issues like parenting and gender roles.

Scoring is done by an LLM grader given per-axis instructions, with human-written reference responses used to validate the grader's scores during iteration. This is a model grading another model against a rubric — a setup OpenAI hopes to apply to any model, not just its own.

The rubric is deliberately strict: even the reference responses do not score zero. That framing keeps the absolute numbers from being read as a clean bill of health and positions the evaluation as a tracking signal over time rather than a pass/fail gate.

The asymmetry that stress-testing exposed

The headline results: near-objective behavior on neutral or slightly slanted prompts, moderate bias on emotionally charged ones. GPT-5 instant and GPT-5 thinking reduce bias roughly 30% versus GPT-4o and o3, and hold up better under charged prompts. Worst-case scores for the older models were 0.138 for o3 and 0.107 for GPT-4o.

One finding stands out because OpenAI states it plainly: strongly charged liberal prompts pull harder on objectivity than charged conservative prompts, across model families. That is a directional admission, not a symmetric 'both sides' claim, and it identifies where the remaining work sits.

OpenAI also applied the method to production traffic and estimates fewer than 0.01% of ChatGPT responses show signs of political bias — attributing the low rate both to how rarely users ask slanted questions and to model robustness. Notably, web search is out of scope, so retrieval and source selection are excluded from these numbers.

What a released rubric gives teams building on these models

The practical takeaway for anyone shipping a chat product is that the five axes are reusable measurement primitives. Escalation, invalidation, and asymmetric coverage are the same failure modes that surface in customer support, health, and any domain where a model mirrors a user's charged framing back at them.

OpenAI stops short of releasing the prompt set or grader code, but the operational definitions and the grader instructions are specific enough to reimplement. A team can build its own reference answers for its own domain and grade against them, rather than relying on multiple-choice benchmarks like the Political Compass that the post criticizes as too narrow.

The specific implication: the evaluation shows bias is concentrated in a small set of measurable behaviors under adversarial pressure, which means it can be monitored continuously and fixed narrowly — and any team using GPT-5 in a user-facing role now has a documented picture of where objectivity degrades and a template for measuring it in their own traffic.