OpenAI's IndQA measures cultural reasoning in 12 Indian languages

What IndQA actually is

IndQA is a benchmark of 2,278 questions spanning 12 languages and 10 cultural domains, created in partnership with 261 domain experts across India. The languages include Bengali, Hindi, Kannada, Marathi, Odia, Telugu, Gujarati, Malayalam, Punjabi, Tamil, English—and, notably, Hinglish.

OpenAI states the reasoning for building it directly: existing multilingual benchmarks like MMMLU are saturated, with top models clustering near high scores, and most non-English tests reduce to translation or multiple-choice tasks.

They don't adequately capture what really matters for evaluating an AI system's language capabilities—understanding context, culture, history, and the things that matter to people where they live.Montana Labs

The domains are concrete: Architecture & Design, Arts & Culture, Everyday Life, Food & Cuisine, History, Law & Ethics, Literature & Linguistics, Media & Entertainment, Religion & Spirituality, and Sports & Recreation. Each item ships with a culturally grounded prompt, an English translation for auditability, weighted rubric criteria, and an expert-written ideal answer.

The Hinglish inclusion is a frontend signal

OpenAI added Hinglish as its own language category, explaining the choice: "We specifically added Hinglish given the prevalence of code-switching in conversations." This is the detail teams building chat interfaces for Indian users should notice.

Users typing into a text box rarely commit to a single language. They mix Hindi and English mid-sentence, switch scripts, and expect the interface to keep up. A benchmark that treats code-switched input as a first-class category, rather than a degenerate case of two clean languages, aligns evaluation with how people actually type into a product's front door.

OpenAI grounds the market rationale too: India has roughly a billion people who don't use English as their primary language, 22 official languages, and is described as ChatGPT's second largest market. The Hinglish category treats messy, real input as the target rather than an edge case.

Rubric grading and adversarial filtering

IndQA grades each response against criteria written by domain experts for that specific question, with weighted point values, checked by a model-based grader. The final score is the sum of points for satisfied criteria. This is closer to grading an essay against a rubric than scoring a multiple-choice answer key.

The construction method carries a deliberate bias. Every question was tested against GPT-4o, o3, GPT-4.5, and partially GPT-5, and only questions a majority of these models failed were kept. OpenAI is explicit that this preserves headroom—but also that it confounds interpretation.

Because questions were filtered to those GPT‑4o, OpenAI o3, GPT‑4.5, and (post public launch) GPT‑5 could not answer sufficiently, question selection is adversarial against these models. This potentially confounds the relative performance of GPT‑5, and could disadvantage all OpenAI models compared to non-OpenAI models.Montana Labs

OpenAI also warns that IndQA is not a language leaderboard, because questions are not identical across languages; cross-language scores are not direct comparisons. The stated use is tracking improvement over time within a model family.

What this changes for teams shipping Indian-language interfaces

IndQA does not hand product teams a ranking they can copy into a vendor decision. Its adversarial filtering and per-language question sets are structured to measure progress inside a model family, not to declare a winner across providers.

The transferable lesson is methodological. If your users interact in Bengali, Tamil, or Hinglish, translation accuracy and multiple-choice tests will tell you little about whether the model reasons correctly about the culture those users live in. The IndQA pattern—native-authored prompts, expert rubrics with weighted criteria, an auditable English translation, and questions the current best models fail—is a template teams can reproduce for their own domains.

OpenAI frames exactly that as the intended outcome, saying it hopes the release will inform and inspire new benchmark creation, especially in languages and domains poorly covered today. For anyone whose frontend serves non-English speakers, the practical takeaway is to evaluate on culturally grounded, expert-graded tasks in the actual language of the input—including the mixed-language input people really send.