LLMs are increasingly deployed to produce structured data from unstructured and semi-structured sources, parsing invoices, medical records, meeting transcripts, and converting PDFs to database rows.

For deterministic output, the next step in a workflow reads a specific key and expects a specific type. A hallucinated invoice_total or an array ordered incorrectly because of inaccurate date values silently breaks downstream systems. Yet existing benchmarks either check schema compliance alone or evaluate value correctness within a single source domain.

Top 5 at a glance

A side-by-side look at the top 5 models across all seven metrics. The structural metrics (JSON Pass, Path Recall, Structure Coverage, Type Safety) cluster near the ceiling for every model, while Value Accuracy and Perfect Response separate them.

The problem with current structured output benchmarks

Most benchmarks collapse "structured output quality" into a single number: does the response parse, and does it validate against the schema? That's necessary, not sufficient.

References to existing benchmarks: JSONSchemaBench | StructEval | DeepJSONEval | LLMStructBench | ExtractBench | STED

How SOB works

SOB evaluates structured output across three modalities using the same scoring harness. The goal is to isolate the extraction capability from every other ability a model has.

Three sources, one scoring pipeline

Every record is paired with a JSON Schema and a ground-truth answer that was verified against the source context through human authoring with an LLM cross-check, so a missing or hallucinated value is unambiguously wrong.

To isolate the structured-output capability from vision and ASR quality, image and audio records are converted to text-normalized context before scoring. Models see the same modality-stripped context, and the differences that remain are attributable to how they handle schemas, nesting, and value grounding under different content distributions.

Seven metrics, not one

SOB reports seven metrics per record so you can see exactly where a model fails:

Value Accuracy is the metric that matters for production. It's the share of fields a downstream system can trust without a human review step.

Scoring gates

Two gates prevent inflated scores from schema-only wins:

• Hardening gate: If JSON parse fails, downstream semantic metrics are zeroed for that record.
• Coverage gate: Value Accuracy is only credited on fields the model actually returned, with missing paths counting as wrong.

Each schema is tagged medium (nested objects or arrays of scalars, depth 2) or hard (arrays of objects, or depth ≥3). Hard schemas dominate the corpus (61% on text, 88% on image, 98% on audio), and the final leaderboard is schema-complexity-weighted (easy = 1.0, medium = 2.0, hard = 3.0) so hard schemas contribute more to the ranking than medium ones.

The results

We ran SOB on all models at temperature 0.0, max output 2048 tokens and no reasoning/thinking wherever the provider allows it, so the score reflects pure structured output and extraction capability.

A note on reasoning-locked models

A handful of models in the leaderboard cannot have reasoning fully turned off, so they ran in their lowest-reasoning configuration:

• GPT-5 and GPT-5-Mini: the API only allows a minimum reasoning effort, not a full disable. The lowest accepted value also changed between generations, from "minimal" on GPT-5 / GPT-5-Mini to "none" on GPT-5.5.
• Gemini-3.1-Pro and Gemini-3-Flash-Preview: thinking is built into the model and can be set to its lowest budget but not switched off.
• DS-R1-Distill-32B: chain-of-thought is intrinsic to the model, baked in during distillation.
• Claude-Opus-4.7: Anthropic deprecated the temperature parameter for this model, so it runs without one (the rest are at temperature 0.0).
• Phi-4: only evaluated on text and image. Excluded from audio because its 16K context window cannot fit the longer AMI transcripts, so audio counts use 27 models.

That means the locked models above are scored with a small reasoning advantage the others do not get, yet several non-reasoning models still beat them on Value Accuracy.

Unified leaderboard

View the full leaderboard

The top six are within 1 point of each other on overall score, but swap freely across individual metrics. Rank order is metric-specific, not absolute. Gemini-3.1-Pro tops the unified Value Accuracy leaderboard at 82.0%, edging out GLM-5.1 (80.6%), GLM-4.7 (80.4%), and Qwen3.5-35B (80.1%), three open-weight models that beat several frontier closed-source releases on the metric that matters most for production extraction.

Cost: the top two rarely win in production

The two leaders are also the two most expensive models on this list. GPT-5.4 runs $2.50 /$15.00 per 1M input/output tokens, Gemini-3.1-Pro $2.00 /$12.00. Compare that to the models trailing them by under 2 points on Value Accuracy:

Prices are per 1M tokens on each provider's standard public API tier. A pipeline running Qwen3.5-35B, GLM-4.7, or Gemini-2.5-Flash costs anywhere from 6x to 11x less per correct field than the benchmark leader, while losing at most ~2 percentage points of Value Accuracy. That is why the top two models, despite leading SOB, are rarely the default choice for production structured-output pipelines.

Per-metric charts

Each chart re-sorts all 28 models on that single metric, so you can see which models win each category (not just the overall average).

To expose the gaps, each chart's x-axis starts from a floor appropriate to that metric (e.g. 60% for Value Accuracy, 80% for JSON Pass). Without that, the top cluster looks identical.

Value Accuracy

The metric production systems care about. Gemini-3.1-Pro leads at 82.0%, with three open-weight models (GLM-5.1, GLM-4.7, Qwen3.5-35B) trailing it inside a 2-point band.

Faithfulness

How often values are grounded in context instead of hallucinated.

JSON Pass Rate

Almost every modern model clears 95%+ in the unified leaderboard. This is why a pass-rate-only benchmark can't separate them anymore.

Path Recall

Whether all required keys appear in the output.

Structure Coverage

Whether nested objects and arrays are present with the correct shape.

Type Safety

Whether leaf values respect the declared JSON Schema types (no strings where numbers are expected).

Perfect Response Rate

The fraction of records where every single leaf value is exactly right. This is the hardest metric and collapses to roughly half even for the best models.

The JSON-pass vs Value-Accuracy gap

The single most important view: most models clear 95%+ on JSON Pass, but Value Accuracy sits 15 to 30 points lower. That gap is the space where structured output benchmarks have been lying to us.

The gap column is the headline. Every model on this list passes JSON parsing 97%+ of the time, but actual leaf-value extraction drops by 17 to 26 points. GLM-5.1 has the tightest gap (16.9 pp) and the highest Value Accuracy on the list, while Schematron-8B passes JSON 98.7% of the time but lands the lowest Value Accuracy at 73.1%, a 25.6 point fall.

Modalities diverge more than we expected

The same model scores very differently across text, image, and audio, even when every model gets the same text-normalized context. Audio is the hardest by far. The transcripts are long (~7,300 tokens on average) and full of overlapping speakers, so models struggle to pull out the right values.

Best Value Accuracy by modality across all valid models:

No single model wins all three. GPT-5.4 ranks 5th on text but 13th on images. Schematron-8B ranks 26th on text but 15th on images. Gemma-4-31B ranks 18th on text but 1st on images.

Patterns worth internalizing

1. Valid JSON ≠ correct JSON. JSON Pass and Value Accuracy diverge by 15 to 30 points on every frontier model.
2. Structural metrics mask value errors. Path Recall / Structure Coverage / Type Safety can all read ~99% while 20 to 30% of leaf values are still wrong, and Perfect Response collapses to about half even for the top models.
3. Model size is not a predictor. GLM-5.1, GLM-4.7, and Qwen3.5-35B (open-weight) beat GPT-5, Claude-Opus-4.7, and Claude-Sonnet-4.6 on unified Value Accuracy. Phi-4 (14B) edges out GPT-5 and GPT-5-Mini on text.
4. Generation-over-generation gains are flat. GPT-5.5 trails GPT-5.4 by 0.3 pp on unified Value Accuracy, GPT-5.4 only edges GPT-4.1 by ~2 pp, and Claude-Opus-4.7 beats Opus-4.6 by less than 1 pp. Scaling compute alone hasn't closed the value-accuracy gap.
5. Structured hallucinations are the hardest bug. The value is type-correct, schema-valid, and plausible, so it slips through most guardrails. On one audio record the ground truth is "target_market_age": "15 to 35 years" and a model returns "25 to 35" — invisible without field-level checks.
6. Modalities don't transfer. Text-trained structured output behavior degrades sharply when the source is a transcribed conversation. Best Value Accuracy drops from 84.5% on text to 67.2% on images to 23.7% on audio.
7. Rankings shift across modalities. Gemini-3.1-Pro leads text, Gemma-4-31B leads images, Gemini-2.5-Flash leads audio. No single model dominates all three, so a text-only leaderboard would mask the gaps.

What's next

SOB is a first step, not a finish line. We'll keep growing the benchmark along several axes:

• More datasets, including newer datasets with increasing complexity and difficulty.
• More schemas and difficulty tiers, including recursive types, unions, and large enum spaces.
• Continuous re-evaluation as new models ship, and ongoing transparent tracking of our own models against the same scoring harness so we can measure ourselves honestly.

Why we released SOB?

Our goal is to be the best general model for deterministic tasks and a key aspect of determinism is controllable and consistent output structure. The first step to making structured output better is to measure it and hold ourselves against the best.

• Docs: Structured Output in Interfaze
• Playground: Try structured output on Interfaze
• Paper: arXiv
• Dataset: Hugging Face

SOB launch video