First-pass accuracy is a flattering number. Production systems pay for every attempt, including the ones that fail schema checks and run again. When you evaluate Mistral Large 4 on structured extraction or classification, measure retries, tokens, and elapsed time per accepted record. Otherwise you will promote a prompt that looks accurate in a notebook and expensive under load.
Why Accuracy Alone Misleads
Hidden work behind a green cell
A report that says ninety percent valid JSON may omit that a third of successes needed a second completion. Operators inherit that hidden work as latency and spend. Publish attempt histograms beside pass rates so design reviews see the shape of failure, not only the final acceptance fraction after unbounded patience.
Retries change the ranking
Template A might win first-pass schema rate while Template B wins dollars per accepted row once retries are included. Client policies that retry validation failures by re-prompting will reorder winners again. Any benchmark that does not state the retry policy is incomplete and should not drive a production pin.
Semantic pass is not schema pass
Schema-valid objects can still be wrong for the business. Count semantic acceptance separately after deterministic checks. A pipeline that retries only on parse errors but accepts plausible nonsense will look efficient and still corrupt stores. Accuracy claims need both layers named explicitly in the harness output.
Preview pins and drift
Mistral Large 4 preview checkpoints can move. A retry profile collected last week may not match this week’s pin. Record model id, date, temperature, and max tokens on every run. Without pins, retry benchmarks become stories rather than evidence you can re-run after a provider change.
What to Measure on Each Item
Attempt count
Store how many provider calls were required before accept, permanent reject, or budget exhaustion. Cap attempts in the harness the same way production will. Unlimited lab retries invent a success rate no client will afford. Publish the share of items that used one, two, or three attempts as a small distribution, not only a mean.
Tokens per attempt and per success
Sum input and output tokens across attempts for each fixture item. Divide by semantic accepts to get tokens per success. Items that never accept should appear in a separate failure cost bucket so they still affect the total. Omitting permanent failures makes a brittle template look cheaper than it is in aggregate.
Time per success
Use monotonic clocks from first request start to final decision. Include backoff sleeps if production does. Exclude manual pauses. Report percentiles across the fixture set. A template with fewer retries but long completions can still lose on user-visible time, which is a different decision than pure token cost.
Outcome classes
Classify each item as accepted, schema_rejected, semantic_rejected, timeout, or budget_exhausted. Low-cardinality classes keep metrics usable. These classes also tell you whether to fix prompts, validators, or client timeouts when a regression appears after a model pin change.
Harness Design
Fixed fixture pack
Build fixtures that mirror production difficulty: messy text, borderline enums, empty fields, and adversarial instructions. Version the pack. Changing fixtures between comparisons invalidates trends. Scrub secrets. For multimodal structured tasks, keep image inputs stable across template variants so vision tokens do not confound text-template tests.
One variable at a time
Compare prompt templates, schema strictness, or repair strategies separately. If you change temperature and template together, you will not know which moved the retry curve. Write the comparison matrix before the run. Exploratory noodling belongs in a scratch branch, not in the numbered result you present to stakeholders.
Production-like client policy
Copy timeout, max attempts, and whether validation failures re-invoke the model. Lab clients that always repair with a longer prompt overstate what a disciplined gateway allows. If production fails closed on schema errors, the benchmark should too, unless the experiment is explicitly about a proposed repair policy.
Recording requirements
Field | Purpose |
|---|---|
model id | Pin against preview drift |
template version | Compare prompt changes |
schema version | Attribute contract shifts |
attempt n tokens | Build cost curves |
attempt n latency | Build time curves |
final outcome class | Separate failure modes |
Interpreting Retry Curves
High first-pass, low repair value
If almost all accepts occur on attempt one, investing in elaborate repair prompts wastes tokens. Focus on the small hard cohort with specialized templates or human review rather than global second attempts. Global retries tax the easy majority to save the difficult minority without proof of need.
Low first-pass, effective second attempt
When attempt two recovers many schema failures, a single controlled repair may pay off. Still measure whether repair uses a cheaper instruction, a stricter schema reminder, or only a blind resend. Blind resends often reprint the same error and should lose to targeted repair or to fail-closed behavior.
Semantic failures after valid JSON
Retries rarely fix systematic semantic mistakes if the snapshot data is incomplete. More completions will not create a missing foreign key. Route those items to data quality fixes. Retry budgets aimed at semantic misses usually burn money without raising true acceptance.
Pattern guide
Pattern | Likely cause | Better lever than more retries |
|---|---|---|
Many decode errors | Truncation, max tokens | Raise completion budget carefully; tighten schema |
Many schema errors | Prompt or contract mismatch | Schema-first prompt; few-shot only if accurate |
Many semantic errors | Weak context or hard task | Better retrieval; human review |
Timeouts with good schema | Latency path | Measure TTFT and queue; resize workers |
Cost and Latency Math Teams Skip
Price the whole path
Multiply token totals by verified rate-card figures for input and output, including cached input if you rely on it. Verify before publication against the live Mistral price sheet for your account. Promo preview rates should be labeled so finance does not treat them as steady state.
Retries under concurrency
Retry storms synchronize on shared 429 responses and amplify queue delay. A single-threaded harness understates that risk. Include a concurrent mode with the same attempt caps to see whether retry policy still looks attractive when workers compete. If concurrent retries collapse success latency, simplify the policy.
Opportunity cost of delay
Even when retries eventually accept, product metrics may suffer. Report time-to-accept percentiles next to dollars-per-accept. Stakeholders who only see accuracy will not understand why a slightly less accurate first-pass template ships faster and cheaper at p95.
When fail-closed wins
If attempt two recovers little, fail-closed on schema errors and spend engineering time on fixtures and schema clarity. The benchmark’s job is to reveal that plateau. Continuing to retry past the plateau is a habit, not a strategy, and it should lose in a fair tokens-per-success table.
Reporting Without Self-Deception
Show distributions
Means hide that a few pathological fixtures consume most retries. Provide histograms or quantiles for attempts and tokens. Call out top offenders and decide whether they belong in production automation at all. Some items should route to humans by policy after one failed attempt.
Separate lab from production
Lab fixtures estimate relative template quality. Production traces validate the estimate. If production attempt counts diverge, investigate traffic mix and prompt paths that never appeared in the pack. Hypothetical parity between lab and production is not something to assert without data.
Avoid vanity dashboards
A single accuracy tile is vanity if retries are invisible. Prefer a small panel: first-pass schema rate, accept rate after policy, tokens per accept, p95 time to decision, and budget exhaust rate. That set answers whether a change is shippable.
Governance of changes
Require harness results before promoting a new structured-output template or client retry flag. Store results next to the template version in the repo. Culture matters here: if people can ship prompt edits without numbers, the benchmark will rot into a one-off slide.
What To Do With The Results
Choosing a template
Pick the candidate with the best tokens per semantic accept inside your latency budget, not the one with the prettiest first-pass score. If two candidates tie on cost, prefer the simpler template for operability. Complexity in prompts is ongoing liability when Mistral Large 4 pins change and fixtures expand.
Choosing a client policy
Allow transport retries. Treat schema failures as non-retryable unless a measured repair path recovers enough accepts to pay for itself under concurrency. Document the policy in the gateway or client so application teams cannot override it with ad hoc loops that bypass accounting.
Choosing what not to automate
Fixtures that remain expensive after reasonable retries are candidates for human review or for product rules that refuse automation. Structured output is a tool, not an obligation to force every record through the model until the budget snaps. The benchmark should make those refusals easier to defend.
A short worked comparison mindset
Imagine Template Short and Template Strict on the same one hundred fixtures under a two-attempt cap. You care which yields more accepts per million tokens and which finishes p95 sooner. Until those two numbers exist, arguments about wording are premature. After they exist, wording arguments yield to arithmetic.
Accuracy answers whether the model can produce a shape you like once. Retry-aware benchmarks answer whether you can afford to run that process as a system. For Mistral Large 4 structured outputs, ship the configuration that minimizes tokens and time per accepted record under a pinned model, a fixed schema, and a retry policy you are willing to run at peak concurrency. If a change only improves first-pass accuracy by shifting work into silent second attempts, it has not improved the system—it has moved the cost out of the slide and into the invoice.
