The costliest production failures aren't crashes—they're model outputs that pass json.loads but are semantically wrong: missing required fields, wrong types, or undeclared keys. Once that object hits storage or downstream logic, the safety boundary is already breached.
The fix is to keep the model useful while making the application the final authority. This article builds a FastAPI service that:
Calls Mistral Large 4 for structured extraction
Enforces a JSON Schema on the request
Validates the response with Pydantic
Rejects anything that doesn't match the declared contract—never trusting the model's own claim of validity
The target is a single endpoint: free-text document in, typed extraction object out, with clear error codes for incomplete or malformed data. Written for Python 3.12 using the official Mistral SDK, designed to drop into a larger production gateway with minimal change.
Target Architecture and Service Boundaries
Service responsibility split
The FastAPI process owns three responsibilities only: receiving the raw text, constructing a constrained request to Mistral Large 4, and validating the returned payload against a Pydantic model. Everything else—persistence, authentication, rate limiting—lives outside this service. The model is treated as an untrusted proposer; the application decides acceptance.
Request and response contracts
Incoming requests carry a single string field named document. Outgoing success responses carry a typed object defined by a Pydantic model. Failure responses carry a machine-readable error code and a human-readable message that never leaks internal schema details to the caller. This separation keeps the public API stable even when the internal schema evolves.
Isolation of model configuration
Model name, temperature, and max tokens are read from environment variables at process start. They are never hard-coded in request handlers. Changing the model version therefore requires only a configuration change and a redeploy, not a code change.
Explicit failure surface
Any validation failure, network timeout, or non-200 response from the Mistral API is mapped to one of three HTTP status codes: 422 for schema or type violations, 504 for timeouts, and 502 for upstream API errors. The service never returns a partial object.
Repository Structure and Environment Assumptions
Directory layout
A minimal repository contains four top-level items: app/ for the FastAPI code, schemas/ for the JSON Schema and Pydantic models, tests/ for unit and integration tests, and a pyproject.toml that pins every dependency. No notebooks or ad-hoc scripts are allowed in the production path.
Dependency versions
Python 3.12 is required. The Mistral SDK is pinned to a specific minor version. Pydantic v2 and FastAPI are used together so that the same model class can generate both the OpenAPI documentation and the runtime validation. All versions are declared in pyproject.toml and locked with a lockfile.
Environment variables
Three variables are mandatory: MISTRAL_API_KEY, MISTRAL_MODEL (set to the Large 4 identifier), and REQUEST_TIMEOUT_SECONDS. Optional variables control temperature and max tokens. Missing required variables cause the process to exit at startup rather than fail on the first request.
Local development versus production
In local development the same code path is used; only the base URL of the Mistral API changes. This guarantees that the validation logic exercised in tests is identical to the logic that will run in production.
Implementation of the Constrained Request
JSON Schema definition
The schema is declared once as a Python dictionary that can be serialized and sent to the model. It lists every required field, every type, and every enum constraint. Optional fields are explicitly marked so that the model cannot invent them. The schema is versioned by a constant string that is logged with every request.
Prompt construction
The system prompt contains only two sentences: the role of the extractor and the instruction to return only the JSON object that matches the supplied schema. No examples are embedded in the prompt; the schema itself is the sole source of truth. This keeps the token cost predictable and removes the risk of example drift.
SDK call with response format
The Mistral client is invoked with the response_format parameter set to the JSON Schema. Temperature is fixed at zero for this endpoint. The call is wrapped in a timeout context so that a slow model response cannot hang the FastAPI worker indefinitely.
Immediate post-response parsing
As soon as the raw string arrives it is passed through json.loads. Any decoding error is converted into a 422 response before Pydantic is even invoked. This two-stage check surfaces syntax problems separately from semantic or type problems.
Validation, Error Handling, and Retry Logic
Pydantic model validation
The decoded dictionary is handed to a Pydantic model that mirrors the JSON Schema. Pydantic performs type coercion only where the schema explicitly allows it; everything else raises a validation error. The error detail is reduced to a short list of field paths and messages before it is returned to the caller.
Comparison of validation stages
Stage | Responsibility | Failure Mode Detected | HTTP Status |
|---|---|---|---|
JSON decode | Syntactic correctness | Malformed string, trailing commas | 422 |
JSON Schema (model) | Structural compliance | Missing required keys, extra keys | 422 |
Pydantic | Type and value constraints | Wrong types, out-of-range numbers | 422 |
Application logic | Business invariants | Semantic contradictions | 422 |
Retry policy
Only network-level failures and 5xx responses from the Mistral API trigger a retry. Schema validation failures never retry; they are deterministic for a given prompt and temperature. A maximum of two retries with exponential backoff is applied. Each attempt is logged with the attempt number and the latency of the previous call.
Timeout and circuit behavior
A hard timeout is set on the HTTP client. If the timeout fires, the service returns 504 and increments a metric. After three consecutive timeouts the service can optionally open a circuit breaker that returns 503 for a short cool-down period. The circuit logic lives outside the request handler so that it can be shared across endpoints.
Testing Strategy and Failure Injection
Unit tests for the schema
A set of fixture JSON objects is maintained: one that is fully valid, one missing each required field, one with wrong types, and one with extra keys. Each fixture is run through the validation path and the expected status code is asserted. These tests run in milliseconds and do not call the model.
Integration tests against a recorded response
A small set of recorded model responses is stored in the repository. The FastAPI test client is pointed at a mock transport that returns these recordings. This allows the full request path to be exercised without consuming API credits or depending on live model behavior.
Adversarial fixtures
Additional fixtures deliberately contain valid JSON that violates business rules (for example, a negative quantity when the schema only constrains the type). These fixtures confirm that the application layer, not the model, enforces the final invariants.
Continuous regression
Every change to the schema or the Pydantic model triggers the full fixture suite. A GitHub Actions workflow runs the suite on every pull request and fails the build if any fixture produces an unexpected status code.
Latency, Token, and Cost Considerations
Token accounting
The service logs the number of prompt tokens and completion tokens returned by the Mistral API for every successful call. These numbers are attached to the request ID so that downstream cost attribution is possible. Hypothetical measurements under a fixed document length show that the schema itself accounts for a stable fraction of the prompt tokens; lengthening the document increases only the variable portion.
Latency budget
The end-to-end latency target for this endpoint is under two seconds at the 95th percentile under light load. The dominant component is the model’s generation time. Validation and network overhead are measured separately so that regressions in either can be isolated.
Cost of retries
Because validation failures do not retry, the only additional cost comes from network retries. Under normal conditions the retry rate is expected to stay below one percent. The service exposes a metric for retry count so that an unexpected spike can be investigated.
Comparison of cost drivers
Driver | Controllable by application | Typical contribution | Mitigation |
|---|---|---|---|
Schema size | Yes | Fixed | Keep schema minimal |
Document length | Partially | Variable | Truncate or summarize upstream |
Temperature > 0 | Yes | Increases variance | Keep at zero for extraction |
Retries on 5xx | Partially | Rare | Circuit breaker and backoff |
Unnecessary fields | Yes | Fixed | Remove optional fields not required |
Production Verdict and Implementation Checklist
The service described here is suitable for staging and for production workloads that can tolerate a small latency overhead in exchange for deterministic validation. It is not a complete production gateway; authentication, multi-tenancy, and durable logging must still be added. The core contract—model proposes, application validates—is production-ready once the surrounding infrastructure is in place.
Before deploying, confirm the following:
All required environment variables are present and the process exits cleanly when any is missing.
The JSON Schema and the Pydantic model are kept in sync by a single source of truth.
Unit fixtures cover every required field and every type constraint.
Integration tests run against recorded responses and pass.
Metrics for latency, token usage, validation failures, and retries are exported.
The public error messages never expose internal schema details.
A documented rollback plan exists for schema changes that break existing clients.
When these items are satisfied, the endpoint can be placed behind an API gateway and exposed to internal consumers. The model remains useful for extraction; the application remains the final authority on what enters the system of record.
