Unconstrained classification is a frequent source of operational surprise. When a model is free to emit any string as a category, it will eventually invent labels that no downstream handler understands. The resulting document is either dropped, sent to a default queue that was never designed for it, or processed by the wrong pipeline. In each case the failure is silent until a human notices missing or corrupted data.
This article constructs a document router that eliminates that class of failure. Mistral Large 4 is given a JSON Schema that enumerates every allowed label and forbids additional properties. The model’s response is validated against the schema and against a Pydantic model. Only after the label is confirmed to be one of the declared values does the service hand the document to the corresponding handler. Handlers themselves are ordinary Python callables; the model never executes code or chooses a side effect.
The implementation is a single FastAPI application written for Python 3.12. It uses the official Mistral SDK, Pydantic v2, and a simple in-process dispatcher. The same structure can later be split into separate services if traffic or team boundaries require it.
Routing Architecture and Trust Boundaries
Closed label set
All possible classification outcomes are known at deployment time and are listed in the JSON Schema. The model cannot invent a new label; any attempt to do so fails validation and returns a 422. Adding a new label is a deliberate code change that also registers a new handler.
Dispatcher as pure mapping
The dispatcher is a dictionary that maps each validated label to a callable. Looking up a missing key is impossible after validation, so the dispatcher itself cannot raise a KeyError under normal operation. This keeps the control flow linear and easy to audit.
Side-effect isolation
Handlers may perform I/O, enqueue work, or call other services. The classification step itself performs none of those actions. Consequently a classification failure never leaves a partial side effect.
Request identity
Every request receives a unique ID that is propagated to the model call, the validation logs, and the handler. This ID is the sole correlation key used for debugging and for idempotent retries.
Project Structure and Runtime Contract
Layout
The repository contains router/ for the FastAPI application and dispatcher, schemas/ for the classification schema and Pydantic models, handlers/ for the per-label callables, and tests/ for fixtures and integration tests. A pyproject.toml pins all dependencies.
Environment requirements
MISTRAL_API_KEY and MISTRAL_MODEL are required. An optional CLASSIFICATION_TIMEOUT_SECONDS defaults to a conservative value. The process exits at startup if the required variables are missing.
Handler registration
Handlers are registered at import time by a decorator or by an explicit dictionary. The set of registered handlers is compared to the set of schema labels during application startup; any mismatch aborts the process. This prevents a deploy in which a label exists in the schema but has no handler.
Versioning
The schema carries a version string that is logged with every classification. Changing the set of labels increments the version and requires a coordinated change to the handler registry.
Building the Constrained Classification Request
Schema with enum constraint
The JSON Schema defines a single object with one required property, label, whose value must be a member of a fixed enum. Additional properties are forbidden. The schema is the only place the allowed labels are listed.
System prompt discipline
The system prompt instructs the model to act as a document classifier and to return only the JSON object defined by the schema. No examples of documents or labels are included; the enum supplies the entire decision space.
SDK invocation
The Mistral client is called with the response format set to the schema, temperature zero, and a modest max-tokens limit. The call is wrapped in a timeout so that a slow response cannot block the worker.
Immediate validation
The returned string is decoded and validated against both the JSON Schema and the corresponding Pydantic model. Any deviation produces a 422 before the dispatcher is invoked.
Dispatch, Error Handling, and Observability
Lookup and execution
After validation the label is used as a key into the handler dictionary. The handler receives the original document and the request ID. Its return value becomes the HTTP response body.
Error mapping
Model or validation failures map to 422. Handler exceptions map to 500 and are logged with the full stack trace. Timeouts map to 504. The client never receives an unvalidated label.
Metrics
Counters are emitted for each label, for validation failures, and for handler errors. A histogram records end-to-end latency. These signals make it possible to detect a sudden shift in label distribution or a rise in rejection rate.
Comparison of routing failure modes
Failure mode | Detected by | HTTP status | Retry recommended | Operational action |
|---|---|---|---|---|
Invalid JSON from model | JSON decoder | 422 | No | Inspect prompt or model version |
Label not in enum | JSON Schema / Pydantic | 422 | No | Check for model regression |
Handler missing for label | Startup check | N/A | N/A | Fix registration before deploy |
Handler raises | Exception handler | 500 | Depends | Alert and inspect handler logs |
Model timeout | Client timeout | 504 | Yes, with backoff | Check upstream latency or increase budget |
Testing the Router
Label coverage fixtures
A fixture exists for every declared label. Each fixture contains a short document that should produce that label. The tests assert both the HTTP status and the label that was dispatched.
Negative fixtures
Additional fixtures contain documents that should not match any label strongly, or that attempt to force an out-of-enum response. The tests confirm that these produce 422 and never reach a handler.
Handler isolation
Handlers are unit-tested independently of the classifier. A mock dispatcher can inject any validated label so that handler behavior can be verified without calling the model.
Startup assertion tests
A dedicated test starts the application with a deliberately mismatched schema and handler set and asserts that the process exits with a non-zero code. This protects against registration drift.
Latency, Token Use, and Scaling Considerations
Classification cost
Because the schema is small and the completion is a single enum value, both prompt and completion token counts stay low. The dominant cost is the model’s per-request overhead rather than token volume.
Latency budget
The classification call is expected to complete well under one second for typical document sizes. The subsequent handler may take longer; its latency is measured separately so that slow handlers do not pollute the classification metric.
Horizontal scaling
The router is stateless. Multiple instances can sit behind a load balancer. The only shared resource is the Mistral API quota, which should be monitored and, if necessary, protected by a shared rate limiter.
Decision table for scaling levers
Lever | Effect on latency | Effect on cost | Complexity introduced |
|---|---|---|---|
Increase replicas | Reduces queue time | None (API cost same) | Load-balancer configuration |
Shorten documents | Reduces model time | Reduces token cost | Upstream truncation logic |
Cache frequent documents | Eliminates model call | Eliminates token cost | Cache invalidation policy |
Lower max tokens further | Negligible | Small savings | Risk of truncation |
Batch classification | Amortizes overhead | Potential savings | Batching and ordering logic |
Production Verdict and Checklist
The router is ready for production traffic once authentication, rate limiting, and durable logging are added. The central guarantee—no document is ever dispatched under an undeclared label—is enforced by the schema and the startup registration check. Monitoring should focus on the distribution of labels and the rate of validation rejections; either can signal a change in input traffic or a model behavior shift.
Before releasing, verify:
Schema enum and handler registry contain identical sets of labels.
Startup aborts on any mismatch.
Every label has at least one positive fixture that reaches the correct handler.
Negative fixtures produce 422 and never invoke handlers.
Request ID is present in all logs and handler calls.
Metrics for label counts, validation failures, and latency are exported.
Timeout and retry policy for the model call are documented and tested.
A clear process exists for adding a new label (schema change, handler, fixtures, version bump).
When these conditions hold, the service can safely classify documents and route them under application control, keeping the model inside a finite and auditable decision space.
