A request that reaches Mistral Large 4 can stall, return a rate-limit response, or arrive as JSON that does not match the contract the caller expects. Treating that call as a local function hides those failures inside the business path. The client below classifies every outcome, retries only what can change, and returns a validated object or a typed exception.
Client Responsibilities and Trust Boundaries
Client responsibilities
The client builds HTTP requests, sets per-call timeouts, categorizes results, and validates outgoing payloads. It does not handle data persistence, business logic, or authorization — those belong to the host service. Its minimal interface keeps retry rules easy to audit.
Retry classification
Retryable failures: transport timeouts, connection resets, 429 and 5xx HTTP errors. Terminal failures: most 4xx responses, JSON parsing errors, schema violations. Retrying terminal cases wastes tokens and latency. Failure types are explicitly defined in code instead of generic exception handling.
Explicit configuration
Timeout, max retries, base backoff and max sleep are constructor parameters. Production loads them from environment variables. Missing mandatory config crashes the service before receiving traffic; tests can inject custom values without global patches.
Logging & metrics
Each attempt logs request ID, attempt count, duration and failure type. Token usage is logged when parsing succeeds. A metrics callback is supported, with no hard dependency on monitoring libraries. Logs alone can reconstruct failure sequences for incident review.
Repository Shape and Runtime Assumptions
Package layout
The package contains a client module, an error module, and a schemas module. Tests sit beside recorded fixtures for success, rate limit, server error, malformed JSON, and schema mismatch. A lockfile pins the Mistral SDK, Pydantic v2, and the HTTP transport. No notebook or ad-hoc script is on the import path used in production.
Runtime contract
Python 3.12 is the supported interpreter. The model name is an environment variable, not a string literal inside the request method, so a model change is a configuration change. The API key is read once at construction and is never written to logs. Fixture tests do not require a live key.
Startup checks
Construction verifies that the timeout is positive, the attempt budget is at least one, and the base backoff is non-negative. Invalid combinations raise before any socket opens. This catches misconfigured deployments during rollout rather than on the first user request.
What this client does not do
It does not open a circuit breaker, share a process-wide concurrency limit, or bill callers. Those controls belong in a gateway when many services share one key. The client documents that limit so teams do not assume a single wrapper is a full traffic policy.
Timeouts, Backoff, and Cancellation
Deadline on each attempt
Each HTTP call runs inside its own timeout. When the deadline fires, the attempt is aborted and marked retryable. The same timeout applies to every attempt so the worst-case duration is computable: attempts times timeout, plus the sum of sleeps. Callers use that number when they set their own upstream deadline.
Jittered exponential sleep
After a retryable failure the client sleeps for the base interval multiplied by two raised to the attempt index, plus a small random jitter, capped at the configured maximum. Jitter keeps concurrent callers from retrying in lockstep after a shared 429. The sleep is interruptible so cancellation does not wait out the full interval.
Attempt budget
The client checks the remaining budget before each call. When the budget is exhausted it raises an aggregate error that includes the last underlying exception and the attempt history. The history lists class, status code when present, and latency. Operators can see whether the failure was a timeout storm or a short burst of 500s.
Cancellation
An optional token lets an orchestrator abort the sequence. Cancellation during sleep or during the HTTP call raises an error that is not retried. The client does not swallow cancellation and continue with another attempt, because the caller has already decided the work is no longer wanted.
Validation and Error Mapping
Decode before schema
The body is parsed as JSON only after a successful HTTP status that the client treats as success. A decode error becomes a terminal validation error. The client does not retry it and does not attempt to repair truncated JSON. Repair would hide a max-token or transport problem and could accept a partial object.
Schema as the return type
The decoded object is passed to a Pydantic model that declares required fields and types. Extra keys fail when the model forbids them. A successful return is always that model instance. Callers do not receive a dictionary and then validate again in inconsistent ways. If two endpoints need different shapes, they use different models rather than a shared untyped dict.
Classification used by callers
Outcome | Retryable | Exception | Caller response |
|---|---|---|---|
Transport timeout or reset | Yes | Retryable transport error | Backoff within the client, then surface if the budget ends |
HTTP 429 | Yes | Retryable rate-limit error | Same, and inspect quota if the rate stays high |
HTTP 5xx | Yes | Retryable server error | Same; alert if the success ratio drops |
Other HTTP 4xx | No | Non-retryable client error | Fix the request; do not loop |
JSON or schema failure | No | Validation error | Treat as a contract break, not a blip |
Validated body | No | None | Continue with the typed object |
What validation does not prove
A schema-valid object can still be wrong for the business. The client guarantees structure and types. The service that calls it must still enforce invariants the schema cannot express, such as foreign keys or mutual exclusion between fields. Passing validation is permission to inspect the object, not permission to write it.
Tests That Lock the Policy
Fixtures for every class
Each failure class has a recorded response or a transport exception. Tests assert the exception type, the number of attempts, and that terminal failures stop at one attempt. Success fixtures assert the returned model and that token fields were captured when the fixture includes them.
Budget and sleep
A sequence of retryable failures runs with a fake clock. The test checks that sleep durations follow the capped exponential schedule and that the aggregate error appears when the budget ends. Using a fake clock keeps the suite fast and independent of wall time.
Cancellation and validation
One test sets the cancellation token during a simulated sleep and expects no further attempt. Another feeds a body that parses but fails the model and expects a validation error with field detail and a single attempt. These tests are the regression gate for the policy.
Live calls stay out of CI
The default suite never calls the live API. A separate manual script can send one request when a model version changes, and its output is reviewed by a person. That script is not a pass-fail check, because live model behavior is not a stable unit-test oracle.
Latency, Cost, and Operating Limits
What the caller must budget
Total latency includes generation time, failed attempts, and sleep. A two-retry policy with a ten-second timeout can occupy a worker for tens of seconds if every attempt hangs. Services that share a small worker pool should set a lower timeout or a smaller budget rather than discover the pile-up in production.
Cost of retries
Each attempt can consume prompt and completion tokens. The client logs both when the provider returns them. A rising retry rate is therefore a cost event as well as a reliability event. Terminal validation failures are cheaper because they do not repeat, which is another reason not to retry them.
Policy comparison
Policy | Extra latency | Token multiplier | Fits |
|---|---|---|---|
No retry | None beyond one timeout | One call | Paths that must fail fast |
Two retries, capped sleep | Bounded and documented | Up to three calls | Most request-path services |
Five retries | Often longer than the user will wait | Up to six calls | Batch jobs with their own queue timeout |
Gateway circuit plus two retries | Bounded by the circuit | Controlled across callers | Many services sharing one key |
Shared key warning
Nothing in this client stops ten services from retrying the same 429 at once. If that is the deployment, put a gateway or a shared limiter in front of the key and keep this client as the per-call policy inside that gateway.
Production Verdict
Ready when the budget is accepted
The client is appropriate for production once the owning team accepts the worst-case latency and token multiplier and wires logs to the request identifier they already use. It is not a full edge policy.
Checklist before adoption
Confirm construction fails when required settings are missing. Confirm tests cover every row of the classification table. Confirm cancellation does not continue. Confirm validation runs before any return value is handed to business code. Confirm token counts are logged. Confirm the documented worst case is within the service deadline. Confirm a person owns the response when the retry rate or validation-failure rate moves.
What remains outside
Authentication of your own users, per-tenant quotas, and circuit state across processes stay outside this package. Add them where the key and the traffic actually meet. Do not copy retry logic into every handler once this client exists.
Operating rule
Measure the failure path. A quiet success metric that ignores retries will hide both cost and tail latency. Alert on attempt count and on terminal validation errors separately, because they demand different fixes: capacity or provider health on one side, prompt or schema drift on the other.
