A model call is not a library import. Mistral Large 4 brings latency, cost, partial failures, and nondeterministic content into any path that touches it. Where you place the call decides who blocks on the provider, who holds the API key, and how cleanly you can apply retries without amplifying side effects. Three patterns cover most backends.
Placement Options
In-process on the request path
The service that handles the user request constructs the prompt, calls Mistral, validates the response, and continues. Fewer network hops. The product process also absorbs provider stalls, holds the key, and couples scaling of web workers to model traffic. This pattern is honest for early features: you see real failure modes quickly. It becomes expensive when several teams copy the same client with slightly different timeouts and none of them own the aggregate rate limit.
Dedicated model gateway
A separate service owns provider credentials, timeouts, retries, schema validation, and cost metrics. Product services call a narrow internal API and receive a typed result or a classified error. Policy changes happen once. Product teams never embed the raw Mistral key. The gateway is also the natural place to attach per-tenant budgets, model-id pins, and redaction filters before prompts leave your network.
Asynchronous workers
The request path enqueues work and returns a job id. Workers call Large 4, validate, and write results. User latency is decoupled from model latency. You take on queue semantics, idempotency, and eventual consistency. This is the right shape for batch enrichment, offline eval, and any flow where a spinner waiting on a long completion would be a product failure even when the model succeeds.
Choosing axes
Latency sensitivity, number of calling services, key custody, and whether the user can wait dominate the choice. Quality of the model does not remove the need for a placement decision. Draw the diagram before you paste an SDK call into a controller. If you cannot say which process holds the key and which process may write to the database, the design is incomplete.
In-Process Placement
When it is enough
Single team, low-to-moderate QPS, early feature work, and a strict client with timeouts already in process. You learn real prompts before extracting infrastructure. Keep the client thin: transport, timeout, retry classification, and schema validation only. Business rules stay outside the client so extraction to a gateway later does not rewrite domain logic.
Failure behavior
A slow completion occupies a worker thread or async slot in the product service. A retry storm under 429 responses multiplies load on the same pool that serves unrelated routes. Guard with short deadlines and small retry budgets. Separate thread pools or worker queues for model calls from those for cheap database reads when the runtime allows it. Without isolation, a provider blip becomes a site-wide latency incident.
Configuration discipline
Read model id, timeout, and max tokens from environment. Fail startup if required values are missing. Log request ids with token counts for every attempt. Treat “default model latest” as a misconfiguration in production. Preview checkpoints move; pinned ids and dated eval packs are how you keep change control when Mistral updates the preview.
Limits
In-process placement scales model capacity only by scaling the product service. It is a poor long-term home for many teams sharing one key or one schema policy. When the second product service needs the same structured extraction, duplicate clients drift. That drift is the usual trigger to extract a gateway.
Gateway Placement
Boundary definition
The gateway accepts a versioned internal request: task type, inputs, schema identifier, and tracing metadata. It returns a validated payload or an error code. It does not write product databases. Keeping side effects out of the gateway preserves a clean trust story: the gateway may talk to Mistral; only product services may talk to customer data stores with write credentials.
Centralized controls
Retries for transport failures, rejection of schema-invalid bodies, rate-limit handling, and per-tenant cost counters live in one codebase. Product services stay thin. You can enforce a single maximum prompt size, a single PII redaction policy, and a single model-id allowlist. Centralization fails when product teams pressure you into an escape hatch that accepts raw prompts with no schema. Resist that pressure or the gateway becomes a reverse proxy with branding.
Failure containment
When Mistral is unhealthy, the gateway can open a circuit and fail fast. Callers degrade without filling their own worker pools on doomed requests. Pair circuits with explicit degradation behavior in each product: cached answers, queue-for-later, or user-visible partial functionality. A 503 without a product plan is only slightly better than a hang.
Contract risk
If teams bypass the typed API with free-form “just send this prompt” escapes, the gateway becomes a pass-through and validation evaporates. Keep the internal schema strict; add versions when needs grow. Publish the schema package as versioned artifacts both sides import. Contract tests in CI should fail the build when product and gateway disagree on required fields.
Async Worker Placement
Mechanics
Persist a job with an idempotency key, status, and payload reference. Workers claim jobs, call Large 4, validate, and store outcomes. Clients poll or subscribe for completion. Store prompt references rather than huge blobs inline when possible, and encrypt sensitive payloads at rest. Workers should load least-privilege credentials separate from the API tier.
Consistency
Define pending, succeeded, and failed clearly. A retry of the same key must not double-apply side effects. Re-planning on every retry without care produces divergent actions. Distinguish “retry provider transport” from “request a new plan from the model.” The first is often safe. The second can change the intended action and needs application policy.
Isolation benefits
Web tiers stay responsive. Worker autoscaling tracks model demand. Provider brownouts become queue lag instead of site-wide latency spikes—acceptable only when the product can wait. Expose queue age and failure rate on dashboards beside model latency so on-call can tell backlog from provider outage.
Operational cost
You need claim fencing, dead-letter queues, and visibility into attempt history. Teams that cannot operate a queue should prefer a gateway with hard timeouts. Dead letters need a human or automated triage path; a pile of failed enrichment jobs is still a production issue even when HTTP users are fine.
Comparison and Mandatory Controls
Placement comparison
Dimension | In-process | Gateway | Async workers |
|---|---|---|---|
Extra hop | None | Internal RPC | Queue wait |
Key custody | Product service | Gateway | Workers |
Provider stall impact | Blocks request workers | Contained if circuit works | Queue age grows |
Policy updates | Per service | Central | Central on workers |
User wait | Required | Required within budget | Optional |
Ops complexity | Lowest | Moderate | Highest |
Controls that never disappear
Control | Why | Applies to |
|---|---|---|
Hard timeouts | Bound worker occupancy | All |
Schema validation | Reject invalid structure | All, before side effects |
Semantic checks | Valid JSON can be wrong | All writes |
Idempotency keys | Client and worker retries | Gateway and async; recommended in-process |
AuthZ on tools | Model proposals are untrusted | All tool paths |
Token and cost metrics | Budget and regression detection | All |
Reliability and Failure Boundaries
Invalid and partial structure
Large 4 may return truncated or schema-invalid JSON under token limits or timeouts. Decode, validate, and fail closed. Do not repair with guesswork on the hot path.
Tool-call hazards
Partial tool arguments, unknown names, and repeated calls appear in agent designs. Cap steps, allowlist tools, and execute only after validation. Permission boundaries belong in the executor, not in the prompt text alone.
Prompt injection and scope
User content inside prompts can attempt to expand scope. Treat retrieved documents and ticket text as hostile. Keep system instructions out of user-editable stores, and enforce scope in code.
Retries
Retry transport and selected 5xx or 429 responses with backoff. Do not retry semantic validation failures. Async workers must distinguish “try the provider again” from “create a new plan.”
Testing and Rollout
Fixture suites
Store provider responses as fixtures for invalid JSON, valid-but-wrong fields, and slow responses. Unit-test validators without spending tokens.
Contract tests for gateways
Publish a schema package both sides import. Breaking changes require a version bump. CI should fail if product and gateway schemas diverge.
Load and budget tests
Measure p95 latency and cost per successful task at expected concurrency. Hypothetical capacity plans must be replaced with measured numbers on your stack before launch.
Migration path
Begin in-process with a strict client. Extract a gateway when a second service needs the same policy. Move to async only for paths that can tolerate delay and already have job status UX.
Production Verdict
Recommended default
For most multi-service backends, put Mistral Large 4 behind a gateway that owns the key, timeouts, retries, and schema validation. Keep product side effects in product services. Use async workers for batch and non-interactive work.
Checklist
Document the placement per route. Pin model ids. Validate before writes. Authorize tools. Emit latency and token metrics. Define idempotency. Test failure fixtures. Re-evaluate after weights release if you plan to self-host.
Boundary
Where the model sits is an isolation choice. What the model is allowed to change is an authorization choice. Make the model useful. Keep the system deterministic: the model proposes; the application decides.
