Design a Model Gateway for Cost Tracking and Rate Limits

A gateway sits between product services and Mistral Large 4 so credentials, policy, and money are managed once. Without it, every microservice invents its own retries until the shared key is throttled. With it, you can answer who spent what, enforce budgets, and validate outputs before callers see them.

Gateway Responsibilities

Single egress for provider calls

Only the gateway holds the Mistral API key. Product services authenticate to the gateway with internal identity, not with the upstream key. Rotation becomes a gateway operation. Leakage blast radius shrinks to one service’s secret store.

Policy enforcement point

Timeouts, retry classification, model allowlists, maximum prompt sizes, and schema validation run here. Callers should not each implement slightly different backoff math. Inconsistency is how some services hammer the provider while others behave.

Accounting plane

Every request records caller service, tenant, route, model id, tokens in, tokens out, latency, and outcome class. Accounting is not optional telemetry; it is how finance and engineering share a fact base when invoices arrive.

What the gateway is not

It is not the system of record for business entities and not a general workflow engine. It returns validated model results or errors. Side effects on customer data stay in product services after they accept the gateway response.

Authentication and Tenancy

Internal caller auth

Use service identity mutual TLS or signed tokens between mesh peers. Reject anonymous calls even on private networks. Map identity to a caller registration that declares allowed models and peak rates.

Tenant propagation

Pass tenant ids from product services as attested claims, not as free-form model-visible strings the gateway trusts without check. Budgets and rate limits key on tenant plus caller where required. Prevent one tenant from borrowing another’s budget through header spoofing by verifying claims against your auth layer.

Model allowlists

Register which model ids each caller may invoke. Preview pins and production pins differ. “Any model string” is how unexpected spend appears when a debug flag ships.

Error mapping

Translate upstream 401 and 403 into internal codes without leaking key material. Translate validation failures into 422-class responses callers can handle. Keep a stable public error taxonomy for the gateway’s clients.

Rate Limits and Backpressure

Limit dimensions

Apply limits per API key toward Mistral, per caller service, and per tenant. Global key limits protect the account. Caller limits stop a single buggy deploy from consuming the org quota. Tenant limits enforce product fairness.

Algorithm choices

Token buckets or sliding windows work if parameters are explicit and measured. Document burst size and refill. Prefer failing fast with retry-after metadata over queueing unbounded request bodies in gateway memory.

Interaction with retries

Retries consume the same budget. The gateway should count attempts toward rate limits and toward cost. Blind client-side retries around the gateway recreate the thundering herd the gateway exists to prevent.

Load behavior table

Cost Tracking

Token capture

Persist prompt and completion token counts from provider responses when present. For streaming, account final usage records. If the upstream omits usage, estimate cautiously and mark estimates distinctly so finance does not treat them as exact.

Price tables

Configure price per model per million tokens for input, output, and cached input if applicable. Verify before publication against current Mistral rates; store effective dates on price rows so historical cost can be recomputed when list prices change.

Budget enforcement

Hard budgets reject new paid calls when exhausted. Soft budgets alert only. Product preference depends on risk tolerance; document which routes use which mode. Refresh budget windows on predictable schedules and expose remaining budget to internal dashboards.

Cost metrics that matter

Report dollars per successful validated response, not only tokens per request. A route with cheap tokens and high semantic failure rates is still expensive. Join cost to outcome class so retries and validation failures are visible as spend without value.

Validation and Safety at the Edge

Response validation

Optionally validate structured outputs against registered schemas before returning to callers. Central validation raises baseline quality but must version schemas carefully. Callers that need custom semantic checks still apply them locally; the gateway handles structure and size.

Prompt size guards

Reject oversize prompts early. Long context is a cost amplifier. Enforce per-route ceilings lower than model maxima when product SLOs demand it. Count images and attachments toward cost and size policy for multimodal Large 4 calls.

PII redaction hooks

Support optional redaction filters before egress for routes that should not send raw secrets. Redaction is imperfect; still classify data and prefer not to send production secrets to hosted previews without approval. Log that redaction ran without logging the secret values.

Timeout policy

Set connect and total deadlines per route. Publish worst-case latency including allowed retries. Callers must set their own deadlines above the gateway’s so abandoned requests do not continue spending after the user leaves.

Implementation Notes

Repository shape

Separate admin config, request path, accounting writers, and provider adapters. Pin Mistral client versions. Keep price and limit tables declarative so changes do not require rewriting handlers. Avoid embedding environment-specific model ids in business logic.

Data stores

Use a fast store for rate counters and a durable store for usage ledger rows. Counters can be approximate under failure; ledgers should be append-only for audit. Reconcile with provider invoices on a schedule and document unexplained variance rather than silently rewriting history.

Testing

Fixture upstream timeouts, 429s, invalid JSON, and usage payloads. Prove limits trip deterministically under simulated parallel callers. Prove budget exhaustion blocks spend. Prove schema invalid responses are not retried. Load-test accounting write paths so metrics do not drop under peak traffic.

Rollout

Start in shadow mode: enforce auth and record cost without hard rejecting on budget. Then enable limits for non-critical callers. Finally enforce on revenue paths. Sudden hard enforcement without visibility creates self-inflicted outages that look like provider problems.

Production Verdict

Why the gateway pays for itself

Central keys, central limits, and central ledgers convert Large 4 from an unscoped shared bill into a managed dependency. Product teams move faster when they do not each reinvent rate logic, and platform teams can answer cost questions with data.

Checklist

Caller auth required. Model allowlists. Per-caller and per-tenant limits. Usage ledger with token fields. Dated price tables. Budget modes documented. Timeouts and bounded retries. Optional schema validation. Dashboards for spend and 429 rates. Runbooks for upstream throttle events.

Failure philosophy

When unsure whether to spend, fail closed on hard-budget routes. When upstream is unhealthy, shed load early. When validation fails, do not retry. Measure the failure path: rejected requests and invalid outputs are part of operating the gateway, not noise.

Boundary

The gateway makes Mistral Large 4 reachable under policy. It does not decide business truth. Callers still apply semantic checks before writes. The model proposes. The application decides. Cost controls ensure that decision happens within a budget you chose deliberately.