Production AI Architecture
Advanced 12 min read Production AI Architecture

Multi-Tenant Isolation

Many tenants, one stack, zero leakage

Sharing one model stack across tenants means isolating data, budgets, rate limits and egress. Row-level access enforced before retrieval, per-tenant prompt/context, spend attribution and audit — one leak between tenants is a product-ending bug.

multi-tenancyisolationtenant budgetsdata access
Prerequisites: GenAI Gateway
Before we start

What you will learn

  • Enforce data isolation before retrieval
  • Isolate rate limits and cost budgets per tenant
  • Audit tool actions and output per tenant
Introduction

What is this?

Multi-tenant AI serves many customers from one inference tier while isolating their data and spend: retrieval and context filtered by tenant at the source, prompts/keys namespaced, rate limits and budgets enforced per tenant, and tool use logged per tenant with approval. Shared cost subtleties — cache pollution, KV context bleed — need explicit design.

Motivation

Why do you need it?

  • One tenant's documents leaking into another's context is catastrophic trust loss.
  • A noisy tenant must not starve others or blow the shared bill.
  • Compliance requires per-tenant action logs even when models are shared.
Think about it

Mental model

Imagine

Multi-tenancy

Think of

An apartment building

Everyone shares the boiler (inference tier) and lobby, but each unit is locked (data scoping), each unit pays its own bills (budgets), and the door has a log (audit).

Mechanics

How it works

  • Tag every document/embedding with tenant_id; filter all retrieval and tools by it.
  • Namespace prompts and messages per tenant; never join histories across tenants.
  • Rate-limit and budget per tenant at the gateway; cache keys include tenant.
  • Log tool calls and approvals per tenant; prompt versions are tenant-shared config or overrides.

Tenant boundaries.

Req
Scope
Serve
Log
Real world

In production

  • A SaaS knowledge-bot filters vector search by the calling org's tenant id in every query.
  • A legal platform enforces per-tenant document roles plus tool approvals logged to each org.
  • A marketplace isolates embeddings by tenant partition in the vector store.
Guidelines

Best practices

  • Filter by tenant at the store, not after retrieval — defense in depth to every layer.
  • Partition vector/search namespaces per tenant where indexes allow.
  • Re-check tenant scoping on every new tool call, not just the initial request.
  • Budget and alert per tenant; breaches page, degraded tenants degrade alone.
Watch out

Common mistakes

  • Retrieval filters only on the caller's model output (post-hoc).
  • Cache keys missing tenant → one org's answer served to another.
  • Tenant ids in prompts rather than enforced in layers.
When things break

Debugging tips

  • Cross-tenant answers → check the retrieval filter and cache key had tenant ids.
  • One tenant spiking spend → their budget alert fires; inspect their prompt and cache-hit trend.
  • Tenant config drift → version prompt overrides per tenant and diff them in traces.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Tenant isolation must be enforced?

2

Tenant ids should appear in cache keys.

Go deeper

Official documentation

Postgres row security
docs
Qdrant payload filters
docs