Guided production scenarios with step-by-step environments.
Environment and dependency management is the most common source of 'it worked on my machine'. venv, pip pinning and lockfiles make AI services reproducible end to end.
AI pipelines move data through lists, dicts, generators and sets every stage. Choosing the right structure decides memory use, lookup speed and how cleanly your prompt/retrieval code composes.
LLM calls, vector lookups and web fetches are all I/O-bound. Async Python runs thousands of them concurrently on one thread — the difference between minutes and seconds in a batch embedding pipeline.
LLM and vector API secrets must be created, stored, rotated and scoped as carefully as any credential. Keys in env vars, per-service keys, short-lived tokens and auditability are non-negotiable in production AI.
Provider rate limits and token-rate caps determine throughput and reliability. Exponential backoff with jitter, honoring Retry-After, and distinguishing retryable from non-retryable failures are core skills.
Model SDKs wrap HTTP in ergonomic clients. Learn the shared pattern — chat completions, streaming, tool schemas, embeddings — so switching providers or wrapping a gateway is a config change, not a rewrite.
Most enterprise knowledge lives in PDFs, HTML and docx. Extracting clean text — pages, structure, tables, metadata — is the first step of every RAG and document-GPT pipeline, and it shapes retrieval quality.
Every AI pipeline ingests data that lies: nulls where numbers should be, empty strings, schema drift, adversarial payloads. Validation contracts fail fast and make the pipeline trustworthy.
Corpora, prompts, embeddings and eval sets change over time. Versioning datasets links a model's behavior to the exact data it was built from — essential for reproducible evals and auditable AI.
Features are what the model sees; labels are the truth you train against. Dataset quality — coverage, labeling consistency, class balance — usually matters more than the model architecture.
To know how a model performs on new data, it must be evaluated on data it never saw in training or tuning. Leak-free, stratified, time-aware splits are the foundation of every honest number.
Accuracy hides more than it shows. Precision/recall, F1, ROC-AUC, calibration and per-segment breakdowns turn 'the model is good' into 'the model is good for this decision under these conditions'.
A neural network is a stack of weighted ladders: inputs pass through layers of weighted sums + activations to a prediction. Understanding the forward pass is the base layer for everything from CNNs to transformers.
ReLU, sigmoid, tanh, softmax — activations shape what a layer can represent and how gradients flow. Choosing wrong means vanishing gradients, dead neurons or unstable training.
A training loop is forward → loss → backward → update, repeated in epochs over batches. Optimizers (SGD, Adam), batch size, LR schedules and gradient clipping turn that loop from fragile to production-usable.
Large language models predict the next token autoregressively; generation is sampling that prediction, and 'reasoning' emerges from repeated next-token decisions. That framing explains capabilities, limits and costs.
Sampling settings shape the probability distribution — temperature flattens or sharpens it, top-p and top-k cut the tail. Deterministic tasks want low variance; creative tasks want room to roam.
LLM products consume hosted APIs: model selection, inputs (messages/tools/params), outputs (content, usage, finish reason), errors and rate limits. This concept wires a chat call with production hygiene.
Self-attention is attention where every position IS both attendee and attended — tokens find each other within the same sequence. It is the workhorse layer inside every transformer block.
Attention is position-agnostic by default — without position signals, 'the dog bit the man' equals 'the man bit the dog'. Positional encoding injects order, and its design constrains context length.
Decoder models must never peek at the future: a causal mask zeroes attention to later positions so each token sees only itself and its predecessors. It's the grammar license of next-token training and streaming generation.
Models read tokens, not words or characters. Tokenization chops text into a vocabulary of subword units — a process with its own costs, quirks and failure modes.
BPE builds a vocabulary by repeatedly merging the most frequent pairs of characters/subwords until the budget runs out. It is the standard subword algorithm behind most LLMs.
Special tokens are reserved vocabulary entries with syntax power: end-of-sequence, padding, role markers, tools and JSON boundaries. Misusing them silently breaks fine-tunes and chats.
Embeddings map text to fixed-length vectors where meaning shapes geometry: similar text lands near each other. They power retrieval, clustering, dedup and classifiers without retraining a model per task.
Cosine similarity measures orientation, not magnitude: identical direction = 1 regardless of length. Choosing the right metric — cosine vs Euclidean vs dot — decides whether your thresholds mean anything.
Embedding models differ by dimension, training data, multilingual coverage and cost. Choosing the right one (for your text and task) matters more than any post-processing.
Embeddings are judged on your task: retrieval hit rate, dedup F1, clustering coherence. Building small labeled eval sets and threshold sweeps turns 'feels right' into numbers you can ship.
Brute-force similarity over millions of vectors is too slow — indexing (HNSW, IVF, PQ) trades small recall loss for massive speed. Understanding index structure explains storage, latency and accuracy trade-offs.
Similarity search finds the nearest vectors to a query under a metric. Correcting the knobs — k, threshold, index metric — is where 'search works' separates real products from demos.
Vector search misses exact strings and rare tokens that lexical search nails, and vice versa. Hybrid search — combining BM25-style keyword with vector — covers both, and its success depends on fusion and weighting.
Real searches are scoped: 'in this repo', 'only faq'. Metadata filters narrow the ANN walk first, not after. Getting filter-aware search right beats post-filtering for speed and correctness.
Serving vectors at scale is systems engineering: observability of recall, freshness of index, safe deployments and capacity planning. The store matters, but the wa around it makes retrieval trustworthy.
Chunking decides how the corpus is cut for retrieval — and it decides whether answers exist in one retrievable piece. Structure-aware chunking with overlap beats raw character splits.
Retrieval connects the question to the right chunks: embedding search, keyword search, hybrid fusion, and reranking. The retriever determines what the LLM can even use.
A retriever shortlists widely; a reranker scores precisely. Reranking (cross-encoder style) reliably lifts top-k precision and is the cheapest big win in many RAG stacks.
Citations turn RAG from a claim machine into an evidence machine. Design the citation format, verify them, and make them appear in the UI in the right places.
RAG evaluation is three metrics, not one: retrieval quality (recall), groundedness (are claims supported), and answer quality (is it useful). Measured together, they direct every optimization.
When a RAG answer is wrong, the bug lives in one of three stages — corpus, retrieval, generation. Logging the right artifacts turns a mystery into a diagnosis.
The system prompt is your product's operating manual: identity, rules, policies, output contract. Seen as code, it gets versioning, testing and rollback like any other artifact.
A few worked examples in the prompt teach format, tone and edge-case behavior far better than prose. Standouts: quality examples beat quantity, and bad examples are contagious.
Prompts need regression tests: golden inputs with expected behaviors, run on every change and every model swap. A prompt test suite is cheap insurance against silent quality rot.
LLMs are fluent in prose and sloppy with schemas. Structured outputs — JSON under a schema — turn free text into parseable contracts, and parsing failures into design failures.
Parse, validate, and repair on failure: a typed guard rail around generative output. Strict validation with retry-with-errors keeps structured calls honest under noise.
LLM-friendly schemas are flat, optional-heavy, and enum-driven. Designing your contract to the strengths of generation — not to the dreams of a relational DBA — raises compliance and unblocks the product.
One schema → TypeScript types, runtime validators and model constraints together. Type safety removes the sharp boundary between 'the model's output' and 'my code's types'.
A tool's schema decides how often the model calls it and how reliably. Name, description and argument JSON Schema are product decisions: crisp semantics raise call quality, tight validation keeps execution safe.
Dispatch resolves a tool_call to a real handler: a registry, schema validation, allow/deny gates, and an execution envelope that returns results. It is where intent becomes side effects.
Tool calls fail: timeouts, bad args, service errors, rate limits. A typed error envelope returned to the model continues the loop gracefully; robust retry logic with backoff keeps agents working under transient failures.
A function-calling protocol — like the Anthropic tool-use format or OpenAI's — formalizes how models declare calls. Cross-provider abstractions (AI SDK, LangChain) normalize dialects so your app survives model swaps.
A code-execution tool lets the model run writes, math, manipulation or experiments. Powerful, and dangerous: it needs a sandbox, timeouts, resource caps, and a read-only bias. Done right it converts 'estimate' into 'compute'.
Governance decides which tools exist, who may invoke them, and with what limits — permission model, approval flow for writes, rate caps, and audit. It's the policy layer between model intent and real systems.
Tools widen the attack surface: prompt injection through tool inputs, SSRF from URL tools, exfiltration through write tools. Security means treating model-controllable arguments as untrusted input to every system they reach.
An agent is a model in a loop: it plans, calls tools, observes results, and iterates toward a goal — autonomously or with human checkpoints. Agents turn single-shot LLM calls into goal-driven systems.
The agent loop is the core runtime: reasoning turn, tool execution, result ingestion, capped iteration. Getting the loop right — message hygiene, termination, progress checks — determines reliability more than model choice.
A planner breaks a goal into ordered, executable steps before acting. Planning reduces drift on long tasks: the plan is a checkpointable contract the loop and humans can inspect, approve, and correct.
Memory is what an agent carries beyond its current context: working context, episodic history of past tasks, and semantic knowledge retrieved on demand. Good memory design decides whether an agent feels coherent — or forgets mid-sentence.
Evaluating agents needs more than final-answer scoring: tool selection, step correctness, safety, and honest failure each count. Trajectory-aware evals on golden tasks capture what single-answer grading misses.
Agents come in recognizable shapes: single-loop, router/planner, orchestrator-workers, evaluator-optimizer. Choosing the pattern — not inventing one — determines complexity, reliability and cost.
Orchestration is the runtime that decides which agent runs next, with what context, and when it's done — via fixed graph, router, or adaptive planner. Orchestrators turn patterns into products.
Multi-stage agents need state that is typed, versioned, inspectable and resumable — a single source of truth flowing through every stage. State schema design is architecture: where it lives decides what breaks.
Agents fail in recognizable ways: tool errors, exhausted caps, divergence, hallucinated success. Failure handling makes each failure explicit — detect, classify, retry-or-degrade, and report honestly with usable partial state.
Golden tests are curated input→expected-output pairs with rubrics — the ground truth your system must keep meeting. Small, sharp, and versioned, they're the regression net every change runs against.
Use a model to grade models: rubric-driven, fast, scalable evaluation where human annotation can't scale. Judges need calibration, position-bias control, and human spot-checks to stay trustworthy.
Offline evals are deterministic, repeatable runs of the system against golden data — the backbone of trusted iteration. Designing the run (seed, mocks, versions) makes results comparable across code changes.
Eval datasets are a living asset: gathered from real traffic, labeled consistently, kept clean and with coverage balance. How you manage eval data determines how much you can trust the gates built on it.
Guardrails are policy-enforcing checks around model input and output: what may enter (safe topics, formats), and what may leave (no toxic, private or unfounded content). They turn 'the model should behave' into measurable rules.
Input filters gate what the model is allowed to see: disallowed topics, format limits, injection-shaped text, and size caps on the way in. Filtering before inference blocks junk cheaply — before it costs tokens or acts.
Output filters police what leaves the system: toxicity, unsafe content, leaked private data, and rule-breaking claims. They run after generation and before delivery — the last line of policy defense.
The sentry pattern uses a second model to watch the first: an interloper that detects policy violations in output or drift in input, at a lower cost. Watchers who watch the watchers keep large systems legible.
A threat model enumerates who can attack your LLM system and how: injection, data theft, tool abuse, denial of service, jailbreaks. Naming the threats is the first and most valuable security step.
Prompt injection instructs a model from within user-or-content text: 'ignore your rules'. Defense is structural — separable instruction, untrusted-input framing, content-vs-code discipline — not hoping the model resists.
Leakage means model-visible data escaping the system: via responses, logs, embeddings, memory, or evals/training. Containment is about every copy — because models amplify whatever they can reach.
When an AI system misbehaves — leaks data, takes an odd action, or gets jailbroken at scale — you need a runbook: detect via logs, contain by killing tools and traffic, triage the blast radius, and root-cause with model and trace data.
Fine-tuning updates part of a pre-trained model's weights on your data — teaching style, formats, and behaviors no prompt reliably conveys. It's a quality lever with real cost and data requirements.
Tuning quality is dataset quality: clean, representative, deduplicated, correctly formatted instruction-response pairs plus held-out splits. Garbage in becomes a tuned model that memorizes garbage.
Parameter-efficient fine-tuning (LoRA especially) updates small adapter matrices rather than the whole model: cheaper, faster, and serial-compatible. Adapters become portable artifacts you can A/B and swap.
Base-model upgrades change your tuned adapters' world: re-evaluate, re-tune, or re-baseline deliberately. Update management is a planned cadence with rollback, not a fire drill.
Serving is how a trained model becomes an endpoint: choose hosted APIs, self-hosted inference servers, or edge deployables, then decide on batching, streaming and scales. Most teams start managed and graduate to self-hosted only for latency, data or cost reasons.
Managed providers abstract GPUs, scaling and retries behind a key; self-hosting puts them on you but lowers marginal cost and keeps data in-house. Choose with a decision matrix of latency, data, cost, and team capacity.
Batching packs many prompts into one forward pass so a GPU amortizes memory movement across requests. Continuous batching joins requests mid-generation; contiguous batching blocks are the naive and slower baseline.
Streaming sends tokens to the client as they are generated over SSE instead of after the full completion. It cuts perceived latency to the first token, keeps long generations from timing out, and is table stakes for chat UIs.
A latency budget breaks the user-visible deadline into measurable slices — network, gateway, queue, prefill, decode, app logic — so you tune the slice that hurts, not the whole stack. TTFT and inter-token latency are the two numbers that matter.
Throughput is tokens per second across the fleet, tuned by batch size, KV-cache budget, request concurrency and model precision. High throughput usually costs p99 latency, so you tune to a point, not to maximum.
Quantization stores weights in fewer bits (int8, int4) so the same GPU fits a bigger batch, a longer context or a larger model. It trades a little quality for speed and memory — always validated on your own evals, not assumed.
Speculative decoding has a small, fast draft model propose tokens while a larger model verifies batches of them in parallel, so you get big-model quality at a much higher tokens/sec. It shines on compute-scarce memory-bound paths.
Response caching stores exact LLM outputs for identical requests so repeat calls return instantly for free. It's the cheapest latency and cost win in AI apps — if your prompts are cacheable and your answers tolerate staleness.
Semantic caching keys replies by meaning, not exact text: a near-duplicate question reuses the stored answer when embedding similarity clears a threshold. It saves on repetitive-but-not-identical traffic at the price of a distance computation and collision risk.
Every cache is a time bomb of stale answers unless invalidation is explicit. TTL, version keys and event-driven purges keep cached LLM responses honest when products, prices or facts change.
LLM apps fail interestingly: garbage output with a 200 status. Observability means tracing each generation, its inputs, cost, latency and downstream outcome so you can see why an answer was wrong — not just that the API was up.
LLM metrics aggregate latency, token spend, cache hits and error rates; logs capture raw events with the trace id. Token counters deserve first-class status — they drive cost dashboards, drift flags and capacity planning.
Live evaluation samples production traffic and scores outputs with LLM judges, checks label or human feedback. It detects silent regressions after model swaps and prompt edits, turning observability into a closed loop.
Langfuse, Arize Phoenix, LangSmith, Helicone and Datadog provide tracing UIs, eval pipelines, prompt registries and dataset management out of the box. Pick by cost, self-hosting and eval depth — most teams start with one and standardize contracts around it.
You pay per token — prompt + completion — with output tokens often priced several times higher than input. System prompts, retrieved context and long chat histories are the hidden spend; cache reads and input discount tiers change the math.
Route easy requests to cheap small models and hard ones to frontier models. Classify by intent, size or confidence; escalate on failure. Tiers create 5-20x price spreads for work that mostly doesn't need the big model.
Exact-key caching and provider prompt-cache discounts cut repeated spend; semantic caching collapses near-duplicate requests. Each has a correctness cost — staleness — so pair caching with careful TTLs and versioned keys.
A production LLM stack threads request flow through gateway, context assembly, model calls, guardrails, tracing and storage. Stateless compute, redundant providers, prompt and model versioning, and evals gate every change.
The gateway is the single front door for model traffic: unified auth, routing, provider failover, rate limiting, caching, cost logging and observability. It turns provider-account sprawl and secret-in-app code into one controlled point.
LLM systems add new attack surfaces: prompt injection, data exfiltration through context (oracle attacks), PII in stored traces, and model access controls. Defenses span input/output guardrails, least-privilege tool scoping, redaction, and audit trails.
Inference capacity is about tokens per second from concurrency, batch size and hardware — plus the provider mix that absorbs spikes. Compute CUDA-visible batch throughput estimates, plan headroom and prepare throttling and overflow paths before the spike, not during it.