AI System Design · 23 topics
Design a RAG system for customer support over product documentation
Clarify first: document types (PDF, HTML, tickets), update frequency, languages, expected QPS, latency target (for example under 3 s), and whether answers must cite sources.
Ingestion: parse and clean documents, split into structure-aware chunks (about 300-800 tokens, small overlap), attach metadata (product, version, language, access level, updated_at), embed, and store in a vector DB plus a keyword index.
Query path: rewrite the question using chat history, run hybrid search (BM25 + vectors), rerank the top 20-50 with a cross-encoder, and pass the best 4-8 chunks to the LLM.
Generation: the system prompt says answer only from the context, cite chunk IDs, and say "I don't know" when the context is insufficient; use low temperature.
Safety and fallback: filter by user permissions before retrieval, and hand off to a human agent when confidence is low or the user is unhappy.
Evaluate and monitor: recall@k and MRR for retrieval, faithfulness and relevance for answers on a labeled set, plus thumbs up/down, deflection rate and cost per resolved ticket.
Trade-offs to mention: chunk size vs context, reranker latency vs precision, freshness (incremental re-indexing) vs cost.
A RAG answer is wrong even though the correct document exists. How do you debug it?
Split the pipeline and test each stage on this one query; do not tune the prompt first.
Was it ingested? Check parsing (tables, scanned PDFs), chunk boundaries cutting the answer in half, and metadata filters excluding it.
Was it retrieved? Look at the top-k for that query. If missing, try query rewriting, hybrid search, a better embedding model, or different chunk sizes.
Was it ranked too low? If the chunk sits at rank 30, add a reranker or retrieve more candidates before reranking.
Did the LLM ignore it? If the chunk is in the prompt but the answer is still wrong: shorten the context, reorder chunks, strengthen the grounding instruction, and check for conflicting chunks.
Fix the class of failure: add the query to a regression eval set and re-run all evals before shipping.
How do you decide on a chunking and retrieval strategy, and know when it is not working?
Start from document structure: split on headings and paragraphs, keep tables and code blocks intact, and add 10-15% overlap.
Match size to question type: short factual lookups favor small chunks; reasoning across a section favors larger chunks or parent-child retrieval (retrieve small, return the surrounding section).
Build a labeled set of real questions with the chunks that answer them, then compare strategies on recall@k and MRR instead of guessing.
Warning signs: right document but wrong chunk, partial answers, high latency from oversized context, users repeating the same question.
Change one thing at a time: chunk size, overlap, embedding model, hybrid weights, reranker.
Design a chatbot that answers from an approved knowledge corpus with authorization-aware retrieval
Store an access-control list (ACL) or tenant ID with every chunk at ingestion time.
Enforce permissions at retrieval: pass the user's groups as a metadata filter to both vector and keyword search; never filter after generation.
Keep multi-turn memory short: summarize older turns and rewrite follow-ups into standalone queries.
Show uncertainty: return citations, and refuse or escalate when top retrieval scores fall below a threshold.
Block unsupported claims with a faithfulness check (a second LLM call or NLI model comparing the answer with the retrieved chunks) before returning.
Audit: log user, query, chunk IDs and answer so access issues can be traced.
Design a chatbot over structured and unstructured data
Route first: a classifier or LLM decides whether the question needs SQL (structured), document search (unstructured), or both.
Structured path: text-to-SQL with the schema and a few examples in the prompt, a read-only database role, query validation (SELECT only, row limits, timeouts), then summarize the result.
Unstructured path: standard RAG with citations.
Combine: expose both as tools, let the LLM merge results in the final answer, and show the SQL and sources used.
Risks: wrong joins and hallucinated columns, so validate against the schema, retry with the error message, and evaluate on a set of question-to-SQL pairs.
Design a ChatGPT-style conversational service
Requirements: multi-turn chat, streaming tokens, millions of users, low time-to-first-token (around 1 s), safety, and cost control.
Request path: client, API gateway (auth, rate limits), chat service, inference cluster; stream tokens back with Server-Sent Events or WebSockets.
Conversation state: store messages in a database keyed by conversation ID; trim or summarize history to fit the context window; reuse the shared prefix (system prompt) with prefix/KV caching.
Inference: GPU pools with continuous batching, autoscaling on queue depth, separate pools per model size, and routing of easy requests to smaller models.
Safety: input and output moderation, with a fast classifier in the streaming path.
Reliability: timeouts, retries, a fallback model, and graceful degradation under load (queueing, shorter limits).
Observability: time-to-first-token, tokens per second, error rate, cost per request, and user feedback.
Design an end-to-end batching system for LLM queries
Two modes: offline batch (large jobs, cheap, results in hours) and online dynamic batching (requests grouped within a few milliseconds).
Offline: inputs in object storage, a job queue, workers sending requests under a concurrency limit, results and failures written to a store, retries with backoff, and an idempotency key per item so reruns do not duplicate work.
Online: a request queue plus a scheduler that groups requests by model and similar length; continuous batching in the serving engine (for example vLLM) adds new requests to the running batch at each decoding step.
Trade-off: bigger batches raise throughput but add latency, so set a max wait time and a max batch size.
Add priority queues, per-tenant quotas, and a dead-letter queue for items that keep failing.
Track queue depth, batch size, GPU utilization, and p50/p95/p99 latency.
How do you scale an LLM app from 3 testers to 10 requests per second in production?
Load test with realistic prompts and output lengths; measure p95 latency, error rate, and tokens per second before launch.
Check provider limits (requests and tokens per minute); request a higher quota or add a second provider.
Add resilience: timeouts, retries with exponential backoff and jitter, a circuit breaker, and a fallback model.
Add caching (exact-match and semantic cache for repeated questions) and trim prompts.
Release gradually: staging, then a canary or percentage rollout behind a feature flag, with a fast rollback path.
Monitor from day one: latency, errors, token cost, sampled quality checks, and alerts on thresholds.
Run the evaluation set against the production configuration before and after release.
How do you handle API rate limits and cost, and what if your paid LLM quota runs out?
Prevention: token budgets per user and per feature, shorter prompts, cached responses, smaller models for simple tasks, and a cap on max output tokens.
Handling limits: queue requests, retry on 429 with backoff and jitter (honor the retry-after header), and cap concurrency with a semaphore.
Fallbacks: a multi-provider client that fails over to another provider or a smaller or self-hosted model, with normalized request and response formats.
Degrade gracefully: serve cached or simpler answers, tell users about delays, and move non-urgent work to batch.
Visibility: a cost dashboard per tenant and feature, with alerts at 50%, 80% and 100% of budget.
How would you reduce LLM inference latency and cost?
Model level: a smaller or distilled model, quantization (INT8 or INT4), and speculative decoding.
Serving level: continuous batching, KV-cache and prefix caching, an optimized engine (vLLM, TensorRT-LLM), and streaming for perceived speed.
Application level: shorter prompts, fewer but better retrieved chunks, caching repeated queries, routing by difficulty, and parallelizing independent calls.
Measure quality as well as latency before and after, since compression can hurt accuracy.
Design an AI agent that books travel or takes actions on a user's behalf
Core loop: the LLM plans, picks a tool, reads the result, and repeats until done, with a hard cap on steps.
Tools: search flights, check calendar, hold booking, pay; each with a strict JSON schema.
Separate roles: the LLM proposes; the application validates arguments, checks the user's permissions, enforces policy and rate limits, and executes.
Irreversible actions: require explicit user confirmation before paying or booking, use idempotency keys so retries never double-book, and support cancel or undo.
State: store trip constraints and decisions outside the prompt so the agent can resume after a failure.
Failure handling: tool timeouts, retries, falling back to asking the user, and logging every tool call for audit.
Evaluate on task success, tool-call accuracy, steps taken, and cost.
Design an AI coding assistant or code reviewer
Context building: index the repo (code chunks, symbols, dependency graph) and retrieve relevant files; for review, send the diff plus related code.
Model routing: a small fast model for autocomplete (a few hundred ms), a larger model for chat, review, and multi-file edits.
Safety: run generated code only in a sandbox, scan for secrets and license issues, and require approval before anything executes.
Output quality: run linters, type checks and tests on suggestions; filter low-confidence review comments to avoid noise.
Privacy: respect repo permissions, redact secrets, and do not train on customer code by default.
Metrics: suggestion acceptance rate, edits after acceptance, test pass rate, and latency.
How would you evaluate an agent beyond checking its final answer?
Evaluate the trajectory: did it choose the right tools, with correct arguments, in a sensible order, without loops?
Metrics: task success rate, tool-call accuracy, number of steps, recovery from tool errors, cost and latency per task, and policy violations.
Methods: scripted test cases with expected tool calls, LLM-as-judge with a rubric over the full trace (calibrated against human labels), and simulated users with mocked tools for repeatable runs.
Record full traces (prompts, tool calls, results) so every failure is replayable.
Online: sample real sessions for review, track user corrections and abandonment, and A/B test changes.
How would you build an evaluation harness for an LLM system, offline and online?
Offline: a versioned dataset of real and edge-case inputs with expected outputs or rubrics, run automatically on every prompt, model, or retrieval change (CI for quality).
Scoring mix: exact match or code checks where possible, retrieval metrics (recall@k, MRR), LLM-as-judge for faithfulness and helpfulness, and human review of a sample.
Compare against a baseline and block releases that regress key metrics.
Online: shadow traffic, A/B tests, thumbs up/down, escalation and re-ask rates, and sampled review of production logs.
Feed production failures back into the offline dataset.
How do you detect and reduce hallucinations?
Reduce: ground answers in retrieved context, instruct the model to say "I don't know", lower the temperature, require citations, and constrain output with schemas.
Fix retrieval first, since many hallucinations come from missing or wrong context.
Detect: a faithfulness check (is each claim supported by the retrieved text), self-consistency across samples, and confidence thresholds on retrieval scores.
Handle: refuse or hand off when unsupported, and show sources so users can verify.
Measure on a labeled set and track the unsupported-claim rate over time.
How do you protect an LLM application against prompt injection and unsafe outputs?
Treat retrieved documents, web pages and tool results as untrusted data, never as instructions.
Layer defenses: input moderation, output moderation, a policy engine that validates tool calls, least-privilege tools, and a sandbox for code execution.
Never let the model decide permissions: the application checks the user's rights for every action.
Keep system instructions separate from user and retrieved content, flag instruction-like text in documents, and require confirmation for sensitive actions.
Red-team regularly with known injection prompts, and log safety decisions for audit.
Design a document ingestion and processing pipeline for an AI system
Intake: an upload API or connectors (Drive, S3, wiki) put events on a queue, so processing is asynchronous and retryable.
Stages: file type detection, parsing and OCR for scans, cleaning, structure-aware chunking, optional metadata extraction with an LLM, and batched embedding.
Storage: raw files in object storage, chunks and metadata in a database, vectors in a vector store; keep a version and content hash per document.
Updates: detect changes by hash and re-embed only changed chunks; handle deletes so removed content stops being retrievable.
Reliability: idempotent steps, a dead-letter queue, per-stage metrics, and quality checks (empty text, garbled OCR).
Scale: workers autoscale on queue depth, and batched embedding calls cut cost.
Design a semantic search system over company documents with an LLM
Index: embed chunks and keep a keyword index too; store metadata for filters (team, date, permissions).
Retrieve with hybrid search and rerank, then optionally let an LLM summarize the top results with citations.
Query understanding: rewrite or expand queries and detect filters ("last quarter", "from legal").
Enforce document-level permissions at query time.
Quality: build a labeled query-document set, measure recall@k, NDCG and click-through, and log zero-result queries.
Scale: an approximate nearest-neighbor index (HNSW or IVF), sharding by tenant or size, and caching popular queries.
Design an LLM-enhanced recommendation system
Keep the classic funnel: candidate generation (cheap, from embeddings and collaborative filtering), then a ranking model, then re-ranking with business rules.
Where LLMs help: generating item descriptions and tags, cold-start items and users, explaining recommendations in natural language, and handling conversational requests.
Where they do not: scoring millions of items per request is too slow and costly, so use LLMs offline or only on a short candidate list.
Precompute embeddings and LLM-generated features in batch, and cache generated explanations.
Evaluate with offline ranking metrics (NDCG, recall), then A/B test for engagement, diversity, and cost.
When would you choose fine-tuning over RAG, and when is prompt engineering enough?
Start with prompting: cheapest and fastest, and enough when the task is simple and examples fit in the prompt.
Use RAG when the problem is knowledge: facts that change, private data, or a need for citations.
Use fine-tuning when the problem is behavior: consistent style or format, domain-specific reasoning, lower latency and cost by moving to a smaller model, or tasks with plenty of examples.
They combine well: fine-tune for format and tone, RAG for facts.
Decide with an eval set: try the cheapest option first and move on only if the metrics fall short.
Design a multi-agent system
Use it only when a single agent with tools is not enough.
Typical design: an orchestrator routes tasks to specialist agents (retrieval, coding, analysis) that share state through clear contracts.
Control flow: step and cost limits, timeouts, loop detection, and a human in the loop for risky actions.
Keep roles separate: the LLM reasons, the orchestrator controls flow, a policy layer governs, and a sandbox executes.
Observability: trace every agent call and tool result under one request ID.
Evaluate end-to-end task success first, then per-agent metrics.
Design a real-time voice assistant
Pipeline: streaming speech-to-text, then an LLM (with tools), then streaming text-to-speech.
Latency budget: aim for roughly 1-2 s from end of speech to first audio by streaming at every stage.
Conversation handling: voice activity detection, interruption (barge-in) support, and short spoken-style responses.
Reliability: fallbacks for noisy audio, confirmation for risky actions, and a text transcript for logging and evaluation.
Metrics: word error rate, time to first audio, task success, and interruption handling quality.
How would you monitor and improve an LLM application in production?
Log per request: prompt version, model, retrieved chunk IDs, tool calls, latency, token counts, cost, and safety decisions (store prompt hashes if raw text is sensitive).
Dashboards and alerts: latency percentiles, error and refusal rates, cost per user, groundedness score, and drift in query topics.
Feedback loop: user ratings, escalations, and sampled human review feed a failure set.
Safe changes: version prompts and models, run the eval suite on every change, and roll out with canaries and A/B tests.
Keep a regression set so fixed bugs stay fixed.