RAG and Vector Databases · 54 topics
What is RAG, and what problem does it solve that fine-tuning does not?
Retrieval-augmented generation fetches relevant information from an external knowledge base at query time and gives it to the LLM as context, so answers are grounded in your data.
It solves the knowledge problem: private, large or frequently changing information, up-to-date facts, source citations, and per-user access control, none of which fine-tuning handles well because facts baked into weights are hard to update, audit or restrict.
Fine-tuning changes behavior (style, format, skills); RAG changes what the model knows at answer time. They combine well.
Updating a RAG system means updating documents in an index, not retraining a model.
Walk through a RAG system end to end.
Offline indexing: load documents, parse and clean them, split into chunks, enrich with metadata, embed the chunks, and store vectors (plus a keyword index) in a database.
Online query: take the user question, optionally rewrite it using chat history, embed it, retrieve candidates with hybrid search and permission filters, rerank, and select the top chunks within a token budget.
Generation: build a prompt containing instructions, the chunks (with source IDs) and the question; call the LLM; return the answer with citations.
Surrounding it: evaluation, caching, monitoring, security and an update pipeline that keeps the index fresh.
What is the difference between naive RAG, advanced RAG and modular RAG?
Naive RAG: chunk, embed, retrieve the top-k by similarity, stuff them into the prompt, and generate; a single retrieval pass with no checks.
Advanced RAG adds improvements before, during and after retrieval: query rewriting, hybrid search, metadata filtering, reranking, context compression and answer verification.
Modular RAG treats components as swappable modules (routers, retrievers for different sources, rerankers, validators, memory) and composes them in flexible pipelines, loops or agents, so the finance and legal knowledge bases can evolve independently.
For most enterprise use cases, advanced RAG is the right starting point; add agentic behavior only when queries demand it.
How do you build the prompt and assemble context for the LLM?
Structure: a clear instruction (answer only from the context, say when the answer is not there, cite sources), the retrieved chunks each wrapped with an ID and source metadata, then the user question.
Budget tokens: count tokens, reserve space for the answer and chat history, rank chunks by relevance, and truncate or drop the weakest rather than cutting a chunk mid-sentence.
Order chunks deliberately (best first, or best at the start and end), deduplicate overlapping chunks, and merge adjacent chunks from the same document.
Treat retrieved text as untrusted data and delimit it, to reduce prompt injection risk.
How do you get the model to cite sources and stay grounded?
Number the chunks in the prompt and require inline citations that refer to those IDs; ask for exact quotes for key claims where precision matters.
Validate afterward: check that every cited ID exists and that cited passages support the claim (a verifier model or entailment check), and drop or flag unsupported sentences.
Show users the sources so they can verify, and use low temperature and instructions to refuse when evidence is missing.
What is the "lost in the middle" problem, and how do you mitigate it?
LLMs tend to use information at the beginning and end of a long context better than information in the middle, so a relevant chunk buried among many can be ignored.
Mitigate by retrieving fewer but better chunks (reranking), placing the most relevant ones first and last, compressing or summarizing context, and splitting complex questions into steps.
Test with your own prompts and models, since the effect varies across models and context sizes.
What is context compression in RAG?
It shortens retrieved context before generation so the prompt contains mostly useful information: extract only the sentences relevant to the question, summarize chunks, or use a small model to filter irrelevant passages.
Benefits: lower cost and latency, less distraction, and room for more sources; risks: dropping a crucial detail or adding a summarization error.
Evaluate it by comparing answer quality with and without compression on your test set.
Do long-context models make RAG obsolete?
No: they complement it. Long context is convenient for a few documents, but stuffing a large corpus is expensive and slow on every query, accuracy for details in the middle can drop, and corpora of millions of documents will not fit anyway.
RAG gives selective, cheaper access, per-user permissions, citations, and freshness.
A common pattern is to retrieve generously, rerank, and pass a moderately large but curated context, using long context for the answer-synthesis step.
What are the common failure modes of RAG systems?
Ingestion: bad parsing, lost tables, wrong chunk boundaries, missing or stale documents.
Retrieval: relevant chunk not retrieved (vocabulary mismatch, poor embeddings, filters), or ranked too low, or irrelevant chunks dominate.
Context: too many or conflicting chunks, truncated context, lost in the middle.
Generation: ignoring context, hallucinating beyond it, wrong format, failing to say "I don't know".
Diagnose by inspecting each stage on failing queries, and measure retrieval and generation separately.
How do you reduce hallucinations in a RAG application?
Improve retrieval first (hybrid search, reranking, better chunks), because missing evidence is the main cause.
Constrain generation: answer only from context, require citations, low temperature, and allow abstaining.
Add checks: an answerability or faithfulness verification pass, similarity thresholds that trigger a fallback, and corrective retrieval when evidence looks weak.
Measure the unsupported-claim rate on a test set and monitor it in production with sampled review.
How do you handle conflicting or outdated documents?
Store version, effective date and source authority in metadata, then filter or boost newer and more authoritative sources at retrieval time.
Remove or archive superseded documents, and re-index changed ones promptly.
In the prompt, tell the model how to resolve conflicts (prefer the latest, cite both and flag the discrepancy), and surface dates in citations so users can judge.
What should the system do when the knowledge base cannot answer the question?
Detect weak evidence with score thresholds, reranker confidence, or an answerability check, and respond honestly that the information is not available instead of guessing.
Offer useful alternatives: ask a clarifying question, suggest related topics, route to a human or another data source.
Log these queries; they are the best signal for missing documentation.
How do you handle multi-turn conversations in RAG?
Rewrite each turn into a standalone query using recent history, so pronouns and references resolve before retrieval.
Keep conversation memory compact (recent turns plus a summary), and avoid retrieving again when the follow-up can be answered from earlier context.
Detect topic changes and reset retrieval context when needed, and make sure permissions and filters are applied on every turn.
What is Corrective RAG (CRAG), and how does it improve reliability?
CRAG adds a quality gate between retrieval and generation: a lightweight evaluator scores the retrieved documents as correct, ambiguous or incorrect.
If they are good, they are refined (keep only the relevant parts) and used; if they are bad, the system discards them and re-retrieves with a reformulated query or a different source such as web search; ambiguous cases combine both.
Standard RAG forces the LLM to make do with whatever it got, so noisy retrieval causes hallucination; CRAG reduces that, at the cost of an extra evaluation step and possible extra retrieval calls.
It works with black-box LLMs because it does not change the generator.
What is Self-RAG, and how does it differ from standard RAG?
Standard RAG always retrieves, always uses what it gets, and never verifies the answer.
Self-RAG trains the model to emit special reflection tokens that decide whether retrieval is needed, whether each retrieved passage is relevant, and whether the generated text is supported and useful; the model can then retrieve on demand, filter passages and critique its output.
Trade-off: it requires fine-tuning the generator, so it cannot be applied directly to closed API models; CRAG-style external checks give similar benefits without modifying the LLM.
What is agentic RAG, and when is it worth the extra cost?
An agent decides how to retrieve: it plans, chooses among tools (vector search, keyword search, SQL, web, APIs), checks whether results are sufficient, asks clarifying questions, and repeats until it can answer.
It handles multi-hop and multi-source questions that one-shot retrieval cannot, such as combining a PDF fact, a graph relationship and a database total.
Costs: higher latency and token use, harder evaluation, and risk of loops; cap steps and budget, log traces, and use it only when a well-tuned single-step pipeline fails on your important queries.
What is GraphRAG, and when does it outperform vector RAG?
GraphRAG uses an LLM to extract entities and relationships from the corpus into a knowledge graph, groups the graph into communities, and pre-summarizes each community; queries use graph traversal for local, entity-centric questions and community summaries for global ones.
It shines on multi-hop and relationship-heavy questions ("how is entity A connected to event B?") and global questions such as "what are the recurring themes across all 500 papers?", which no single chunk can answer.
Costs: expensive indexing (many LLM calls), schema and entity-resolution errors, and more maintenance when documents change; for simple lookup questions, plain vector or hybrid search is cheaper and often just as good.
How do you handle multi-hop questions in RAG?
A multi-hop question needs facts from several chunks (for example "who is the CFO of the company that acquired Acme?"), so single-shot retrieval and reranking usually fail because no single chunk contains the answer.
Decomposition: an LLM splits the question into sub-questions, retrieves for each, and assembles the answer; predictable but adds latency and cost.
Iterative or agentic retrieval: retrieve, reason, form the next query from partial answers, and repeat; more flexible, higher latency.
Graph-based retrieval: represent entities and relations in a graph so multi-hop becomes traversal; best for relationship-heavy domains. Often decomposition is enough.
What are HyDE and HyPE, and how do they differ?
HyDE (hypothetical document embeddings): at query time an LLM writes a hypothetical answer to the question, and you embed that text to retrieve similar real passages; it bridges the style gap between short queries and long documents but adds an LLM call to every query.
HyPE (hypothetical prompt or question embeddings): at indexing time an LLM generates questions each chunk could answer and you embed those, so a user question matches question-to-question; the cost is paid offline, with no added query latency but more storage and ingestion cost.
For tight latency budgets, prefer HyPE or cached rewrites over query-time HyDE.
What is RAG-Fusion or multi-query retrieval?
An LLM generates several variants of the user query (paraphrases, sub-aspects), each is retrieved separately, and the result lists are merged, commonly with reciprocal rank fusion.
It improves recall on ambiguous or underspecified questions, at the cost of extra LLM and retrieval calls.
Order of techniques by added latency: cached rewrite, HyPE (offline), small-model rewrite, HyDE, multi-query, decomposition, then full agentic loops.
How would you route queries across multiple knowledge sources?
A router (rules, a classifier, or an LLM with tool descriptions) decides which index or tool fits the question: product docs, policy documents, a SQL database, web search, or no retrieval at all for simple chat.
Keep sources separate when their schemas, permissions or update cycles differ, and merge results only when the question spans several.
Track routing accuracy as its own metric, provide a safe fallback (search everything or ask a clarifying question), and monitor which routes are used.
How do you do RAG over structured data such as databases and spreadsheets?
Use text-to-SQL (or an API) for questions with exact answers like counts, sums and filters: give the model the schema with descriptions and examples, use a read-only role, validate the query (SELECT only, row limits, timeouts), execute it, and have the LLM explain the result.
Use semantic retrieval over text columns or documentation for descriptive questions, and combine both when a question mixes them.
Risks: wrong joins and invented columns, so validate against the schema, retry with the error message, and evaluate on question-to-SQL pairs; never let embeddings answer an aggregation question.
What is multimodal RAG, and how do you build it?
Multimodal RAG retrieves and reasons over images, tables, charts and slides as well as text.
Approaches: convert non-text content to text (OCR, captions or vision-model descriptions, table linearization) and index it with normal text embeddings; or embed images directly with a multimodal embedding model; or use page-image retrievers that embed whole page screenshots, and pass retrieved images to a vision-capable LLM.
Costs and risks: heavier storage and inference, errors in extracted figures, and evaluation that must include visual questions.
How can knowledge graphs be combined with text RAG?
Graph-enhanced retrieval: use entity linking and relationship reasoning to expand or filter the query before text retrieval.
Structured RAG: retrieve triples or subgraphs directly and add them to the prompt as facts.
Hybrid reasoning: give the model both text passages and graph facts, with text for detail and the graph for relationships and consistency.
Maintaining the graph (extraction quality, entity resolution, updates) is the main cost, so adopt it only for domains where relationships drive the questions.
What is LongRAG?
LongRAG uses much larger retrieval units (whole documents or groups of related sections, thousands of tokens) so the retriever has fewer, longer items to rank and a long-context reader receives complete context instead of fragments.
It reduces the chance that a relevant fragment is missed or stripped of context, and makes the retriever's job easier, but it needs a long-context LLM, raises cost per query, and can reintroduce the lost-in-the-middle problem.
Choose it when documents are coherent units and a long-context model is affordable.
How do caching and memory fit into a RAG system?
Caching: reuse embeddings of repeated queries, cache retrieval results and final answers for popular questions, and use a semantic cache for near-duplicate questions, with expiry tied to document updates and with permission checks so one user's cached answer is never shown to another who lacks access.
Memory: keep short-term chat context compact, and store long-term user facts or past resolutions in a separate retrievable store that users can inspect and delete.
Both reduce latency and cost but must be invalidated correctly when sources change.
What is a vector database, and how does it differ from a traditional database?
A vector database stores high-dimensional embeddings alongside metadata and answers nearest-neighbor queries (find the k most similar vectors), using specialized indexes.
Traditional databases index exact values and ranges (B-trees, hash indexes) and answer equality or range queries; those indexes do not help when you need similarity in hundreds of dimensions.
Vector search is approximate by design, so the key trade-off is recall versus latency versus memory, and results are ranked by similarity rather than matched exactly.
Many traditional systems now offer vector support (PostgreSQL with pgvector, Elasticsearch, OpenSearch), so the real question is dedicated versus integrated.
What are the main concepts in a vector database's data model?
A collection (or index or namespace) holds records; each record has an ID, a vector, and a payload of metadata (such as text, source, tenant and timestamps).
The collection is configured with the vector dimension, a distance metric (cosine, dot product or Euclidean) that must match the embedding model, and an index type with its parameters.
Operations: upsert (insert or update by ID), delete, query by vector with optional metadata filters, and fetch by ID; many systems also support hybrid (keyword plus vector) queries and batch operations.
How do you measure whether an ANN index is accurate enough?
Compute recall@k against exact (brute-force) search: for a sample of real queries, the share of the true top-k neighbors that the approximate index returns.
Plot recall against latency as you vary search parameters (efSearch or nprobe), and pick the cheapest setting that meets your recall target.
Remember that ANN recall is different from retrieval relevance: an index can have 99% recall against exact search and still return irrelevant chunks if the embeddings or chunking are poor.
What are the trade-offs between HNSW, IVF and disk-based indexes in a database?
HNSW: the modern default (pgvector, Pinecone, Weaviate and Qdrant all use it); highest recall and lowest latency, supports steady inserts, but stores the graph alongside the vectors, often 1.5 to 3 times the raw embedding size, and is awkward for bulk rebuilds and deletes.
IVF (and IVF-PQ): cluster-based, cheaper to build, more memory-efficient, lower recall at the same latency, tunable by how many clusters you search; good for very large, mostly static datasets.
Disk-based graph indexes (such as DiskANN) keep most data on SSD to serve very large datasets with limited RAM, trading some latency.
Choose with a benchmark on your own data, filters and update pattern.
Which vector database would you choose for a new RAG project?
Ask first: how many vectors, how often do they change, do you need heavy metadata filtering or hybrid search, do you already run PostgreSQL, and how much operations work can you take on?
Prototype or under about 10,000 chunks: an in-memory array or a library is enough. Moderate scale with an existing Postgres: pgvector. Zero-ops preference: a managed service such as Pinecone. Self-host with strong filtering and control: Qdrant or Weaviate. Hundreds of millions to billions of vectors: Milvus or a custom FAISS or Vespa deployment. Already on Elasticsearch or OpenSearch: use its vector fields.
The embedding model and chunking usually matter more to quality than the database brand; pick the boring option that fits the workload and avoid lock-in.
Compare pgvector, Pinecone, Qdrant, Weaviate, Milvus, Chroma and FAISS.
pgvector: a PostgreSQL extension; one system for relational data and vectors, ACID transactions, SQL filtering and joins; limited by single-node memory and CPU at large scale.
Pinecone: fully managed and serverless-friendly, minimal operations; closed source, cost grows with scale, and you have less control over indexing and consistency details.
Qdrant: open-source (Rust), strong filtered search and quantization, self-hosted or managed.
Weaviate: open-source with native hybrid (BM25 plus vector) search, built-in vectorizer modules, strong multi-tenancy.
Milvus: open-source, built for billion-scale with multiple index types and a distributed architecture, but heavier to operate.
Chroma: simple, developer-friendly for prototypes and small apps. FAISS: a library (not a database) for fast similarity search and building custom systems, with no built-in persistence, metadata or replication.
When does pgvector beat a dedicated vector database, and when does it not?
It wins when your data already lives in PostgreSQL: transactions across relational and vector data, one backup and security story, joins and rich SQL filters, and no new infrastructure; often enough for up to a few million vectors.
It struggles when the index no longer fits in memory on one machine, when you need very high query throughput, advanced filtered or hybrid search at scale, or independent scaling of vector workloads from your transactional database.
A practical approach: start with pgvector, measure, and migrate to a dedicated store if recall, latency or cost stop working.
How do vector databases handle metadata filtering, and why is it hard?
Post-filtering searches first and filters afterwards: simple, but can return too few results when the filter is selective.
Pre-filtering restricts candidates before or during the search; with HNSW a very selective filter can break graph connectivity and slow queries, so good systems use filter-aware traversal, payload indexes, or switch to exact search over the small filtered set.
Design for it: index the fields you filter on, keep filter cardinality in mind, test tail latency with the real filters (tenant, date, permissions), and consider partitioning by the most common filter.
How do upserts, deletes and consistency work in vector databases?
Writes go to a log or in-memory segment first, then are merged into the index in the background; many systems are eventually consistent, so a just-written vector may not appear in queries for a short window (and reads right after an upsert might miss it).
Deletes are usually soft (tombstones or deletion markers) and reclaimed by background compaction; graph indexes like HNSW handle deletes awkwardly, so heavy churn needs periodic rebuilds or compaction.
Design implications: do not assume read-your-writes, make ingestion idempotent (stable IDs, upserts), keep content hashes, and verify that deleted documents are really gone for compliance.
How do you shard and replicate a vector database?
Sharding splits vectors across nodes (by hash, tenant or collection) so each shard searches a portion and a coordinator merges the top-k results (scatter-gather); it scales memory and write throughput but adds query fan-out latency.
Replication keeps multiple copies for availability and read throughput; reads can be spread across replicas, with eventual consistency between them.
Practical concerns: shard count versus fan-out cost, rebalancing when adding nodes, per-shard recall (each shard must return enough candidates), and multi-region replication for latency and disaster recovery.
How do you estimate capacity for a vector database?
Raw storage is vectors times dimensions times bytes per value: 10 million vectors of 1,536 dimensions in float32 is about 61 GB; HNSW graph overhead often takes this to roughly 90 to 185 GB (1.5 to 3 times) of RAM if the index is memory-resident.
Add payload and metadata storage, replicas for availability, and headroom for growth, rebuilds and compaction.
Shrink it with scalar (INT8) or binary quantization, product quantization, lower or Matryoshka-truncated dimensions, and disk-based indexes; then check recall again, since each saving costs some accuracy.
How do you design multi-tenancy in a vector database?
Options: one collection or index per tenant (strong isolation and easy deletion, but overhead grows with tenant count), a namespace or partition per tenant (a common balance), or a shared collection with a tenant ID filter (efficient for many small tenants but needs fast, well-tested filtering).
Consider isolation guarantees, per-tenant deletion and export (privacy requests), noisy-neighbor limits (quotas, rate limits), backups, and the cost of many small indexes.
Always enforce the tenant filter in the database query, never after retrieval, and test cross-tenant leakage.
How do you benchmark a vector database for your workload?
Use your own vectors, query distribution, metadata filters and update pattern; public benchmarks such as ANN-Benchmarks help for algorithms, but vendor tables rarely match your filters or hardware.
Measure recall@k against exact search, queries per second, p50, p95 and p99 latency under concurrent load, indexing and build time, memory and disk, behavior during writes and compaction, and cost.
Test failure and growth scenarios (node loss, rebalancing, 10 times the data), and compare at equal recall rather than at default settings.
How do you back up, migrate and recover a vector database?
Back up vectors and metadata (snapshots or exports), but also keep the source documents and the pipeline that produced the embeddings: if the index is lost you can rebuild it, though re-embedding millions of chunks costs time and money.
Migrating between databases: export vectors and metadata, transform to the target format, bulk import, recreate indexes with tuned parameters, then validate by running the same query set on both systems and comparing recall and results before switching traffic.
Record the embedding model, version and chunking configuration with the index so you know what any backup contains.
When is Elasticsearch or OpenSearch with vector fields the right choice?
When you already run a search stack and need strong keyword search, filtering, aggregations and operational tooling together with vector search, so hybrid retrieval stays in one system.
Trade-offs: vector performance and index options may lag specialized vector databases at very large scale, and tuning shards, memory and refresh behavior takes expertise.
Evaluate recall, latency and cost on your data before standardizing on it.
What security controls should a vector database deployment have?
Authentication and role-based access control, network isolation (private networking, no public endpoints), encryption in transit and at rest, and audit logging.
Application-level authorization: store ACL or tenant metadata with each record and filter in the query, because embeddings themselves can leak information about the source text.
Treat the corpus as an attack surface (poisoned or malicious documents), control who can write, and plan data-retention, deletion and residency requirements.
What are the main RAG evaluation metrics (RAGAS), and how do faithfulness and relevance differ?
Faithfulness (groundedness): are the claims in the answer supported by the retrieved context? It detects hallucination, even when the answer sounds right.
Answer relevancy: does the answer actually address the question asked? A faithful answer can still be off-topic or incomplete.
Context precision: how much of the retrieved context is useful, and are the useful chunks ranked high.
Context recall: does the retrieved context contain everything needed to answer (measured against a reference answer or labeled evidence).
Together they separate retrieval problems (precision and recall) from generation problems (faithfulness and relevancy); tools like RAGAS compute these with LLM judges, which you should calibrate against human labels.
How do you evaluate a RAG system when you have no labeled data?
Bootstrap a test set: sample chunks, have an LLM write realistic questions and reference answers from them, filter out vague or leaky ones, and manually review a subset.
Use reference-free metrics (faithfulness, answer relevancy) with LLM judges, plus programmatic checks (citations exist and match, answers in the right format).
Collect real queries from users as soon as possible, label a sample with domain experts, and grow the set with failures; synthetic data helps you start but rarely matches real user behavior.
How do you evaluate retrieval and generation separately, and why does it matter?
Retrieval: recall@k, precision@k, MRR or NDCG against labeled relevant chunks; generation: faithfulness, relevance and correctness given a fixed, known-good context.
Separating them tells you where to invest: low recall means fix ingestion, chunking, embeddings or search; good retrieval with bad answers means fix prompts, context assembly or the model.
Also evaluate end to end (answer correctness on real questions), since component metrics can look fine while the experience is poor.
How do you test changes safely and catch regressions in a RAG system?
Maintain a versioned evaluation suite and run it automatically on every change to chunking, embeddings, index settings, rerankers, prompts or the LLM; block releases that drop key metrics beyond noise.
Compare against the previous production configuration with confidence intervals, and keep a set of previously fixed failures as regression tests.
After offline checks, use shadow traffic, canary rollouts or A/B tests with online signals such as thumbs up and down, re-asked questions and escalations.
What does a latency breakdown for a RAG request look like, and how do you cut it?
Typical stages: query rewrite (100 to 300 ms if used), embedding (20 to 50 ms), retrieval (20 to 100 ms), reranking (50 to 200 ms), prompt assembly (small), and LLM generation, which is usually the biggest part (hundreds of milliseconds to several seconds).
Reduce latency by streaming the answer, running independent steps in parallel, caching embeddings and results, using smaller models for rewriting, limiting reranker candidates, and sending fewer tokens to the LLM.
Set a budget per stage, track p95 per stage, and define fallbacks (skip reranking or use a smaller model when a stage is slow).
How do you estimate and control the cost of a RAG application?
Cost per query is the sum of embedding the query, vector database and keyword search, optional reranker, and LLM input and output tokens; the LLM usually dominates, driven by how many chunk tokens you send.
Reduce it by retrieving fewer and better chunks, compressing context, routing easy questions to smaller models, caching answers and embeddings, using prompt caching for static prefixes, and limiting output length.
Indexing cost is separate (parsing, embeddings, enrichment such as LLM-generated context), so avoid re-embedding unchanged content, and track cost per resolved question rather than only per request.
How do you secure a RAG application end to end?
Permissions: enforce user and tenant access in the retrieval query using metadata or namespaces, and never let the LLM decide who can see what.
Prompt injection: treat retrieved text as untrusted data, delimit it, strip or flag instruction-like content, and limit tools and actions available after reading documents.
Data protection: redact personal data and secrets before indexing, encrypt data, set retention rules, and log retrievals for audit; control who can add documents to prevent knowledge-base poisoning.
Test with adversarial prompts and cross-user probes before release.
What does a production-readiness checklist for RAG include?
Quality: a labeled evaluation set, retrieval and generation metrics, abstention behavior for unanswerable questions, and human review of samples.
Reliability: timeouts, retries and fallbacks for each dependency, graceful degradation, rate limits and load tests.
Operations: index freshness monitoring, ingestion alerting, versioned prompts and models, cost dashboards, tracing of every request with retrieved chunk IDs.
Governance: access control, audit logs, data retention, and a feedback loop that turns user complaints into new evaluation cases.
Design an enterprise knowledge assistant where each employee may only see documents they have permission to read.
Ingestion: connectors for each source (drive, wiki, tickets) that sync content together with its ACL (users, groups, inheritance), with change events keeping both content and permissions fresh.
Index: store group or user permission metadata with every chunk; hybrid search with a filter on the requesting user's groups applied inside the query; reranking only on permitted candidates.
Query path: authenticate the user, resolve group membership, rewrite the question, retrieve with the filter, assemble context with citations, and generate; apply a final check that every cited chunk is permitted.
Risks and tests: stale permissions (revoked access must apply quickly), cache keys that include the permission context, adversarial tests for leakage, and audit logs of what was retrieved for whom.
Design a legal contract question-answering assistant with precise citations.
Parse contracts with layout awareness; chunk by clause and section, keep headings and clause numbers as metadata, and preserve defined terms and cross-references (parent-child retrieval returns the surrounding section).
Retrieve with hybrid search because exact terms and clause numbers matter, rerank, and require answers to quote or cite specific clauses with page references.
Add verification: check that each claim is supported by a cited clause, flag conflicts across documents, and show an uncertainty message or human-review path for low confidence.
Evaluate with lawyer-labeled questions, track faithfulness strictly, and keep documents and answers confidential with strong access control and auditing.
Design a code search and question-answering assistant over a large repository.
Index code by syntax boundaries (functions, classes), with file path, symbol names and docstrings as metadata; also index documentation, issues and commit messages.
Retrieval: hybrid search (identifiers need exact matching), a code-aware embedding model, and graph-aware expansion that follows references to definitions and callers; rerank and assemble context with file paths and line numbers.
Freshness: incremental re-indexing on commits or pull requests, with per-branch handling.
Safety and quality: respect repository permissions, run answers or generated changes through tests where possible, and evaluate on real developer questions with checks such as whether the cited files are correct.
Design a customer support assistant that uses help-center articles and past tickets.
Sources: published help articles (authoritative) and resolved tickets (rich but noisy); keep them in separate indexes or tag them, and prefer articles for policy answers.
Pipeline: classify intent, retrieve from both sources with hybrid search, rerank, answer with citations to articles, and use ticket examples only as supporting context after removing personal data.
Safeguards: refuse when confidence is low and hand off to a human with the conversation and retrieved context attached; never invent policies or prices; check account-specific questions through authenticated tool calls instead of retrieval.
Measure deflection rate, resolution and escalation rates, customer satisfaction, faithfulness, and cost per resolved ticket, and continuously add failures to the evaluation set.